Metadata index-based columnar data access method, device, equipment and medium
By employing metadata indexing in columnar databases, precise access and efficient reuse of columnar data files are achieved, solving the problem of slow metadata reading speed, reducing IO and memory overhead, and improving data access efficiency.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- BEIJING FLYWHEEL DATA TECH CO LTD
- Filing Date
- 2026-02-12
- Publication Date
- 2026-05-29
Smart Images
Figure CN122111336A_ABST
Abstract
Description
Technical Field
[0001] Embodiments of this disclosure relate to the field of computer technology, and more specifically to columnar data access methods, apparatus, electronic devices, and computer-readable media based on metadata indexes. Background Technology
[0002] Currently, in column-oriented databases, data is stored in column-oriented data files as the basic unit. Each column-oriented data file contains the actual data for multiple columns and their corresponding metadata information. Accessing column-oriented data files typically involves the following approach: because the metadata of all columns (including column data type, encoding method, compression information, data offset, statistics, etc.) is centrally stored in the metadata management unit at the end of the file, when reading data from a specific column, the metadata must be read first before the actual data can be accessed.
[0003] However, when using the above method, the following technical problems often arise: In scenarios with a particularly large number of columns, metadata reading becomes extremely slow. In practical applications, column-oriented databases are often used to process wide tables containing hundreds or even thousands of columns (such as log analysis and multidimensional report statistics). As the number of columns increases, the total amount of metadata stored in the metadata management unit expands dramatically, potentially reaching several MB. At this point, reading just one column of data randomly requires reading several MB of metadata, resulting in extremely slow read speeds and significant I / O and memory overhead.
[0004] The information disclosed in this background section is only intended to enhance the understanding of the background of the inventive concept, and therefore may contain information that does not constitute prior art known to those skilled in the art. Summary of the Invention
[0005] The summary portion of this disclosure is intended to provide a brief overview of the concepts, which will be described in detail in the detailed description portion. This summary portion is not intended to identify key or essential features of the claimed technical solutions, nor is it intended to limit the scope of the claimed technical solutions.
[0006] Some embodiments of this disclosure provide columnar data access methods, apparatuses, electronic devices, and computer-readable media based on metadata indexes to address one or more of the technical problems mentioned in the background section above.
[0007] In a first aspect, some embodiments of this disclosure provide a columnar data access method based on metadata indexes, comprising: in response to receiving first access information for a target columnar data file and determining that the number of column fields corresponding to the target columnar data file is higher than a first number, obtaining a metadata index from a target file block stored on a target storage disk for loading into memory for real-time use, wherein the metadata index represents the location information of column metadata corresponding to each column field of the target columnar data file, the target file block includes: a column data block, a columnar metadata block, and the metadata index corresponding to the target columnar data file, the columnar metadata block being the complete column metadata corresponding to the target columnar data file; parsing the first access information for column access fields to obtain at least one column field; determining at least one metadata location information corresponding to the at least one column field based on the metadata index cached in memory; obtaining at least one metadata corresponding to the at least one metadata location information from the columnar metadata block; reading at least one column data corresponding to the at least one metadata from the column data block stored on the target storage disk based on the at least one metadata; and sending the at least one column data and the packaged file corresponding to the first access information to a data request terminal.
[0008] Secondly, some embodiments of this disclosure provide a column-oriented data access device based on metadata indexes, comprising: a first acquisition unit configured to, in response to receiving first access information for a target column-oriented data file and determining that the number of column fields corresponding to the target column-oriented data file is higher than a first number, acquire a metadata index from a target file block stored on a target storage disk, and load it into memory for real-time use, wherein the metadata index represents the location information of column metadata corresponding to each column field of the target column-oriented data file, and the target file block includes: a column data block corresponding to the target column-oriented data file, a column-oriented metadata block, and the metadata index, wherein the column-oriented metadata block is the location information of column metadata corresponding to each column field of the target column-oriented data file. The system includes: a complete column metadata; a parsing unit configured to parse the column access field of the first access information to obtain at least one column field; a determining unit configured to determine at least one metadata location information corresponding to the at least one column field based on the metadata index of the memory cache; a second obtaining unit configured to obtain at least one metadata corresponding to the at least one metadata location information from the column storage metadata block; a reading unit configured to read at least one column data corresponding to the at least one metadata from the column data block stored in the target storage disk based on the at least one metadata; and a sending unit configured to send the at least one column data and the packaged file corresponding to the first access information to the data request terminal.
[0009] Thirdly, some embodiments of this disclosure provide an electronic device, including: one or more processors; and a storage device having one or more programs stored thereon, such that when the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any implementation of the first aspect.
[0010] Fourthly, some embodiments of this disclosure provide a computer-readable medium having a computer program stored thereon, wherein the program, when executed by a processor, implements the method as described in any implementation of the first aspect.
[0011] The various embodiments of this disclosure have the following beneficial effects: Through the metadata-index-based columnar data access method of some embodiments of this disclosure, precise access and efficient reuse of metadata can be achieved through the file data structure corresponding to the target columnar data file, reducing IO and memory overhead. Specifically, the reason for the large IO and memory overhead in the process of accessing related columnar data is that metadata reading is particularly slow in scenarios with a large number of columns. In practical applications, columnar databases are often used to process wide table data containing hundreds or even thousands of columns (such as log analysis, multidimensional report statistics, etc.). As the number of columns increases, the total amount of metadata centrally stored in the metadata management unit will expand rapidly, possibly reaching several MB. At this time, each random reading of a column of data requires reading several MB of metadata, resulting in particularly slow reading and large IO and memory overhead. Based on this, some embodiments of the column-oriented data access method based on metadata indexes disclosed herein firstly, in response to receiving first access information for a target column-oriented data file and determining that the number of column fields corresponding to the target column-oriented data file is higher than a first number, a metadata index is obtained from a target file block stored on the target storage disk and loaded into memory for real-time use. The metadata index represents the location information of the column metadata corresponding to each column field of the target column-oriented data file. The target file block includes: a column data block, a column-oriented metadata block, and the metadata index corresponding to the target column-oriented data file. The column-oriented metadata block contains the complete column metadata corresponding to the target column-oriented data file. Here, when the number of column fields is determined to be higher than the first number, it indicates that the target column-oriented data file has a large number of columns. By using a file structure of column data blocks, column-oriented metadata blocks, and metadata indexes on the target storage disk, accurate access and efficient reuse of subsequent metadata can be achieved, reducing IO and memory overhead. Here, the metadata index is small in size and can be stored in memory. The column-oriented metadata block stores complete metadata for a single column. Through the metadata index, metadata under the corresponding column field can be loaded on demand, avoiding loading all complete metadata into memory and reducing memory overhead. Then, the column access field of the first access information is parsed to obtain at least one column field, thereby determining the column data desired by subsequent first access information. Next, based on the metadata index of the memory cache, at least one metadata location information corresponding to the at least one column field can be accurately and efficiently determined with minimal memory overhead. Furthermore, without loading all metadata, at least one metadata corresponding to the at least one metadata location information can be retrieved from the column-stored metadata block as needed to subsequently obtain the required at least one column data. Further, based on the at least one metadata, the corresponding at least one column data is read from the column data block stored in the target storage disk to obtain the required data.Finally, the packaged file corresponding to at least one column of data and the first access information is sent to the data request terminal. In summary, by splitting the metadata of the target columnar data file into two parts—a "metadata index" and "columnar metadata blocks"—the required data can be retrieved quickly and accurately in scenarios with a large number of columns, while minimizing memory overhead. Attached Figure Description
[0012] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and elements are not necessarily drawn to scale.
[0013] Figure 1 This is a flowchart of some embodiments of the columnar data access method based on metadata indexing according to this disclosure; Figure 2 This is a schematic diagram of the structure of some embodiments of the columnar data access apparatus based on metadata indexing according to the present disclosure; Figure 3 This is a schematic diagram of the structure of an electronic device suitable for implementing some embodiments of the present disclosure. Detailed Implementation
[0014] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.
[0015] It should also be noted that, for ease of description, only the parts relevant to the invention are shown in the accompanying drawings. Unless otherwise specified, the embodiments and features described in this disclosure can be combined with each other.
[0016] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are used only to distinguish different devices, modules or units, and are not used to limit the order of functions performed by these devices, modules or units or their interdependencies.
[0017] It should be noted that the terms "a" and "a plurality of" used in this disclosure are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".
[0018] The names of messages or information exchanged between multiple devices in the embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.
[0019] This disclosure will now be described in detail with reference to the accompanying drawings and embodiments.
[0020] refer to Figure 1 The diagram illustrates a flow 100 of some embodiments of a metadata-indexed columnar data access method according to this disclosure. This metadata-indexed columnar data access method includes the following steps: Step 101: In response to receiving the first access information for the target columnar data file and determining that the number of column fields corresponding to the target columnar data file is higher than the first number, the metadata index is obtained from the target file block stored on the target storage disk and loaded into memory for real-time use.
[0021] In some embodiments, in response to receiving first access information for a target column-oriented data file and determining that the number of column fields corresponding to the target column-oriented data file is higher than a first number, the execution entity (e.g., an electronic device) of the above-described metadata index-based column-oriented data access method can obtain the metadata index from the target file block stored on the target storage disk and load it into memory for real-time use. The target column-oriented data file can be the data file accessed to obtain the required column-oriented data. A column-oriented data file is a data storage structure organized around columns. Its core feature is that all values of the same column in a data table are stored contiguously (rather than row-wise). Column-oriented data files are used to optimize the performance of heavy query scenarios such as Online Analytical Processing (OLAP). Here, column-oriented data can be data stored in the form of columns. The first access information can be a request to access data. That is, the first access information here can be a request for the required data. The number of column fields can be the number of fields corresponding to each column in the column-oriented data file. The number of column fields here can correspond to the number of columns in the target column-oriented data file. The first number can be a threshold for measuring whether the number of columns in the target column-oriented data file is too large. For example, the first number can be 1000. That is, when the number of columns is higher than the first number, it indicates that the target column-oriented data file has a large number of columns. The target storage disk can be a storage disk that stores the relevant information corresponding to the target column-oriented data file. The target file block is the file block stored on the storage disk corresponding to the target column-oriented data file. The metadata index can be an index of the metadata corresponding to the column-oriented data. The metadata index can represent the location information of the column metadata of each column field corresponding to the above-mentioned target column-oriented data file. Here, the data location of the column metadata corresponding to the column resource can be quickly located through the metadata index. The column metadata here is the metadata corresponding to the column data. Metadata can be the descriptive information of the column data. Here, the location information can include: column identifier, disk offset of the metadata block and metadata block length (i.e., corresponding to metadata location information). Metadata location information can be the content representing the location of the metadata. Metadata can include, but is not limited to, at least one of the following: data type, encoding method, compression information, data offset, statistical information. The above-mentioned target file block includes: the column data block, column-oriented metadata block and the above-mentioned metadata index corresponding to the above-mentioned target column-oriented data file. Among them, the column data block can be each actual column data corresponding to the target column-oriented data file. The aforementioned column-oriented metadata block represents the complete column metadata corresponding to the target column-oriented data file. In other words, a column-oriented metadata block can be a collection of metadata description information corresponding to column data. In practice, memory can use an LRU (Least Recently Used) eviction policy to cache metadata indexes.
[0022] Step 102: Parse the column access field of the first access information to obtain at least one column field.
[0023] In some embodiments, the execution entity may parse the first access information into column access fields to obtain at least one column field. The at least one column field may be any column field of the desired column data.
[0024] As an example, the aforementioned execution entity can use a pre-generated field parsing script to parse the column access field of the aforementioned first access information to obtain at least one column field.
[0025] Step 103: Based on the metadata index of the memory cache mentioned above, determine at least one metadata location information corresponding to at least one column field mentioned above.
[0026] In some embodiments, the executing entity can determine at least one metadata location information corresponding to the at least one column field based on the metadata index of the memory cache. There is a one-to-one correspondence between the column fields in the at least one column field and the metadata location information in the at least one metadata location information. For example, the metadata location information may include: offset and length.
[0027] As an example, the aforementioned execution entity can query at least one metadata location information corresponding to at least one column field identifier from the metadata index. The column field identifier can characterize the identity information of the column field.
[0028] Step 104: Obtain at least one metadata corresponding to the at least one metadata location information from the above-mentioned columnar metadata block.
[0029] In some embodiments, the execution entity can obtain at least one metadata corresponding to the at least one metadata location information from the columnar metadata block. There is a one-to-one correspondence between the metadata location information in the at least one metadata location information and the metadata in the at least one metadata. The metadata can be column metadata or descriptive information corresponding to column data.
[0030] As an example, the aforementioned execution entity can query the corresponding metadata from the columnar metadata block according to at least one metadata location information to obtain at least one metadata.
[0031] Step 105: Based on the above-mentioned at least one metadata, read at least one corresponding column data from the column data block stored in the target storage disk.
[0032] In some embodiments, the execution entity may read at least one column data from a column data block stored in the target storage disk based on the at least one metadata. There is a one-to-one correspondence between the metadata in the at least one metadata block and the column data in the at least one column data block. The at least one column data block may be a dataset required to respond to the first access information. Specific implementation details are omitted.
[0033] In some optional implementations of certain embodiments, before step 105, the steps further include: The first step, in response to receiving the first access information and determining that the number of column fields is lower than the second number, is to determine the memory occupancy rate at the current time. The memory occupancy rate can be a value between 0 and 1. The second number can be a threshold used to measure whether the number of corresponding columns in the target columnar data file is insufficient. The second number can be a pre-set value. For example, the second number can be 30. When the number of column fields is less than the second number, it indicates that the number of corresponding columns in the target columnar data file is insufficient.
[0034] The second step involves retrieving a columnar metadata block from the target file block, based on the determination that the aforementioned occupancy rate is less than the third number, and loading it into memory for real-time use. The third number can be a value used to measure whether the corresponding memory occupancy rate is high. In practice, the third number can be a value between 0 and 1. For example, the third number could be 0.4.
[0035] Here, if the occupancy rate is low and the number of columns in the target columnar data file is small, the columnar metadata block can be directly loaded into memory so that each metadata in the columnar metadata block can be directly traversed and at least one metadata required later can be selected.
[0036] The third step involves directly querying at least one metadata element corresponding to at least one of the aforementioned column fields from the aforementioned memory-cached columnar metadata block. Details will not be elaborated further.
[0037] Step 106: Send the packaged file corresponding to at least one column of data and the first access information to the data request terminal.
[0038] In some embodiments, the executing entity may send the packaged file corresponding to the at least one column of data and the first access information to the data requesting terminal. The data requesting terminal may be a requesting terminal that requires at least one column of data. Here, the data requesting terminal may be the terminal that sent the first access information.
[0039] In some optional implementations of certain embodiments, after step 106, the steps further include: The first step, in response to receiving multiple second access messages for a target columnar data file and determining that the number of column fields corresponding to the target columnar data file is higher than a first number, determines whether the total number of access messages corresponding to the multiple second access messages is higher than a third number. Here, the second access message can be a request to access data. That is, the second access message here can be a request for the required data. Each of the multiple second access messages is an access request initiated for the target columnar data file.
[0040] The second step, in response to the determination, is to determine the metadata location information set corresponding to each of the multiple second access information, thus obtaining multiple metadata location information sets. The specific implementation method will not be described in detail here.
[0041] The third step is to filter out a target metadata location information set from the multiple metadata location information sets mentioned above, selecting the set where the metadata location information appears more frequently than the target frequency. The target frequency can be a pre-set frequency value. Here, the target frequency can be a threshold used to measure whether metadata location information appears multiple times. For example, the target frequency could be "10". The target metadata location information can be the metadata location of column data that requires multiple selections for multiple second access information.
[0042] The fourth step is to obtain the target metadata dataset corresponding to the target metadata location information set from the aforementioned columnar metadata block. There is a one-to-one correspondence between the target metadata location information in the target metadata location information set and the target metadata in the target metadata set.
[0043] Fifth, based on the aforementioned target metadata dataset, read the corresponding first column dataset from the column data block stored in the target storage disk. The first column of data in the first dataset has a one-to-one correspondence with the target metadata in the target metadata set. The specific implementation details are omitted here.
[0044] Step 6: For each second access message, perform the following first generation step: Sub-step 1 involves determining the metadata location information group corresponding to the aforementioned second access information. The specific implementation details are omitted here.
[0045] Sub-step 2 involves removing at least one target metadata location information from the aforementioned metadata location information group that exists in the aforementioned target metadata location information set, resulting in a remaining metadata location information group. Here, the remaining metadata location information in the remaining metadata location information group may be metadata location information for which column data determination has not yet been performed.
[0046] Sub-step 3 involves generating the second column dataset corresponding to the second access information based on the aforementioned remaining metadata location information group. There is a one-to-one correspondence between the metadata location information in the remaining metadata location information group and the data in the second column dataset. The specific implementation details are omitted here.
[0047] Sub-step 4 involves obtaining at least one first column of data corresponding to at least one target metadata location information from the aforementioned first column dataset. There is a one-to-one correspondence between the target metadata in the at least one target metadata location information and the first column of data in the at least one first column. The specific acquisition method will not be elaborated further.
[0048] Sub-step 5: Generate the second access information corresponding column dataset based on the second column dataset and at least one first column data.
[0049] As an example, the aforementioned execution entity can combine the second column dataset with at least one first column data information column to obtain a column dataset.
[0050] The seventh step involves sending the obtained datasets of multiple columns and the corresponding packaged files for the aforementioned second access information to the data request terminal. Further details are omitted.
[0051] Here, by identifying the target metadata location information set that appears more frequently than the target frequency, we can avoid the situation where multiple second access information have many identical column fields, causing subsequent memory-based metadata indexes to repeatedly retrieve the same metadata location information, the same metadata, and the same column data, resulting in a waste of query resources and memory overhead.
[0052] In some optional implementations of certain embodiments, after step 106, the steps further include: The first step, in response to receiving multiple third access messages for a target columnar data file and determining that the number of column fields corresponding to the target columnar data file is higher than a first number, is to determine whether the number of access messages corresponding to the multiple third access messages is higher than a third number. Here, the explanation of the multiple third access messages can be found in the explanation of the multiple second access messages.
[0053] The second step, in response to the determination, is to determine the metadata location information set corresponding to each of the multiple third access information, thus obtaining multiple metadata location information sets. Details will not be elaborated further.
[0054] The third step is to perform deduplication on the above multiple metadata location information sets to obtain a deduplicated metadata location information set.
[0055] Here, because multiple third-party access information sets may require the same column data, there may be duplicate metadata location information in multiple metadata location information sets. However, each metadata location deduplication information set is unique and does not contain duplicates.
[0056] The fourth step involves generating a mapping between each piece of metadata location deduplication information and the access identifier corresponding to at least one associated third access information. Each piece of third access information has a corresponding access identifier. The access identifier can represent the identity information of the third access information. For example, the access identifier could be "0981". Here, each of the at least one associated third access information can be access information for the column data corresponding to the metadata location deduplication information. That is, the third access information here can be access information for the field content corresponding to the column field of the metadata location deduplication information. The mapping information can represent the mapping relationship between the metadata location deduplication information and at least one third access information, and can also indirectly represent the requirement relationship of the column field content.
[0057] The fifth step is to generate a mapping association table based on the obtained mapping information set. This mapping association table represents the mapping relationship between metadata location information and third-party access information.
[0058] As an example, the aforementioned execution entity can treat the mapping information as a row information to perform row concatenation and obtain a mapping association table.
[0059] Step 6: In response to determining that the occupancy rate of the aforementioned mapping and association table is lower than the fourth number, the aforementioned mapping and association table is cached in the aforementioned memory. Here, the occupancy rate of the mapping and association table can be the current occupancy rate caused by loading the mapping and association table into memory. The fourth number can be a threshold for measuring whether the memory occupancy rate is high. For example, the fourth number can be a value of 0.7.
[0060] Step 7: Based on the aforementioned metadata, locate the deduplication information set and generate the deduplicated column dataset. The specific implementation method will not be elaborated further. It is important to note that there is a one-to-one correspondence between the metadata locating deduplication information in the metadata locating deduplication information set and the column data in the deduplicated column dataset.
[0061] Step 8: Based on the deduplicated column datasets and the mapping table of the memory cache, generate multiple column datasets. Each column dataset in these multiple datasets has a one-to-one correspondence with a specific piece of third-access information.
[0062] As an example, for each third access information in the mapping association table, the deduplicated column data group corresponding to the third access information is queried from the above mapping association table and used as the column dataset corresponding to the third access information.
[0063] Step 9: Send the obtained multiple column datasets and the packaged files corresponding to the multiple third access information to the data request terminal.
[0064] As an example, the aforementioned execution entity can package each column dataset and its corresponding third access information into a packaged sub-file, and then send the packaged file to the data request terminal.
[0065] In some optional implementations of certain embodiments, the steps further include: The first step, in response to determining that the occupancy rate of the aforementioned mapping association table is higher than the fifth threshold, is to sort the multiple access identifiers corresponding to the multiple third access information based on the number of metadata location information corresponding to each third access information, thus obtaining an access identifier sequence. Here, the fifth threshold can be a threshold used to measure whether the memory occupancy of the mapping association table is high. For example, the fifth threshold could be 90%. There is a one-to-one correspondence between the third access information and the access identifiers. The access identifier can represent the identity information corresponding to the third access information.
[0066] As an example, multiple access identifiers can be sorted in descending order of the number of metadata location information to obtain an access identifier sequence.
[0067] The second step, for the target access identifier in the access identifier sequence, is to perform the following second generation step: Sub-step 1: Determine whether the target access identifier is the first access identifier corresponding to the access identifier sequence. The first access identifier can be the access identifier with the highest number of corresponding metadata location information in the access identifier sequence.
[0068] Sub-step 2, in response to determining that it is not, obtain the currently acquired subset of the third column data and the corresponding subset of the first metadata location information. The currently acquired subset of the third column data is a set of column data corresponding to at least one access identifier in the access identifier sequence preceding the target access identifier. There is a one-to-one correspondence between the first metadata location information in the subset of first metadata location information and the third column data in the subset of third column data. Based on the first metadata location information, the corresponding third column data is obtained.
[0069] Sub-step 3 involves determining the subset of second metadata location information corresponding to the target access identifier based on the aforementioned mapping association table. The specific implementation details are omitted here.
[0070] Sub-step 4: Determine whether there is second metadata location information in the first metadata location information subset that is in the second metadata location information subset.
[0071] Sub-step 5: In response to the determination of existence, based on the aforementioned subset of third column data, determine at least one third column of data corresponding to at least one second metadata location information. Specifically, there is a one-to-one correspondence between the second metadata location information in the at least one second metadata location information and the third column of data in the at least one third column of data.
[0072] As an example, the aforementioned execution entity can query at least one third column of data corresponding to at least one second metadata location information from the third column data subset.
[0073] Sub-step 6: Remove at least one of the above-mentioned second metadata location information from the above-mentioned second metadata location information subset to obtain the removed second metadata location information subset.
[0074] Sub-step 7: In response to determining that the removed second metadata location information subset is not empty, the corresponding fourth column data subset is determined based on the removed second metadata location information subset and using the metadata index cached in memory. There is a one-to-one correspondence between the metadata location information in the removed second metadata location information subset and the fourth column data in the fourth column data subset.
[0075] Sub-step 8: Combine the above-mentioned fourth column data subset with at least one of the above-mentioned third column data to obtain a column data combination, which serves as the column dataset corresponding to the target access identifier.
[0076] Sub-step 9: In response to determining that the target access identifier is the last access identifier in the access identifier sequence, the obtained multiple column datasets and the packaged files corresponding to the above multiple second access information are sent to the data request terminal.
[0077] Third, in response to determining that the target access identifier is not the last access identifier in the access identifier sequence, each of the fourth column data in the fourth column data subset is used as the third column data and added to the third column data subset. Each of the metadata location information in the second metadata location information subset after removal is used as the first metadata location information and added to the first metadata location information subset. The next access identifier corresponding to the target access identifier is determined as the target access identifier, and the second generation step described above is continued.
[0078] In some optional implementations of certain embodiments, after sending the obtained multiple column datasets and the packaged file corresponding to the multiple second access information to the data request terminal in response to determining that the target access identifier is the last access identifier in the access identifier sequence, the method further includes: The first step, in response to determining whether it is the first access identifier or whether there is no corresponding second metadata location information, is to determine the subset of second metadata location information corresponding to the target access identifier based on the above mapping association table. The specific implementation method will not be described in detail.
[0079] The second step involves locating the subset of information based on the aforementioned second metadata, and using the metadata index from the memory cache, determining the corresponding subset of data in the fourth column, which serves as the column dataset corresponding to the aforementioned target access identifier. The specific implementation details are omitted here.
[0080] Third, in response to determining that the subset of second metadata location information after the above removal is empty, at least one third column of data is determined as the column dataset corresponding to the target access identifier.
[0081] In some optional implementations of certain embodiments, after step 106, the steps further include: The first step involves, in response to receiving at least one fourth access message for at least one target columnar data file and determining that the number of column fields corresponding to the target columnar data file is greater than the first number, determining the file association relationship between the at least one target columnar data file. The file association relationship can be an association relationship between the file content of the various target columnar data files. For example, the file association relationship can be an association relationship based on the existence of at least one identical column field, or an association relationship based on the files belonging to the same domain.
[0082] The second step, in response to determining that the above file association characteristics satisfy the target association conditions, is to perform the following construction steps for each target columnar data file: Sub-step 1 generates multiple metadata location information based on the fourth access information corresponding to the target columnar data files. The target association condition can be that at least one column field is identical across the various target columnar data files. The specific implementation method will not be elaborated further.
[0083] Sub-step 2 involves constructing associations between multiple metadata location pieces and their corresponding column field identifiers to obtain multiple field mapping information. Column field identifiers represent the identity information of a column field. Each metadata location piece and its corresponding column field identifier can generate corresponding field mapping information. For example, multiple metadata location pieces may include: metadata location piece A, metadata location piece B, and metadata location piece C. Multiple column field identifiers may include: column field identifier A, column field identifier B, and column field identifier C. Multiple field mapping information may include: {metadata location piece A - column field identifier A}, {metadata location piece B - column field identifier B}, and {metadata location piece C - column field identifier C}.
[0084] The third step involves combining the mapping information according to the rule that the column field identifiers are the same. This combination of multiple field mapping information results in a set of field mapping information. Here, the multiple field mapping information can include: multiple field mapping information corresponding to each target columnar data file. That is, each target columnar data file has corresponding multiple field mapping information to obtain each field mapping information. At least one of the combined field mapping information sets contains the same column field identifier.
[0085] The fourth step is to generate corresponding column data for each field mapping information in the above field mapping information set, based on the metadata location information corresponding to the above field mapping information, as the mapping column data. The specific implementation method will not be described in detail here.
[0086] Fifth, based on the obtained mapped column dataset, generate at least one column dataset corresponding to at least one of the above-mentioned fourth access information. The specific implementation method will not be described in detail.
[0087] Step 6: Send the packaged file corresponding to at least one column dataset and at least one fourth access information to the data request terminal. The specific implementation details are omitted here.
[0088] In some optional implementations of certain embodiments, the steps further include: The first step, in response to the determination that the above file association characteristics do not meet the above target association conditions, is to select the first group of columnar data files whose corresponding file associations meet the above target association conditions from at least one of the above target columnar data files. The specific implementation method will not be described in detail.
[0089] The second step is to generate the first dataset group based on the above construction steps, for the first column of stored data files. The specific implementation method will not be described in detail here.
[0090] The third step is to remove the first columnar data file group from at least one of the aforementioned columnar data files to obtain the second columnar data file group. The specific implementation method will not be described in detail here.
[0091] Fourth, for each of the second columnar data files in the aforementioned second columnar data file group, perform the following third generation step: Sub-step 1 involves parsing the column access fields of the fourth access information corresponding to the second column storage data file to obtain at least one column field.
[0092] Sub-step 2 involves determining at least one metadata location information corresponding to at least one of the above-mentioned column fields based on the metadata index of the memory cache. The specific implementation method will not be described in detail here.
[0093] Sub-step 3 involves retrieving at least one metadata element corresponding to the at least one metadata location information from the aforementioned columnar metadata block. There is a one-to-one correspondence between the metadata location information in the at least one metadata location information and the metadata in the at least one metadata element. The specific implementation details are omitted here.
[0094] Sub-step 4 involves reading at least one column of data from the column data block stored in the target storage disk based on at least one of the aforementioned metadata, and using this data as the second column dataset corresponding to the second column data file. The specific implementation details are omitted here.
[0095] Fifth, based on the obtained first column dataset group and the aforementioned second column dataset group, generate at least one column dataset corresponding to the at least one fourth access piece of information. The specific implementation method will not be elaborated further. Each fourth access piece of information has a corresponding column dataset.
[0096] Step 6: Send the packaged file corresponding to at least one column dataset and at least one fourth access information to the data request terminal.
[0097] The aforementioned "optional" aspect, as another inventive point of this disclosure, solves another technical problem: "In scenarios where there are file associations between target columnar data files and the number of column fields is massive, the common approach is to generate corresponding column datasets one by one for each access information, which suffers from low efficiency. Given the urgent needs in this scenario, how to achieve efficient generation of access response content has become one of the main development directions." Based on this, this application first establishes multiple field mapping information between metadata location information and column field identifiers for each target columnar data file. Then, based on merging rules for column field identifiers that are the same, the mapping information is combined to obtain a set of field mapping information for each target columnar data file. Based on generating the field mapping information set, the corresponding packaged file can be generated completely and efficiently while reducing the waste of resources for querying many fields.
[0098] By determining the file associations between various columnar data files and establishing metadata location information based on file management relationships, The various embodiments of this disclosure have the following beneficial effects: Through the metadata-index-based columnar data access method of some embodiments of this disclosure, precise access and efficient reuse of metadata can be achieved through the file data structure corresponding to the target columnar data file, reducing IO and memory overhead. Specifically, the reason for the large IO and memory overhead in the process of accessing related columnar data is that metadata reading is particularly slow in scenarios with a large number of columns. In practical applications, columnar databases are often used to process wide table data containing hundreds or even thousands of columns (such as log analysis, multidimensional report statistics, etc.). As the number of columns increases, the total amount of metadata centrally stored in the metadata management unit will expand rapidly, possibly reaching several MB. At this time, each random reading of a column of data requires reading several MB of metadata, resulting in particularly slow reading and large IO and memory overhead. Based on this, some embodiments of the column-oriented data access method based on metadata indexes disclosed herein firstly, in response to receiving first access information for a target column-oriented data file and determining that the number of column fields corresponding to the target column-oriented data file is higher than a first number, a metadata index is obtained from a target file block stored on the target storage disk and loaded into memory for real-time use. The metadata index represents the location information of the column metadata corresponding to each column field of the target column-oriented data file. The target file block includes: a column data block, a column-oriented metadata block, and the metadata index corresponding to the target column-oriented data file. The column-oriented metadata block contains the complete column metadata corresponding to the target column-oriented data file. Here, when the number of column fields is determined to be higher than the first number, it indicates that the target column-oriented data file has a large number of columns. By using a file structure of column data blocks, column-oriented metadata blocks, and metadata indexes on the target storage disk, accurate access and efficient reuse of subsequent metadata can be achieved, reducing IO and memory overhead. Here, the metadata index is small in size and can be stored in memory. The column-oriented metadata block stores complete metadata for a single column. Through the metadata index, metadata under the corresponding column field can be loaded on demand, avoiding loading all complete metadata into memory and reducing memory overhead. Then, the column access field of the first access information is parsed to obtain at least one column field, thereby determining the column data desired by subsequent first access information. Next, based on the metadata index of the memory cache, at least one metadata location information corresponding to the at least one column field can be accurately and efficiently determined with minimal memory overhead. Furthermore, without loading all metadata, at least one metadata corresponding to the at least one metadata location information can be retrieved from the column-stored metadata block as needed to subsequently obtain the required at least one column data. Further, based on the at least one metadata, the corresponding at least one column data is read from the column data block stored in the target storage disk to obtain the required data.Finally, the packaged file corresponding to at least one column of data and the first access information is sent to the data request terminal. In summary, by splitting the metadata of the target columnar data file into two parts—a "metadata index" and "columnar metadata blocks"—the required data can be retrieved quickly and accurately in scenarios with a large number of columns, while minimizing memory overhead.
[0099] Further reference Figure 2 As an implementation of the methods shown in the above figures, this disclosure provides some embodiments of a columnar data access device based on metadata indexing. These device embodiments are similar to... Figure 1 Corresponding to the method embodiments shown, this metadata-indexed columnar data access device can be specifically applied to various electronic devices.
[0100] like Figure 2 As shown, a columnar data access device 200 based on metadata index includes: a first acquisition unit 201, a parsing unit 202, a determining unit 203, a second acquisition unit 204, a reading unit 205, and a sending unit 206. The first acquisition unit 201 is configured to, in response to receiving first access information for a target columnar data file and determining that the number of column fields corresponding to the target columnar data file is higher than a first number, acquire a metadata index from a target file block stored on a target storage disk, and load it into memory for real-time use. The metadata index represents the location information of the column metadata corresponding to each column field of the target columnar data file. The target file block includes: a column data block corresponding to the target columnar data file, a columnar metadata block, and the metadata index. The columnar metadata block is the complete column metadata corresponding to the target columnar data file. The parsing unit 202 is configured to... The first access information is parsed to obtain at least one column field; the determining unit 203 is configured to determine at least one metadata location information corresponding to the at least one column field based on the metadata index of the memory cache; the second obtaining unit 204 is configured to obtain at least one metadata corresponding to the at least one metadata location information from the column storage metadata block; the reading unit 205 is configured to read at least one column data corresponding to the at least one metadata from the column data block stored in the target storage disk based on the at least one metadata; and the sending unit 206 is configured to send the at least one column data and the packaged file corresponding to the first access information to the data request terminal.
[0101] It is understandable that the units described in the metadata-indexed columnar data access device 200 are related to the reference. Figure 1The steps in the described method correspond to each other. Therefore, the operations, features, and beneficial effects described above for the method also apply to the metadata-indexed columnar data access device 200 and the units contained therein, and will not be repeated here.
[0102] The following is for reference. Figure 3 It shows a schematic diagram of the structure of an electronic device (e.g., an electronic device) 300 suitable for implementing some embodiments of the present disclosure. Figure 3 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments of this disclosure.
[0103] like Figure 3 As shown, the electronic device 300 may include a processing unit (e.g., a central processing unit, a graphics processing unit, etc.) 301, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 302 or a program loaded from a storage device 308 into a random access memory (RAM) 303. The RAM 303 also stores various programs and data required for the operation of the electronic device 300. The processing unit 301, ROM 302, and RAM 303 are interconnected via a bus 304. An input / output (I / O) interface 305 is also connected to the bus 304.
[0104] Typically, the following devices can be connected to I / O interface 305: input devices 306 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 307 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 308 including, for example, magnetic tapes, hard disks, etc.; and communication devices 309. Communication device 309 allows electronic device 300 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 3 An electronic device 300 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively. Figure 3 Each box shown can represent a device or multiple devices as needed.
[0105] In particular, according to some embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, some embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication device 309, or installed from storage device 308, or installed from ROM 302. When the computer program is executed by processing device 301, it performs the functions defined in the methods of some embodiments of this disclosure.
[0106] It should be noted that, in some embodiments of this disclosure, the computer-readable medium described above may be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In some embodiments of this disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In some embodiments of this disclosure, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.
[0107] In some implementations, clients and servers can communicate using any currently known or future-developed network protocol such as HTTP (Hypertext Transfer Protocol) and can interconnect with digital data communication (e.g., communication networks) of any form or medium. Examples of communication networks include local area networks (“LANs”), wide area networks (“WANs”), the Internet (e.g., the Internet of Things), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any currently known or future-developed networks.
[0108] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device. The aforementioned computer-readable medium carries one or more programs. When the electronic device executes the aforementioned one or more programs, the electronic device causes the following actions: In response to receiving first access information for a target columnar data file and determining that the number of column fields corresponding to the target columnar data file is higher than a first number, the electronic device retrieves a metadata index from a target file block stored on a target storage disk and loads it into memory for real-time use. The metadata index represents the location information of the column metadata corresponding to each column field of the target columnar data file. The target file block includes: a column data block corresponding to the target columnar data file, a columnar metadata block, and the aforementioned metadata index. The columnar metadata block is the complete column metadata corresponding to the target columnar data file. The electronic device parses the first access information to obtain at least one column field. Based on the metadata index cached in memory, the electronic device determines at least one metadata location information corresponding to the at least one column field. The electronic device retrieves at least one metadata corresponding to the at least one metadata location information from the columnar metadata block. Based on the at least one metadata, the electronic device reads at least one column data corresponding to the column data block stored on the target storage disk. The electronic device then sends the at least one column data and the packaged file corresponding to the first access information to a data request terminal.
[0109] Computer program code for performing operations of some embodiments of this disclosure can be written in one or more programming languages or a combination thereof, including object-oriented programming languages such as Java, Smalltalk, and C++, and conventional procedural programming languages such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).
[0110] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.
[0111] The units described in some embodiments of this disclosure can be implemented in software or hardware. The described units can also be housed in a processor; for example, a processor may be described as including a first acquisition unit, a parsing unit, a determining unit, a second acquisition unit, a reading unit, and a sending unit. The names of these units do not necessarily limit the specific unit; for example, the sending unit may also be described as "a unit that sends the packaged file corresponding to at least one column of data and the first access information to a data request terminal."
[0112] The functions described above in this document can be performed at least in part by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), system-on-a-chip (SoCs), complex programmable logic devices (CPLDs), and so on.
[0113] The above description is merely a selection of preferred embodiments of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of the invention involved in the embodiments of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described inventive concept. For example, technical solutions formed by substituting the above-described features with (but not limited to) technical features with similar functions disclosed in the embodiments of this disclosure.
Claims
1. A columnar data access method based on metadata indexing, comprising: In response to receiving a first access message for a target columnar data file and determining that the number of column fields corresponding to the target columnar data file is higher than a first number, a metadata index is obtained from the target file block stored on the target storage disk and loaded into memory for real-time use. The metadata index represents the location information of the column metadata corresponding to each column field of the target columnar data file. The target file block includes: a column data block, a columnar metadata block, and the metadata index corresponding to the target columnar data file. The columnar metadata block is the complete column metadata corresponding to the target columnar data file. The first access information is parsed to obtain at least one column field; Based on the metadata index of the memory cache, determine at least one metadata location information corresponding to the at least one column field; Obtain at least one metadata corresponding to the at least one metadata location information from the columnar metadata block; Based on the at least one metadata, at least one column of data is read from the column data block stored in the target storage disk. Send the packaged file corresponding to the at least one column of data and the first access information to the data request terminal.
2. The method according to claim 1, wherein, Before reading at least one column data from the column data block stored in the target storage disk based on the at least one metadata, the method further includes: In response to receiving the first access information and determining that the number of column fields is less than the second number, the memory occupancy rate at the current time is determined; In response to determining that the occupancy rate is less than a third number, a columnar metadata block is obtained from the target file block and loaded into memory for real-time use; Query at least one metadata corresponding to at least one column field directly from the columnar metadata block of the memory cache.
3. The method according to claim 1, wherein, The method further includes: In response to receiving multiple second access messages for a target columnar data file and determining that the number of column fields corresponding to the target columnar data file is higher than a first number, determine whether the number of access messages corresponding to the multiple second access messages is higher than a third number; In response to the determination, the metadata location information set corresponding to each of the plurality of second access information is determined, thereby obtaining a plurality of metadata location information sets; From the multiple metadata location information sets, a target metadata location information set is selected where the frequency of occurrence of metadata location information is higher than the target frequency; Obtain the target metadata dataset corresponding to the target metadata location information set from the columnar metadata block; Based on the target metadata, read the corresponding first column dataset from the column data block stored in the target storage disk; For each second access message, perform the following first generation step: Determine the metadata location information group corresponding to the second access information; Remove at least one target metadata location information that exists in the target metadata location information set from the metadata location information group to obtain the remaining metadata location information group; Based on the remaining metadata location information group, generate the second column dataset corresponding to the second access information; Obtain at least one first column of data corresponding to at least one target metadata location information from the first column of the dataset; Generate the second access information corresponding column dataset based on the second column dataset and the at least one first column data; The obtained multiple column datasets and the packaged files corresponding to the multiple second access information are sent to the data request terminal.
4. The method according to claim 1, wherein, The method further includes: In response to receiving multiple third access messages for a target columnar data file and determining that the number of column fields corresponding to the target columnar data file is higher than a first number, determine whether the number of access messages corresponding to the multiple third access messages is higher than a third number; In response to the determination, the metadata location information set corresponding to each of the plurality of third access information is determined, thereby obtaining a plurality of metadata location information sets; The metadata location information sets are deduplicated to obtain a deduplicated metadata location information set. For each metadata location deduplication information, generate mapping information between the metadata location deduplication information and the access identifier corresponding to at least one third access information associated with it; Based on the obtained mapping information set, generate a mapping association table; In response to determining that the occupancy rate of the mapping association table is lower than the fourth number, the mapping association table is cached in the memory; Based on the metadata, locate the deduplication information set and generate the deduplicated column dataset; Based on the deduplicated column dataset and the mapping association table of the memory cache, multiple column datasets are generated; The obtained multiple column datasets and the packaged files corresponding to the multiple third access information are sent to the data request terminal.
5. The method according to claim 4, wherein, The method further includes: In response to determining that the occupancy rate corresponding to the mapping association table is higher than the fifth number, the multiple access identifiers corresponding to the multiple third access information are sorted according to the number of metadata location information corresponding to each third access information to obtain an access identifier sequence; For the target access identifier in the access identifier sequence, perform the following second generation step: Determine whether the target access identifier is the first access identifier in the access identifier sequence; In response to the determination that it is not true, obtain the currently acquired subset of the third column of data and the corresponding subset of the first metadata location information; Based on the mapping association table, determine the subset of second metadata location information corresponding to the target access identifier; Determine whether there is second metadata location information in the first metadata location information subset within the second metadata location information subset; In response to the determination of existence, at least one third column of data corresponding to at least one second metadata location information is determined based on the third column of data subset; Remove at least one second metadata location information from the second metadata location information subset to obtain a subset of second metadata location information after removal; In response to determining that the removed second metadata location information subset is not empty, the corresponding fourth column data subset is determined based on the removed second metadata location information subset and using the metadata index of the memory cache; The subset of data in the fourth column and the data in at least one third column are combined to obtain a column data combination, which serves as the column dataset corresponding to the target access identifier. In response to determining that the target access identifier is the last access identifier in the access identifier sequence, the obtained multiple column datasets and the packaged file corresponding to the multiple third access information are sent to the data request terminal; In response to determining that the target access identifier is not the last access identifier in the access identifier sequence, each of the fourth column data in the fourth column data subset is used as the third column data and added to the third column data subset. Each of the metadata location information in the second metadata location information subset after removal is used as the first metadata location information and added to the first metadata location information subset. The next access identifier corresponding to the target access identifier is determined as the target access identifier, and the second generation step is continued.
6. The method according to claim 5, wherein, After the method responds to determining that the target access identifier is the last access identifier in the access identifier sequence and sends the resulting multiple column datasets and the packaged file corresponding to the multiple third access information to the data request terminal, the method further includes: In response to determining that it is the first access identifier or that there is no corresponding second metadata location information, a subset of the second metadata location information corresponding to the target access identifier is determined according to the mapping association table; Based on the second metadata location information subset, the corresponding fourth column data subset is determined using the metadata index of the memory cache, and used as the column dataset corresponding to the target access identifier; In response to determining that the subset of second metadata location information after removal is empty, at least one third column of data is determined as the column dataset corresponding to the target access identifier.
7. A columnar data access device based on metadata indexing, comprising: The first acquisition unit is configured to, in response to receiving first access information for a target columnar data file and determining that the number of column fields corresponding to the target columnar data file is higher than a first number, acquire a metadata index from a target file block stored on a target storage disk, and load it into memory for real-time use. The metadata index represents the location information of the column metadata corresponding to each column field of the target columnar data file. The target file block includes: a column data block, a columnar metadata block, and the metadata index corresponding to the target columnar data file. The columnar metadata block is the complete column metadata corresponding to the target columnar data file. The parsing unit is configured to parse the first access information into column access fields to obtain at least one column field; The determining unit is configured to determine at least one metadata location information corresponding to the at least one column field based on the metadata index of the memory cache. The second acquisition unit is configured to acquire at least one metadata corresponding to the at least one metadata location information from the columnar metadata block; The reading unit is configured to read at least one column data from a column data block stored in the target storage disk based on the at least one metadata. The sending unit is configured to send the packaged file corresponding to the at least one column data and the first access information to the data request terminal.
8. An electronic device, comprising: One or more processors; Storage device, on which one or more programs are stored, When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1-6.
9. A computer-readable medium having a computer program stored thereon, wherein, When the program is executed by the processor, it implements the method as described in any one of claims 1-6.