A data query method and device, electronic equipment, storage medium and computer program product

Abstract

The application provides a data query method and device, electronic equipment, storage medium and computer program product, relates to the technical field of database technology, and the method comprises the following steps: processing a numerical secondary column to be queried based on a preset model, and determining relative position information of the secondary column to be queried in a corresponding secondary column group; wherein, the preset model is used for simulating the distribution of each secondary column in the numerical secondary column group; based on the relative position information, a target physical address corresponding to the secondary column to be queried is determined in an index file, and target position information is queried based on the target physical address; wherein, the index file is used for representing the corresponding relationship among the dictionary value corresponding to each secondary column, the relative position of the position information storage, and the physical address.

Description

Technical Field

[0001] This application relates to the field of database technology, and in particular to a data query method, apparatus, electronic device, storage medium, and computer program product. Background Technology

[0002] A cloud-native data warehouse is a database based on a cloud environment, with independent scalability for storage and computing, and strong analytical capabilities. Tables in a data warehouse use a primary key column to uniquely identify each record; other columns besides the primary key column are called secondary columns. Users can query these secondary columns to analyze patterns in specific metrics, thus improving production. Secondary columns incorporate dictionary and location information during storage, enabling precise indexing. However, querying a secondary column involves multiple read operations to retrieve dictionary information and a binary search to obtain the logical value. This results in low query efficiency due to the need for multiple interactions with the underlying storage medium. Summary of the Invention

[0003] This application provides a data query method, apparatus, electronic device, storage medium, and computer program product.

[0004] The technical solution of this application is implemented as follows:

[0005] This application provides a data query method, including:

[0006] The numericalized second-level column to be queried is processed based on a preset model to determine the relative position information of the second-level column to be queried in the corresponding second-level column group; wherein, the preset model is used to simulate the distribution of each second-level column in the numericalized second-level column group;

[0007] Based on relative location information, the target physical address corresponding to the second-level column to be queried is determined in the index file, and the target location information is queried based on the target physical address; wherein, the index file is used to represent the correspondence between the dictionary value, the relative location of the location information stored, and the physical address corresponding to each second-level column.

[0008] The above scheme also includes the following methods:

[0009] Identify the second-level column groups in the data file, as well as the dictionary value and position information corresponding to each second-level column in the second-level column group;

[0010] If the second-level column is a string, then the numerical metadata is determined based on the dictionary value corresponding to each second-level column and the order relationship between each dictionary value, and the dictionary value corresponding to each dictionary value is determined based on the numerical metadata; wherein, the numerical metadata is used to keep the order between the determined dictionary values ​​consistent with the order between the dictionary values.

[0011] The model metadata of the preset model is determined based on the distribution of each dictionary value; whereby the model metadata is the data used to determine the preset model.

[0012] Store the dictionary value and location information corresponding to each second-level column in order, and obtain the physical address where the dictionary value and location information of each second-level column are stored;

[0013] An index file is constructed based on the dictionary value corresponding to each second-level column, the relative location of the location information storage, the corresponding physical address, as well as the numerical metadata and model metadata.

[0014] In the above scheme, numerical metadata is determined based on the dictionary value corresponding to each second-level column and the order relationship between each dictionary value, and the dictionary value corresponding to each dictionary value is determined based on the numerical metadata, including:

[0015] The character position length is determined based on the maximum string length corresponding to each dictionary value, and the number of different characters and character set at each character position corresponding to each dictionary value are determined.

[0016] The cardinality of each character position is determined based on the right-to-left character position order and the number of distinct characters at each position; wherein, the cardinality of each character position is greater than the cardinality of the previous character position.

[0017] Based on the right-to-left character position order and the order relationship between each dictionary value, a corresponding numerical value is assigned to the character at each character position of each dictionary value; wherein, the numerical metadata includes: the character set at each character position, the cardinality of each character position, and the numerical value assigned to the character at each character position.

[0018] The dictionary value corresponding to each dictionary value is determined based on the cardinality of each character position in each dictionary value and the numerical value corresponding to the character at each character position.

[0019] In the above scheme, the radix for determining the position of each character is based on the right-to-left character position order and the different number of characters, including:

[0020] Based on the character position order from right to left, the product of the number of different characters corresponding to the N character positions preceding each character position is determined as the radix of each character position; where the radix of the first character position is a first predetermined value; N is an integer greater than 0.

[0021] In the above scheme, based on the right-to-left character position order and the order relationship between each dictionary value, a corresponding numerical value is assigned to the character at each character position of each dictionary value, including:

[0022] Sort the characters in the character set at each character position according to the order relationship between each dictionary value;

[0023] For each character position in the sorted character set, the corresponding numerical value is assigned to the characters in the character set sequentially according to the allocation logic of increasing second predetermined value.

[0024] In the above scheme, before processing the numericalized second-level column to be queried based on the preset model, the method also includes:

[0025] The transformation process of the second-level column to be queried determines the corresponding dictionary value to be queried;

[0026] The cardinality of each character position in the dictionary value to be queried is determined based on the numerical metadata, and the numerical value corresponding to the character at each character position is determined.

[0027] Multiply the cardinality of each character position in the dictionary value to be queried by the numerical value corresponding to the character at each character position, and then sum them up to determine the numerical value of the dictionary value after converting it to the numerical value of the second-level column to be queried.

[0028] In the above scheme, the dictionary value and location information corresponding to each second-level column are stored in order, and the physical address storing the dictionary value and location information of each second-level column is obtained, including:

[0029] According to the order of the second-level columns, store the dictionary values ​​and position information corresponding to M second-level columns in each storage page, and obtain the physical address of the dictionary value and position information of each second-level column; where M is an integer greater than 0.

[0030] In the above scheme, an index file is constructed based on the dictionary value corresponding to each second-level column, the relative location of the stored location information, the corresponding physical address, as well as the numerical metadata and model metadata, including:

[0031] Establish an addressing field corresponding to each storage page; wherein, the addressing field includes the dictionary value and physical address of the location information of the second-level column stored in the corresponding storage page;

[0032] An index file is constructed based on storage pages and their corresponding addressing domains, as well as numerical metadata and model metadata.

[0033] In the above scheme, based on relative location information, the target physical address corresponding to the second-level column to be queried is determined in the index file, and the target location information is queried based on the target physical address, including:

[0034] Based on the order of the relative position information representation and the number M of secondary columns stored in each storage page, the corresponding target storage page is determined;

[0035] Determine the target physical address in the target addressing domain corresponding to the target storage page;

[0036] Query target location information based on target physical address.

[0037] In the above scheme, the corresponding target storage page is determined based on the order of the relative position information representation and the number M of secondary columns stored in each storage page, including:

[0038] The starting and ending relative positions of the target storage page are determined by the ratio of the position represented by the relative position information to the number of secondary columns M stored in each storage page, and by the preset error.

[0039] The target storage page is determined based on the consistency detection results of the starting and ending relative positions.

[0040] In the above scheme, querying target location information based on the target physical address includes:

[0041] Based on the target physical address, query intermediate location information and intermediate dictionary values ​​in the target storage page;

[0042] After successful verification based on the intermediate dictionary value, the intermediate location information is determined to be the target location information.

[0043] This application also provides a data query device, including:

[0044] The determination unit is used to process the numericalized second-level column to be queried based on the preset model and determine the relative position information of the second-level column to be queried in the corresponding second-level column group; wherein, the preset model is used to simulate the distribution of each second-level column in the numericalized second-level column group;

[0045] The query unit is used to determine the target physical address of the corresponding storage of the second-level column to be queried in the index file based on the relative position information, and to query the target location information based on the target physical address; wherein, the index file is used to represent the correspondence between the dictionary value, the relative position of the location information stored, and the physical address of each second-level column.

[0046] This application also provides an electronic device, including a memory and a processor. The memory stores a computer program that can run on the processor, and the processor executes the computer program to implement the steps in the above-described method.

[0047] This application also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps in the above-described method.

[0048] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps in the above-described method.

[0049] In this embodiment, the numericalized dictionary values ​​to be queried are processed based on a preset model to determine the relative position information of the second-level column to be queried within the second-level column group. The preset model is used to simulate the distribution of each second-level column in the numericalized second-level column group. Based on the relative position information, the target physical address corresponding to the second-level column to be queried is determined in the index file, and the target location information is queried based on the target physical address. The index file represents the correspondence between the dictionary value, the relative position of the location information stored, and the physical address for each second-level column. Thus, the preset model is used to determine the relative position information, and its high efficiency improves the speed of determining the relative position information, thereby improving the query efficiency of the target location information. Furthermore, the physical address of the target location information can be directly determined in the index file using the relative position information. When faced with multiple second-level columns to be queried, the target location information can be quickly queried through the physical address. Compared to the related technologies that use multiple queries to determine logical values ​​and then query the location using those logical values, this reduces the number of interactions with cloud object storage, thereby improving query efficiency. Attached Figure Description

[0050] Figure 1 Flowchart of the data query method provided in the embodiments of this application Figure 1 ;

[0051] Figure 2 Flowchart of the data query method provided in the embodiments of this application Figure 2 ;

[0052] Figure 3 Flowchart of the data query method provided in the embodiments of this application Figure 3 ;

[0053] Figure 4 Flowchart of the data query method provided in the embodiments of this application Figure 4 ;

[0054] Figure 5 This application provides an example of the effectiveness of the data query method. Figure 1 ;

[0055] Figure 6 Flowchart of the data query method provided in the embodiments of this application Figure 5 ;

[0056] Figure 7 Flowchart of the data query method provided in the embodiments of this application Figure 6 ;

[0057] Figure 8 Flowchart of the data query method provided in the embodiments of this application Figure 7 ;

[0058] Figure 9 This application provides an example of the effectiveness of the data query method. Figure 2 ;

[0059] Figure 10 This application provides an example of the effectiveness of the data query method. Figure 3 ;

[0060] Figure 11 Flowchart of the data query method provided in the embodiments of this application Figure 8 ;

[0061] Figure 12 Schematic diagram of the data query device provided in the embodiments of this application Figure 1 ;

[0062] Figure 13 Flowchart of the data query method provided in the embodiments of this application Figure 9 ;

[0063] Figure 14 Schematic diagram of the data query device provided in the embodiments of this application Figure 2 ;

[0064] Figure 15 This is a schematic diagram of a hardware entity of an electronic device provided in an embodiment of this application. Detailed Implementation

[0065] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application are further described in detail below with reference to the accompanying drawings and embodiments. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0066] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0067] If the application documents contain similar descriptions such as "first / second", the following explanation shall be added: In the following description, the terms "first / second / third" are used only to distinguish similar objects and do not represent a specific order of objects. It is understood that "first / second / third" may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0068] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0069] Please see Figure 1 The following is a flowchart illustrating the data query method provided in the embodiments of this application. Figure 1 , will combine Figure 1 The steps shown are explained below:

[0070] S101. Based on the preset model, process the numericalized second-level column to be queried and determine the relative position information of the second-level column to be queried in the corresponding second-level column group; wherein, the preset model is used to simulate the distribution of each second-level column in the numericalized second-level column group.

[0071] In this embodiment, the data query device can perform deduplication and sorting on the secondary columns in the data file to obtain secondary column groups when storing the data file. Each secondary column in the secondary column group is converted into a dictionary value, and the position information corresponding to each secondary column is extracted. Numerical metadata is determined based on the dictionary values ​​of each secondary column in the corresponding secondary column group and the order relationship between each dictionary value. Each dictionary value is processed according to the numerical metadata to determine the corresponding dictionary value, and the model metadata of the preset model is determined according to the distribution of the dictionary values. When storing the dictionary values ​​and position information of each secondary column, an index file is constructed based on the relative position, physical address, numerical metadata, and model metadata. After obtaining the secondary column to be queried, the data query device can load part of the information in the index file into memory, process the dictionary value of the secondary column to be queried based on the numerical metadata, and determine the dictionary value corresponding to the secondary column to be queried; the data query device can process the dictionary value to be queried based on the preset model determined by the model metadata to determine the order (relative position information) of the secondary column to be queried in the multiple secondary columns of the secondary column group, that is, the order of the dictionary value to be queried in the dictionary value group corresponding to the secondary column group.

[0072] The data query device may include servers, cloud servers, terminals, and other devices with corresponding data query functions and data storage index building functions.

[0073] The dictionary value to be queried reflects the order of the dictionary value corresponding to the second-level column in the second-level column group.

[0074] The preset model is used to simulate the distribution of dictionary values ​​in each second-level column of the second-level column group. The preset model may include a greedy spline interpolation machine algorithm. The model metadata may include the first-order spline slope, intercept, and first interpolation obtained from the analysis of multiple dictionary value groups. Specifically, the model metadata is obtained by simulating the cumulative distribution function of multiple dictionary values ​​using a greedy spline interpolation machine algorithm.

[0075] The data file can be a single data table, which includes primary key columns and secondary columns for multiple objects. For example, the data table could include primary key columns and secondary columns for multiple individuals. The primary key column could include the unique identifier for each individual, and the secondary columns could contain information such as the individual's city, height, and age.

[0076] The dictionary value can be a string obtained by characterizing the corresponding second-level column.

[0077] The dictionary values ​​corresponding to each second-level column in the second-level column group are distributed according to the order of the second-level columns in the second-level column group.

[0078] The relative position information represents the storage location of the dictionary value and position information corresponding to the second-level column, and their order within the storage locations of each second-level column in the second-level column group. The physical address represents the address where the dictionary value and position information corresponding to the second-level column are stored on the disk. The physical address of the disk is a unique code that identifies the actual location of a disk storage unit, and it mainly consists of the following elements: cylinder number, platter number, and sector number.

[0079] In this embodiment, the numerical metadata may include one or more numerical values ​​for each character position in the dictionary value. The dictionary value corresponding to the second-level column to be queried can be determined by adding or multiplying the numerical values ​​corresponding to each character position of the dictionary value to be queried in the second-level column. The numerical metadata is used to maintain consistency between the order of the determined dictionary numerical values ​​and the order of the dictionary values ​​themselves.

[0080] In this embodiment, the second-level column to be queried is only converted to the corresponding dictionary value when it is a string, and then the dictionary value is processed by the preset model. When the second-level column to be queried is a number, it can be processed directly by the preset model.

[0081] S102. Based on relative position information, determine the target physical address of the second-level column to be queried in the index file, and query the target location information based on the target physical address; wherein, the index file is used to characterize the correspondence between the dictionary value, the relative position of the location information stored, and the physical address corresponding to each second-level column.

[0082] In this embodiment, the data query device can determine the target physical address storage location corresponding to the order represented by the relative position information in the index file, and obtain the target physical address. Based on the target physical address, the target position information corresponding to the second-level column to be queried is obtained at once. The index file is used to represent the correspondence between the dictionary value, the relative position of the location information storage, and the physical address corresponding to each second-level column.

[0083] The target location information may include the logical position of the corresponding second-level column in the data file. For example, the target location information may include "011000", which is used to represent the second and third rows of the second-level column to be queried in the corresponding data table.

[0084] In this embodiment, the numericalized second-level column to be queried is processed based on a preset model to determine its relative position within the second-level column group. The preset model simulates the distribution of each second-level column in the numericalized group. Based on the relative position information, the target physical address corresponding to the second-level column to be queried is determined in the index file, and the target location information is queried based on the target physical address. The index file represents the correspondence between the dictionary value, the relative position of the location information, and the physical address for each second-level column. Thus, the preset model is used to determine the relative position information, and its high efficiency improves the speed of determining the relative position information, thereby improving the query efficiency of the target location information. Furthermore, the physical address of the target location information can be directly determined in the index file using the relative position information. When faced with multiple second-level columns to be queried, the target location information can be quickly queried through the physical address. Compared to the related technologies that use multiple queries to determine logical values ​​and then query the location based on those logical values, this reduces the number of interactions with cloud object storage, thereby improving query efficiency.

[0085] Please see Figure 2 The following is a flowchart illustrating the data query method provided in the embodiments of this application. Figure 2 , will combine Figure 2 The steps shown are explained below:

[0086] S201. Determine the second-level column groups in the data file, as well as the dictionary value and position information corresponding to each second-level column in the second-level column group.

[0087] In this embodiment, the data query device can extract multiple secondary columns from a data file and perform deduplication and sorting on these columns to obtain a secondary column group. Each secondary column in the secondary column group is then converted to determine its corresponding dictionary value, and the position information of each secondary column is obtained—that is, the row number of the corresponding secondary column in the data table.

[0088] In this embodiment, the data query device extracts all secondary columns from the data file and removes duplicate secondary columns. The removed secondary columns are then sorted to determine secondary column groups. Each secondary column in the secondary column group can be converted to determine its corresponding dictionary value. After determining the dictionary value, compression processing can be applied to the dictionary value. The data query device calculates the logical position of each secondary column in the secondary column group and determines the position information based on the logical position. Specifically, the row number of each secondary column in the data table can be calculated, and the position information can be determined based on the row number.

[0089] In this embodiment, the second-level columns in the data file are deduplicated and sorted to determine the second-level column groups; each second-level column in the second-level column group is converted into a corresponding dictionary value; and the position information of the second-level column corresponding to each dictionary value in the data file is statistically analyzed. In this way, the statistically analyzed position information can be stored on the disk according to the order of the second-level columns and its physical address can be obtained. During a query, the physical address can be quickly retrieved according to the order of the corresponding second-level columns, thereby improving the query efficiency of the corresponding position information.

[0090] For example, the secondary columns in the data file may include the city names of multiple people, and the data query device can sort the multiple city names to form a secondary column group.

[0091] S202. If the second-level column is a string, then the numerical metadata is determined based on the dictionary value corresponding to each second-level column and the order relationship between each dictionary value, and the dictionary value corresponding to each dictionary value is determined based on the numerical metadata.

[0092] In this embodiment, if the dictionary value corresponding to each second-level column is a string, the data query device can determine the numerical metadata based on the character position length of each dictionary value, the characters at each character position, and the order relationship between each dictionary value. The numerical metadata may include one or more numerical values ​​corresponding to each character position. For each dictionary value, the corresponding dictionary value is determined by adding or multiplying the numerical values ​​corresponding to each character position. The numerical metadata is used to maintain consistency between the determined dictionary numerical values ​​and the order between dictionary values.

[0093] Numerical metadata is used to convert each dictionary value into a corresponding dictionary value, while maintaining the order between the defined dictionary values ​​as consistent with the order between the dictionary values.

[0094] In this embodiment of the application, after the data query device converts each dictionary value into a corresponding dictionary value, it can then convert it into a 64-bit integer value for computer processing.

[0095] S203. Determine the model metadata of the preset model based on the distribution of each dictionary value; wherein, the model metadata is the data used to determine the preset model.

[0096] In this embodiment of the application, the data query device can analyze the distribution of multiple dictionary values, determine the corresponding model metadata (slope of the first-order spline, intercept and first interpolation), and construct a preset model based on the model metadata.

[0097] In this embodiment of the application, if each second-level column is a number, then there is no need to determine the numerical metadata; the corresponding model metadata can be determined directly by analyzing each second-level column.

[0098] S204. Store the dictionary value and location information corresponding to each second-level column in order, and obtain the physical address of the dictionary value and location information of each second-level column.

[0099] In this embodiment of the application, the data query device can store the dictionary value and location information corresponding to each second-level column in the storage page of the disk according to the order between each second-level column, and obtain the physical address where the dictionary value and location information corresponding to each second-level column are stored.

[0100] S205. Construct an index file based on the dictionary value, relative position of the location information stored, corresponding physical address, numerical metadata, and model metadata of each second-level column.

[0101] In this embodiment, the data query device can construct an index file based on the dictionary value corresponding to each second-level column, the relative position and physical address of the location information storage, and the relationship between the determined numerical metadata and model metadata. When it is necessary to query the second-level column to be queried, the index file can be read, and the model metadata and addressing fields can be loaded into memory to determine the corresponding relative position information and target physical address.

[0102] In this embodiment, the steps for constructing the second-level column precise index are as follows: Figure 3 The following steps will be explained:

[0103] S11. Read the secondary columns in the data file, remove duplicates to form dictionary values, and count the position information of each dictionary value in the data file.

[0104] In this embodiment, the data query device can read the secondary columns in the data file, remove duplicates and sort them to form secondary column groups, convert each secondary column in the secondary column group into a corresponding dictionary value, and count the logical position of the secondary column of each dictionary value in the data file. The database system can retrieve an entire row of data based on the logical position in conjunction with the primary index, such as the row number, which is the position information of a row of data in the data file.

[0105] S12. Is the second-level column a string?

[0106] In this embodiment, if the second-level column is a string, then S13 is executed. If the second-level column is not a string, then S14 is executed.

[0107] S13. Convert the dictionary values ​​of the second-level column of type string to numerical values ​​and output the numerical metadata.

[0108] In this embodiment, if the second-level column is a string, the corresponding numeric metadata needs to be determined for the dictionary values ​​of the string-type second-level column for subsequent persistent storage, querying, etc.; and the dictionary value corresponding to each dictionary value is determined using the numeric metadata. If the second-level column is a number, S14 is executed.

[0109] S14. Apply machine learning algorithms to the numericalized dictionary values ​​and output model metadata.

[0110] In this embodiment, model metadata can be determined based on the distribution of multiple dictionary values. The model metadata includes the first-order spline slope, intercept, and first interpolation, which is obtained by simulating the cumulative distribution function of the dictionary values ​​using a greedy spline interpolation machine algorithm.

[0111] S15. Persistently store dictionary values, location information, numerical metadata, model metadata, etc., according to a precise index storage scheme.

[0112] In this embodiment, the second-level column groups in the data file are determined, along with the dictionary values ​​and position information corresponding to each second-level column in the second-level column groups. Numerical metadata is determined based on the dictionary values ​​corresponding to each second-level column and the order relationship between each dictionary value, and the dictionary value corresponding to each dictionary value is determined based on the numerical metadata. Model metadata for a preset model is determined based on the distribution of each dictionary value; wherein, the model metadata is the data used to determine the preset model. The dictionary values ​​and position information corresponding to each second-level column are stored in order, and the physical address storing the dictionary values ​​and position information of each second-level column is obtained. An index file is constructed based on the relative position of the stored dictionary values ​​and position information corresponding to each second-level column, the corresponding physical address, and the numerical metadata and model metadata. In this way, a preset model and an index file with a relative position and physical address correspondence can be constructed, allowing for fast queries based on the high performance of the preset model when querying data. Furthermore, the physical address where the target position information is stored can be directly determined in the index file through the relative position information, allowing for fast querying of the target position information through the physical address when facing multiple second-level columns to be queried, thereby improving query efficiency.

[0113] Please see Figure 4 The following is a flowchart illustrating the data query method provided in the embodiments of this application. Figure 4 , Figure 2 S202 in the process can also be implemented through S301 to S304, combining Figure 4 The steps shown are explained below:

[0114] S301. Determine the character position length based on the maximum string length corresponding to each dictionary value, and determine the number of different characters and character set at each character position corresponding to each dictionary value.

[0115] In this embodiment of the application, when the data query device determines that the second-level column is a string, it can count the maximum length of the string in the multiple dictionary values ​​corresponding to the second-level column group as the character position length, and determine the number of different characters and the character set at each character position.

[0116] For example, the three dictionary values ​​corresponding to the second-level column group can include "aay", "bdz", and "bcz". The character position length corresponding to these three dictionary values ​​is 3. From right to left, the 0th character position contains two different characters (y, z), the 1st character position contains three different characters (a, d, c), and the 2nd character position contains two different characters (a, b).

[0117] S302. Determine the cardinality of each character position based on the right-to-left character position order and the number of different characters at each character position; wherein the cardinality of each character position is greater than the cardinality of the previous character position.

[0118] In this embodiment, the data query device determines the radix of each character position by multiplying the number of different characters in the positions preceding each character position based on the character position order from right to left (i.e., starting from the 0th character position and moving left). The radix of each character position is greater than the radix of the previous character position.

[0119] In this embodiment, the data query device can determine the radix of each character position by multiplying the number of different characters corresponding to the N character positions preceding each character position based on the character position order from right to left; wherein, the radix of the first character position is a first predetermined value; N is an integer greater than 0. Therefore, N is an integer of 1 or higher, because the positions preceding the first character position do not meet the condition of including one character position, so the radix of the first character position must be set separately to the first predetermined value.

[0120] The first predetermined value may include 1.

[0121] For example, the base number for the character positions corresponding to “aay”, “bdz” and “bcz” can be: the base number for the 0th character position is 1 (a first predetermined value), the base number for the 1st character position is 2 (the number of different characters in the 0th position), and the base number for the 2nd character position is 3×2 (the product of the number of different characters in the 0th and 1st positions).

[0122] S303. Based on the right-to-left character position order and the order relationship between each dictionary value, assign a corresponding numerical value to the character at each character position of each dictionary value; wherein, the numerical metadata includes: the character set at each character position, the cardinality of each character position, and the numerical value assigned to the character at each character position.

[0123] In this embodiment, the data query device can assign a corresponding value to the character at each character position of each dictionary value according to a second predetermined value incrementing allocation logic, based on the right-to-left character position order and the order relationship between each dictionary value. The numerical metadata includes: the character set at each character position, the cardinality of each character position, and the numerical value assigned to the character at each character position.

[0124] In this embodiment of the application, the data query device sorts the characters in the character set at each character position according to the order relationship between each dictionary value; for the sorted character set at each character position, the device assigns corresponding values ​​to the characters in the character set sequentially according to the allocation logic of increasing second predetermined value.

[0125] The second predetermined value may include 0. In other embodiments, the second predetermined value may also be other values.

[0126] For example, the characters at position 0 (y, z), position 1 (a, d, c), and position 2 (a, b) are sorted. The data query device can assign 0 to the character "y" at position 0 and 1 to the character "z". It can assign 0 to the character "a" at position 1, 1 to the character "d", and 2 to the character "c". It can assign 0 to the character "a" at position 2 and 1 to the character "b".

[0127] S304. Based on the cardinality of each character position in each dictionary value and the numerical value corresponding to the character at each character position, determine the dictionary value corresponding to each dictionary value.

[0128] In this embodiment of the application, the data query device can determine the dictionary value corresponding to each dictionary value by multiplying the cardinality of each character position in each dictionary value with the value corresponding to the character at that position and then summing the results.

[0129] For example, the dictionary value of “aay” is calculated as 0×(3×2)+0×(2)+0=0, the dictionary value of “bdz” is calculated as 1×(3×2)+1×(2)+1=9, and the dictionary value of “bcz” is calculated as 1×(3×2)+2×(2)+1=11. The three dictionary values ​​are converted to “aay”, “bdz” and “bcz”, and the corresponding dictionary values ​​after numerical conversion are 0, 9 and 11. The order is the same before and after numerical conversion, and the calculation method is simple and efficient.

[0130] In this embodiment, dictionary value quantification must be able to efficiently quantify strings while ensuring the order of dictionary values ​​before and after quantification. If this order is lost during the quantification process, the quantification process becomes meaningless, and the relative position information determined by the preset model based on the quantified dictionary values ​​will be incorrect. For example, Figure 5 Given a set of dictionary values ​​[a, b, c, d, e], after being converted into dictionary values ​​[1, 2, 3, 4, 5] using an ordered numerical conversion algorithm, and then sorted, a machine learning algorithm is applied to obtain a preset model p(x) = x. To predict the position of b using machine learning, b is first converted to the value 2. Using p(2) = 2, it is calculated that the character b is located in the 2nd position of this set of dictionary values, and the calculation result is as expected. However, if an unordered numerical conversion algorithm is used, b is converted to the value 1. Using p(1) = 1, it is calculated that the character b is located in the 1st position of this set of dictionary values, and obviously, the calculation result is not as expected.

[0131] The following is a simple, efficient, and sequential variable-length string numeration algorithm. The algorithm steps are as follows: Figure 6 As shown, the steps will be explained in detail. To make the algorithm principle easier to understand, three dictionary values ​​"aay", "bdz", and "bcz" will be used as examples to explain the calculation principle of each step in detail.

[0132] S21. Determine the maximum string length of this set of dictionary values.

[0133] In this embodiment of the application, the data query device can determine the maximum string length K of the set of dictionary values ​​"aay", "bdz" and "bcz", where K=3.

[0134] S22. Calculate the number of distinct characters appearing at each character position in all dictionary values.

[0135] In this embodiment, the character position length corresponding to the three dictionary values ​​is 3. Two different characters (y, z) appear at position 0, three different characters (a, d, c) appear at position 1, and two different characters (a, b) appear at position 2. This can be expressed as n iN represents the number of distinct characters at the i-th character position. i This represents the character set at the i-th character position.

[0136] S23. Determine the base number for each character position based on the number of different characters appearing at each character position.

[0137] In this embodiment, the base for the character positions corresponding to "aay", "bdz", and "bcz" can be: the base for the 0th character position is 1 (a first predetermined value), the base for the 1st character position is 2 (the number of different characters at the 0th position), and the base for the 2nd character position is 3×2 (the product of the number of different characters at the 0th and 1st character positions). This can be represented by C. i This represents the base number at the i-th character position. When i equals 1, C i According to formula (1), the result is equal to 2. When i equals 0, C i It equals 1.

[0138] Formula (1)

[0139] Among them, C i Let K represent the cardinality at the i-th character position, K represent the maximum length of the string, i represent the position of the i-th character, and n represent the radix. i This represents the number of distinct characters at the i-th character position.

[0140] S24. Sort the different characters that appear at each character position and assign values ​​starting from 0.

[0141] In this embodiment, the data query device can assign the value 0 to the character "y" at position 0 and 1 to the character "z". It can assign 0 to the character "a" at position 1, 1 to the character "d", and 2 to the character "c". It can assign 0 to the character "a" at position 2 and 1 to the character "b". This can be used... <N i Q i > indicates the correspondence between this character and the assigned value.

[0142] S25. Numeric all dictionary values.

[0143] In this embodiment of the application, the data query device can quantify all dictionary values ​​according to formula (2) to determine the corresponding dictionary values.

[0144] Formula (2)

[0145] Where 0 < n < K. Q represents the corresponding dictionary value, q i It is the numerical value corresponding to the character at the i-th position of the dictionary value, C iThis represents the cardinality at the i-th character position, where i represents the i-th character position.

[0146] The numerical metadata includes: N i C i and q i Where C0 equals 1, C1 equals 1, and C2 equals 6. N0 equals (y, z), N1 equals (a, d, c), and N2 equals (a, b).

[0147] In this embodiment, the dictionary value quantification method counts the number of characters appearing at corresponding positions in all strings, generates a radix for each position, maps the characters at corresponding positions in the string to be converted to natural numbers, and finally accumulates the product of the natural number at each position and the radix. The dictionary value quantification algorithm in this embodiment is simple and efficient, while also preserving the sequential characteristics of the strings before conversion. Furthermore, this method has only two parameters, occupies very little memory, and is suitable for persistence.

[0148] In this embodiment, the character position length is determined based on the maximum string length corresponding to each dictionary value, and the number of different characters and character set at each character position corresponding to each dictionary value are determined; the cardinality of each character position is determined based on the right-to-left character position order and the number of different characters at each character position; wherein, the cardinality of each character position is greater than the cardinality of the previous character position; based on the right-to-left character position order and the order relationship between each dictionary value, a corresponding numerical value is assigned to the character at each character position of each dictionary value; wherein, the numerical metadata includes: the character set at each character position, the cardinality of each character position, and the numerical value assigned to the character at each character position; based on the cardinality of each character position in each dictionary value and the numerical value corresponding to the character at each character position, the dictionary value corresponding to each dictionary value is determined. In this way, based on the different number of characters and character sets corresponding to each character position of each dictionary value, the order relationship between each dictionary value can fully reflect the order of the dictionary value corresponding to each dictionary value in all dictionary values. Furthermore, by analyzing the distribution of multiple dictionary values ​​and determining the preset model, the distribution of multiple dictionary values ​​can be accurately simulated, thereby accurately reflecting the relative position information of the dictionary value to be queried. With more accurate relative position information, the corresponding target position information can be accurately found.

[0149] Please see Figure 7 The following is a flowchart illustrating the data query method provided in the embodiments of this application. Figure 6 , will combine Figure 7 The steps shown are explained below:

[0150] S401. The transformation process of the second-level column to be queried determines the corresponding dictionary value to be queried.

[0151] In this embodiment of the application, the data query device can convert the secondary column to be queried into the corresponding dictionary value to be queried.

[0152] S402. Determine the cardinality of each character position in the dictionary value to be queried, and the numerical value corresponding to the character at each character position, based on the numerical metadata.

[0153] In this embodiment of the application, the data query device can determine the cardinality of each character position in the dictionary value to be queried, and the numerical value corresponding to the character at each character position, based on the numerical metadata.

[0154] In this embodiment of the application, it is only necessary to convert the second-level column to be queried into the corresponding dictionary value and convert it into the corresponding dictionary value when the second-level column to be queried is a string.

[0155] S403. Multiply the cardinality of each character position in the dictionary value to be queried by the numerical value corresponding to the character at each character position, and then sum them up to determine the numerical value of the dictionary value to be queried for the second-level column to be queried.

[0156] In this embodiment of the application, the data query device can multiply the cardinality of each character position in the dictionary value to be queried by the numerical value corresponding to the character at each character position and then sum them up to determine the numerical value of the dictionary value to be queried for the second-level column to be queried.

[0157] In this embodiment of the application, it is only necessary to convert the second-level column to be queried into the corresponding dictionary value and convert it into the corresponding dictionary value when the second-level column to be queried is a string.

[0158] In this embodiment, the secondary column to be queried is numerically processed to determine the corresponding dictionary value. This dictionary value can then be processed using a preset model to determine the relative position information of the secondary column within the secondary column group. Furthermore, the physical address where the target location information is stored can be directly determined in the index file using this relative position information. When faced with multiple secondary columns to be queried, the target location information can be quickly retrieved using the physical address. Compared to related technologies that involve querying logical values ​​multiple times and then querying the location using those logical values, this reduces the number of interactions with cloud object storage, thereby improving query efficiency.

[0159] Please see Figure 8 The following is a flowchart illustrating the data query method provided in the embodiments of this application. Figure 7 , Figure 2 S204 to S205 can also be implemented through S501 to S503, combining Figure 8 The steps shown are explained below:

[0160] S501. According to the order between the second-level columns, store the dictionary values ​​and location information corresponding to the M second-level columns in each storage page, and obtain the physical address of the dictionary value and location information of each second-level column.

[0161] In this embodiment, the data query device can store the dictionary values ​​and position information corresponding to M second-level columns in each storage page, according to the order of the second-level columns. It also obtains the physical address of the storage page containing the dictionary value and position information of each second-level column. M is an integer greater than 0.

[0162] In this context, a page is a contiguous set of independently operable memory cells in Random Access Memory (RAM).

[0163] For example, a page can store the dictionary value and position information corresponding to two second-level columns, respectively.

[0164] S502. Establish the addressing field corresponding to each storage page; wherein, the addressing field includes the dictionary value and physical address of the location information of the second-level column stored in the corresponding storage page.

[0165] In this embodiment, the data query device can establish a corresponding addressing field for each storage page. The addressing field includes the dictionary values ​​and physical addresses of the location information of the secondary columns stored in the corresponding storage page.

[0166] For example, in combination Figure 9 . Figure 9 The storage page (Page) distribution information is illustrated. A page includes a dictionary value field, a location information field, and an addressing field. The dictionary value field stores dictionary values, and the location information field stores the corresponding location information. Because the dictionary field and the location information field store the same number of elements, the storage page uses an offset to map a dictionary value and its location information. For example, the second position in the dictionary value field corresponds to the second position in the location information field. The order of the addressing fields corresponds to the corresponding dictionary value field and location information field. The addressing field stores the first byte address of each dictionary value or location information, allowing the dictionary value and location information at that offset to be parsed. The storage page storage scheme in this embodiment isolates and aggregates dictionary values ​​and location information, using offsets to associate dictionary values ​​and their location information. This storage design allows for the use of conventional compression algorithms to reduce the storage space of the dictionary field and location information field.

[0167] S503. Construct an index file based on storage pages and their corresponding addressing domains, as well as numerical metadata and model metadata.

[0168] In this embodiment of the application, the data query device can establish an addressing domain corresponding to each storage page and establish a connection with the numerical metadata and model metadata to form an index file.

[0169] For example, in combination Figure 10 . Figure 10 This paper demonstrates a file-level storage scheme where the entire file is stored across multiple storage pages in cloud object storage. Model metadata, numerical metadata, and addressing domains are loaded into memory during index file initialization. Each storage page stores a fixed number of dictionary values ​​and their location information, resulting in varying page sizes. This storage design in this embodiment converts the relative location information calculated from a pre-defined model into actual physical addresses. Combined with the aforementioned storage page design, it utilizes Hypertext Transfer Protocol (HTTP) range read operations (a method for accessing partial data in object storage) to provide an efficient index query method requiring only one storage medium interaction.

[0170] In this embodiment, following the order of the second-level columns, the dictionary values ​​and location information corresponding to M second-level columns are stored sequentially in each storage page, and the physical address storing the dictionary value and location information of each second-level column is obtained. An addressing field is established for each storage page; wherein the addressing field includes the physical address of the dictionary value and location information of the second-level column stored in the corresponding storage page; an index file is constructed based on the storage page and its corresponding addressing field, as well as numerical metadata and model metadata. Thus, by establishing storage pages for the relative positions of the dictionary values ​​and location information of the second-level columns in the index file, and addressing fields including physical addresses, the physical address of the target location information can be directly determined through the relative position information of the second-level column to be queried. When faced with multiple second-level columns to be queried, the target location information can be quickly queried through the physical address. Compared with the related technology's scheme of determining the logical value through multiple queries and then querying the location through the logical value, this reduces the number of interactions with cloud object storage, thereby improving query efficiency.

[0171] Please see Figure 11 The following is a flowchart illustrating the data query method provided in the embodiments of this application. Figure 8 , Figure 1 S103 in the above can also be implemented through S601 to S603, combining Figure 11 The steps shown are explained below:

[0172] S601. Based on the order of the relative position information representation and the number M of secondary columns stored in each storage page, determine the corresponding target storage page.

[0173] In this embodiment, the data query device can determine the target storage page corresponding to the dictionary value and location information of the secondary column to be queried by comparing the position represented by the relative position information with the number M of secondary columns stored in each storage page.

[0174] In this embodiment, the data query device can determine the starting and ending relative positions of the target storage page based on the ratio of the order of the relative position information representation to the number M of secondary columns stored in each storage page, and a preset error; and determine the target storage page based on the consistency detection result of the starting and ending relative positions. For example, M can be 2. The position of the relative position information representation can be 16, and the preset error can be 1, so 16 divided by M equals 8. Based on the preset error, the starting relative position can be 8, and the ending relative position can be 9.

[0175] Specifically, if the consistency check results indicate that the starting relative position and the ending relative position are the same, then the target storage page is determined to be the storage page corresponding to the starting relative position. If the consistency check results indicate that the starting relative position and the ending relative position are different, then the target storage page includes both the storage page corresponding to the starting relative position and the storage page corresponding to the ending relative position.

[0176] S602. Determine the target physical address in the target addressing domain corresponding to the target storage page.

[0177] In this embodiment, the data query device can determine the target addressing domains in the same order based on the order of the target storage pages. Then, within the target addressing domain, it determines the target physical address of the dictionary value and location information corresponding to the second-level column in the target storage page.

[0178] In this embodiment, the addressing field may include the dictionary value corresponding to the second-level column, the first byte address corresponding to the location information, and the corresponding physical address. The corresponding target physical address can be determined based on the first byte address of the dictionary value in the second-level column to be queried.

[0179] S603. Query the target location information based on the target physical address.

[0180] In this embodiment, the data query device can locate intermediate location information and intermediate dictionary values ​​in the target storage page based on the target physical address. It then performs a matching verification between the intermediate dictionary values ​​and the secondary column to be queried. If the verification is successful, the intermediate location information corresponding to the intermediate dictionary value is determined to be the target location information. The intermediate location information and the intermediate dictionary values ​​have a corresponding relationship, and the logarithm between them can be determined based on the number M of secondary columns stored in the storage page.

[0181] In this embodiment, intermediate location information and intermediate dictionary value are queried in the target storage page based on the target physical address; after verification based on the intermediate dictionary value, the intermediate location information is determined to be the target location information.

[0182] In this embodiment, the data query device obtains one or more intermediate dictionary values ​​and their corresponding intermediate location information from one or two target storage pages via an HTTP range request based on the target physical address. The data query device can determine a secondary column by reversing each intermediate dictionary value, and use this secondary column to match and verify with the secondary column to be queried. If the verification is successful, the intermediate location information corresponding to the intermediate dictionary value is determined to be the target location information.

[0183] In this embodiment, the corresponding target storage page is determined based on the order of the relative position information representation and the number M of secondary columns stored in each storage page; the target physical address is determined in the target addressing domain corresponding to the target storage page; and the target location information is queried based on the target physical address. In this way, when faced with multiple secondary columns to be queried, the target location information can be quickly queried through the physical address. Compared to the related technologies that determine logical values ​​through multiple queries and then query the location through those logical values, this reduces the number of interactions with the cloud object storage, thereby improving query efficiency.

[0184] Combination Figure 12 The data query device in this application embodiment may include: a secondary column preprocessing module 100, a variable-length string processing module 200, a machine learning module 300, and a machine learning-based precise indexing module 400 for cloud object storage.

[0185] Secondary column preprocessing module 100. This module deduplicates the original secondary columns to form secondary column groups, transforms them into dictionary values, and calculates the position information of each dictionary value's secondary column within the data file. Unlike traditional data warehouses, dictionary values ​​and position information do not need to be logically mapped one-to-one. The preprocessing module encodes and compresses both dictionary values ​​and position information.

[0186] Variable-length string processing module 200. This module is responsible for converting a set of dictionary values ​​of variable-length character types into numerical values ​​and outputting the numerical metadata of the strings. A dictionary value of a variable-length character type is then converted into a 64-bit integer based on the numerical metadata.

[0187] Machine Learning Module 300. This module performs machine learning on the cumulative distribution function of the numericalized dictionary values, outputting model metadata for the second-level column group. Based on the model metadata, the relative position of a dictionary value within the group of dictionary values ​​can be calculated.

[0188] The 400 is a machine learning-based precise indexing module for cloud object storage. This indexing module manages dictionary information, location information, numerical metadata, and learning model metadata in a unified manner. It converts the relative location information calculated from the learning model metadata into a physical address, ultimately retrieving the precise location of the second-level column value to be queried.

[0189] Please see Figure 13 The following is a flowchart illustrating the data query method provided in the embodiments of this application. Figure 9 , will combine Figure 13 The steps shown are explained below:

[0190] S31. Initialize the read index file and load the model metadata and addressing domain into memory.

[0191] In this embodiment of the application, the data query device can initialize the read index file and load the model metadata and addressing domain into memory.

[0192] S32. Is the second-level column to be queried a string?

[0193] In this embodiment, the data query device detects whether the secondary column to be queried is a string or a number. If it is a number, step S35 is executed. If it is a string, step S33 is executed.

[0194] S33. Load the numerical metadata into memory.

[0195] In this embodiment of the application, if the secondary column to be queried is a string, then the numerical metadata is loaded into memory.

[0196] S34. Use numerical metadata to quantify the second-level columns to be queried.

[0197] In this embodiment of the application, the dictionary value to be queried in the second-level column to be queried is converted into the corresponding dictionary value using numerical metadata.

[0198] S35. Calculate the relative position information of the second-level column to be queried using the preset model.

[0199] In this embodiment of the application, the data query device can use model metadata to calculate relative position information with an error range based on the values ​​of the dictionary to be queried, with O(sp) representing the starting relative position and O(ep) representing the ending relative position.

[0200] S36. Convert the relative location information into the target physical address.

[0201] In this embodiment, the relative position calculated based on the relative position information can be converted into the physical address corresponding to the target storage page. Assume each storage page stores M pairs of dictionary values ​​and their position information. The relative position is calculated as the logical address of the dictionary value and position information of the second-level column to be searched within the storage page. If the calculated O(sp) equals O(ep), then only one target storage page corresponding to O(sp) needs to be obtained; otherwise, both O(sp) and O(ep) need to be obtained. Then, combining the information in the addressing domain, the target physical address in the target addressing domain is found according to the order of the target storage pages.

[0202] S37. Request the target storage page based on the target physical address, and parse the intermediate dictionary value and intermediate location information in the target storage page.

[0203] In this embodiment of the application, one or two target storage pages can be obtained with a single HTTP range request based on the target physical address, and the intermediate dictionary values ​​and intermediate location information can be parsed out.

[0204] S38. Verify the value of the second-level column to be queried against the value of the intermediate dictionary. If the verification is successful, return the target location information.

[0205] In this embodiment of the application, the secondary column to be queried and the intermediate dictionary value are verified, and the intermediate position information corresponding to the intermediate dictionary value that passes the verification is determined to be the target position information.

[0206] In this embodiment, machine learning is applied to the deduplication, sorting, and numerical dictionary values ​​of the second-level columns, and the corresponding position information of the dictionary values ​​is indexed. Then, based on the machine learning, the relative position information of the numerical dictionary values ​​is calculated, the logical location of the target storage page containing the value of the column to be queried is calculated, and then, combined with the addressing domain information, the target storage page order is mapped to the target physical address. Access to the machine learning-based index data on cloud object storage is achieved through an HTTP range request. Thus, the query process for the second-level columns only requires one interaction with the storage medium. The data query method proposed in this embodiment improves index query performance and reduces user costs by reducing the number of interactions with cloud object storage. Therefore, this proposal possesses the following innovations.

[0207] Please see Figure 14 A schematic diagram of the structure of the data query device provided in the embodiments of this application. Figure 2 , will combine Figure 14 Explanation:

[0208] This application embodiment also provides a data query device 800, including: a determination unit 801 and a query unit 802.

[0209] The determining unit 801 is used to process the numericalized second-level column to be queried based on a preset model, and determine the relative position information of the second-level column to be queried in the corresponding second-level column group; wherein, the preset model is used to simulate the distribution of each second-level column in the numericalized second-level column group;

[0210] The query unit 802 is used to determine the target physical address of the second-level column to be queried in the index file based on the relative position information, and to query the target location information based on the target physical address; wherein, the index file is used to represent the correspondence between the dictionary value, the relative position of the location information stored, and the physical address corresponding to each second-level column.

[0211] In this embodiment of the application, the data query device 800 is used to determine the secondary column group in the data file, as well as the dictionary value and position information corresponding to each secondary column in the secondary column group;

[0212] If the second-level column is a string, then the numerical metadata is determined based on the dictionary value corresponding to each second-level column and the order relationship between each dictionary value, and the dictionary value corresponding to each dictionary value is determined based on the numerical metadata; wherein, the numerical metadata is used to keep the order between the determined dictionary values ​​consistent with the order between the dictionary values.

[0213] The model metadata of the preset model is determined based on the distribution of each dictionary value; whereby the model metadata is the data used to determine the preset model.

[0214] Store the dictionary value and location information corresponding to each second-level column in order, and obtain the physical address where the dictionary value and location information of each second-level column are stored;

[0215] An index file is constructed based on the dictionary value corresponding to each second-level column, the relative location of the location information storage, the corresponding physical address, as well as the numerical metadata and model metadata.

[0216] In this embodiment of the application, the data query device 800 is used to determine the character position length based on the maximum string length corresponding to each dictionary value, and to determine the number of different characters and character set at each character position corresponding to each dictionary value;

[0217] The cardinality of each character position is determined based on the right-to-left character position order and the number of distinct characters at each position; wherein, the cardinality of each character position is greater than the cardinality of the previous character position.

[0218] Based on the right-to-left character position order and the order relationship between each dictionary value, a corresponding numerical value is assigned to the character at each character position of each dictionary value; wherein, the numerical metadata includes: the character set at each character position, the cardinality of each character position, and the numerical value assigned to the character at each character position.

[0219] The dictionary value corresponding to each dictionary value is determined based on the cardinality of each character position in each dictionary value and the numerical value corresponding to the character at each character position.

[0220] In this embodiment of the application, the data query device 800 is used to determine the product of the number of different characters corresponding to the N character positions before each character position based on the character position order from right to left, which is the base of each character position; wherein, the base of the first character position is a first predetermined value; N is an integer greater than 0.

[0221] In this embodiment of the application, the data query device 800 is used to sort the characters in the character set at each character position according to the order relationship between each dictionary value;

[0222] For each character position in the sorted character set, the corresponding numerical value is assigned to the characters in the character set sequentially according to the allocation logic of increasing second predetermined value.

[0223] In this embodiment of the application, the data query device 800 is used to determine the corresponding dictionary value to be queried by the transformation processing of the secondary column to be queried.

[0224] The cardinality of each character position in the dictionary value to be queried is determined based on the numerical metadata, and the numerical value corresponding to the character at each character position is determined.

[0225] Multiply the cardinality of each character position in the dictionary value to be queried by the numerical value corresponding to the character at each character position, and then sum them up to determine the numerical value of the dictionary value after converting it to the numerical value of the second-level column to be queried.

[0226] In this embodiment of the application, the data query device 800 is used to store the dictionary values ​​and location information corresponding to M secondary columns in each storage page according to the order between the secondary columns, and to obtain the physical address of storing the dictionary value and location information of each secondary column; wherein, M is an integer greater than 0.

[0227] In this embodiment of the application, the data query device 800 is used to establish an addressing field corresponding to each storage page; wherein, the addressing field includes the dictionary value and physical address of the location information of the second-level column stored in the corresponding storage page;

[0228] An index file is constructed based on storage pages and their corresponding addressing domains, as well as numerical metadata and model metadata.

[0229] In this embodiment of the application, the query unit 802 in the data query device 800 is used to determine the corresponding target storage page based on the order represented by the relative position information and the number M of the secondary columns stored in each storage page;

[0230] Determine the target physical address in the target addressing domain corresponding to the target storage page;

[0231] Query target location information based on target physical address.

[0232] In this embodiment of the application, the query unit 802 in the data query device 800 is used to determine the starting relative position and ending relative position of the target storage page based on the ratio of the position represented by the relative position information to the number M of secondary columns stored in each storage page, and the preset error.

[0233] The target storage page is determined based on the consistency detection results of the starting and ending relative positions.

[0234] In this embodiment of the application, the query unit 802 in the data query device 800 is used to query intermediate location information and intermediate dictionary values ​​in the target storage page based on the target physical address;

[0235] After successful verification based on the intermediate dictionary value, the intermediate location information is determined to be the target location information.

[0236] It should be noted that, in the embodiments of this application, if the above-described data query method is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiments of this application, or the part that contributes to the related technology, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a data query device (which may be a personal computer, etc.) to execute all or part of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), magnetic disks, or optical disks. Thus, the embodiments of this application are not limited to any specific hardware and software combination.

[0237] Correspondingly, embodiments of this application provide a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps in the method on one side of the data query device 800.

[0238] It should be noted that the descriptions of the storage medium and device embodiments above are similar to the descriptions of the method embodiments above, and have similar beneficial effects. For technical details not disclosed in the storage medium and device embodiments of this application, please refer to the descriptions of the method embodiments of this application for understanding.

[0239] It should be noted that, Figure 15 A schematic diagram of a hardware entity of an electronic device provided in an embodiment of this application, such as... Figure 15As shown, this application embodiment provides an electronic device 900, including a memory 902 and a processor 901. The memory 902 stores a computer program that can run on the processor 901. When the processor 901 executes the program, it implements the steps in the above-described method, wherein;

[0240] Processor 901 typically controls the overall operation of electronic device 900.

[0241] The memory 902 is configured to store instructions and applications executable by the processor 901, and can also cache data to be processed or already processed (e.g., image data, audio data, voice communication data and video communication data) in the processor 901 and various modules in the electronic device 900. It can be implemented by flash memory or random access memory (RAM).

[0242] Correspondingly, this application embodiment also provides a computer program product, including a computer program that can be executed by the processor 901 of the electronic device 900 to complete the steps in the method on one side of the data query device 800.

[0243] It should be understood that the phrase "one embodiment" or "an embodiment" throughout the specification means that a specific feature, structure, or characteristic related to the embodiment is included in at least one embodiment of this application. Therefore, "in one embodiment" or "in an embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. Furthermore, these specific features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. It should be understood that in the various embodiments of this application, the sequence numbers of the above-described processes do not imply a sequential order of execution; the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application. The sequence numbers of the above-described embodiments are merely descriptive and do not represent the superiority or inferiority of the embodiments.

[0244] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0245] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods, such as: multiple units or components can be combined, or integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the various components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the apparatus or units can be electrical, mechanical, or other forms.

[0246] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units. They may be located in one place or distributed across multiple network units. Some or all of the units may be selected to achieve the purpose of this embodiment according to actual needs.

[0247] In addition, each functional unit in the various embodiments of this application can be integrated into one processing unit, or each unit can be a separate unit, or two or more units can be integrated into one unit; the integrated unit can be implemented in hardware or in the form of hardware plus software functional units.

[0248] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media that can store program code, such as mobile storage devices, read-only memory (ROM), magnetic disks, or optical disks.

[0249] Alternatively, if the integrated units described above are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, or the parts that contribute to related technologies, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, ROMs, magnetic disks, or optical disks.

[0250] The above are merely embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application.

Claims

1. A data query method, characterized in that, include: The query dictionary value of the second-level column to be queried is determined based on the numerical metadata processing to identify the corresponding query dictionary value. The query dictionary value is then processed based on model metadata of a preset model to determine the relative position information of the query second-level column within the corresponding second-level column group. The preset model is used to simulate the distribution of each second-level column in the numerically processed second-level column group. The model metadata includes the slope, intercept, and first interpolation of a first-order spline obtained from analyzing multiple dictionary value groups. The numerical metadata includes one or more numerical values ​​for each character position in the dictionary value. Based on the order of the relative position information representation and the number M of secondary columns stored in each storage page, the corresponding target storage page is determined. The target physical address is determined in the target addressing domain corresponding to the target storage page, and the target location information is queried based on the target physical address. The storage page belongs to an index file. The index file is used to represent the correspondence between the dictionary value corresponding to each secondary column, the relative position of the location information storage, and the physical address. M is an integer greater than 0.

2. The data query method according to claim 1, characterized in that, The method further includes: Determine the secondary column group in the data file, and the dictionary value and position information corresponding to each secondary column in the secondary column group; If the second-level column is a string, then numerical metadata is determined based on the dictionary value corresponding to each second-level column and the order relationship between each dictionary value, and the dictionary value corresponding to each dictionary value is determined based on the numerical metadata; wherein, the numerical metadata is used to keep the order between the determined dictionary values ​​consistent with the order between the dictionary values. The model metadata of the preset model is determined based on the distribution of each dictionary value; wherein, the model metadata is data used to determine the preset model; The dictionary value and location information corresponding to each of the secondary columns are stored in order, and the physical address storing the dictionary value and location information of each of the secondary columns is obtained. The index file is constructed based on the dictionary value corresponding to each of the secondary columns, the relative position of the location information storage, the corresponding physical address, the numerical metadata, and the model metadata.

3. The data query method according to claim 2, characterized in that, The step of determining numerical metadata based on the dictionary value corresponding to each of the secondary columns and the order relationship between each dictionary value, and determining the dictionary value corresponding to each dictionary value based on the numerical metadata, includes: The character position length is determined based on the maximum string length corresponding to each dictionary value, and the number of different characters and character set at each character position corresponding to each dictionary value are determined. The cardinality of each character position is determined based on the right-to-left character position order and the number of different characters at each character position; wherein the cardinality of each character position is greater than the cardinality of the previous character position; Based on the right-to-left character position order and the order relationship between each dictionary value, a corresponding numerical value is assigned to the character at each character position for each dictionary value; wherein, the numerical metadata includes: the character set at each character position, the cardinality at each character position, and the numerical value assigned to the character at each character position. The dictionary value corresponding to each dictionary value is determined based on the cardinality of each character position in each dictionary value and the numerical value corresponding to the character at each character position.

4. The data query method according to claim 3, characterized in that, The determination of the cardinality for each character position based on the right-to-left character position order and the number of distinct characters at each position includes: Based on the character position order from right to left, the product of the number of different characters corresponding to the N character positions preceding each character position is determined as the base of each character position; wherein, the base of the first character position is a first predetermined value; N is an integer greater than 0.

5. The data query method according to claim 3, characterized in that, The process of assigning a corresponding numerical value to the character at each character position in each dictionary value based on the right-to-left character position order and the order relationship between each dictionary value includes: Sort the characters in the character set at each character position according to the order relationship between each dictionary value; For each character position in the sorted character set, the corresponding numerical value is assigned to the characters in the character set sequentially according to the allocation logic of increasing second predetermined value.

6. The data query method according to claim 3, characterized in that, Before processing the numericalized secondary column to be queried based on the preset model, the method further includes: The corresponding dictionary value to be queried is determined by transforming and processing the second-level column to be queried; Based on the numerical metadata, determine the cardinality of each character position in the dictionary value to be queried, and the numerical value corresponding to the character at each character position; The cardinality of each character position in the dictionary value to be queried is multiplied by the numerical value corresponding to the character at each character position, and then summed to determine the numerical value of the dictionary value to be queried for the second-level column to be queried.

7. The data query method according to any one of claims 2 to 6, characterized in that, The step of storing the dictionary value and location information corresponding to each of the secondary columns in order, and obtaining the physical address storing the dictionary value and location information of each of the secondary columns, includes: According to the order among the secondary columns, the dictionary values ​​and location information corresponding to M secondary columns are stored sequentially in each storage page, and the physical address storing the dictionary value and location information of each secondary column is obtained.

8. The data query method according to claim 7, characterized in that, The construction of the index file based on the dictionary value corresponding to each of the secondary columns, the relative position of the stored location information, the corresponding physical address, the numerical metadata, and the model metadata includes: Establish an addressing field corresponding to each of the aforementioned storage pages; wherein, the addressing field includes the dictionary value of the secondary column and the physical address of the location information stored in the corresponding storage page; The index file is constructed based on the storage page and the corresponding addressing domain, as well as the numerical metadata and the model metadata.

9. The data query method according to claim 8, characterized in that, The determination of the corresponding target storage page based on the order represented by the relative position information and the number M of secondary columns stored in each storage page includes: The starting and ending relative positions of the target storage page are determined based on the ratio of the position represented by the relative position information to the number M of the secondary columns stored in each storage page, and a preset error. The target storage page is determined based on the consistency detection results of the starting relative position and the ending relative position.

10. The data query method according to claim 8, characterized in that, The process of querying the target location information based on the target physical address includes: Based on the target physical address, query the intermediate location information and intermediate dictionary value in the target storage page; After the verification based on the intermediate dictionary value is passed, the intermediate location information is determined to be the target location information.

11. A data query device, characterized in that, include: A determining unit is used to process the numerical metadata of the dictionary value to be queried for the second-level column to be queried, determine the corresponding dictionary value to be queried, and process the dictionary value to be queried based on the model metadata of a preset model to determine the relative position information of the second-level column to be queried in the corresponding second-level column group; wherein, the preset model is used to simulate the distribution of each second-level column in the numerical second-level column group; the model metadata includes the slope, intercept, and first interpolation of the first spline obtained from the analysis of multiple dictionary value groups; the numerical metadata includes one or more values ​​for each character position in the dictionary value; The query unit is used to determine the corresponding target storage page based on the order of the relative position information and the number M of secondary columns stored in each storage page, determine the target physical address in the target addressing domain corresponding to the target storage page, and query the target location information based on the target physical address; wherein, the storage page belongs to an index file; the index file is used to represent the correspondence between the dictionary value corresponding to each of the secondary columns, the relative position of the location information storage, and the physical address; M is an integer greater than 0.

12. An electronic device, characterized in that, It includes a memory and a processor, the memory storing a computer program that can run on the processor, the processor executing the computer program to implement the steps of the method according to any one of claims 1 to 10.

13. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 10.

14. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 10.

Images