Data loading method and apparatus

Abstract

The application provides a data loading method and device, the method comprising: obtaining an accurate query condition and a fuzzy query condition. Determining a first query code corresponding to the accurate query condition and a second query code corresponding to the fuzzy query condition, the first query code and the second query code being query codes corresponding to a preset database. Querying and processing in the preset database according to the first query code, and querying and processing in the preset database according to the second query code to obtain target data. Loading the target data to a preset flow table. The data query is supported through the accurate query condition and the fuzzy query condition, which improves the flow data query capability of the preset flow table in the preset database.

Description

Technical Field

[0001] This application relates to computer technology, and more particularly to a data loading method and apparatus. Background Technology

[0002] As a distributed data stream computing system, Flink Stream Tables can execute arbitrary data programs through data parallelism and pipelined processing. One of the important operations before using Flink Stream Tables for data processing is loading data from the data storage system HBase into the Flink Stream Tables.

[0003] In related technologies, Flink streaming tables typically perform queries in HBase based on exact query conditions and load the retrieved data into the Flink streaming table. This HBase is only compatible with exact query conditions; that is, it can only retrieve data corresponding to the exact query conditions from HBase.

[0004] However, existing technologies have limitations in loading streaming data from HBase into Flink streaming tables. This is because the streaming processing system Flink can only query streaming data in HBase using a single query method with precise query conditions, resulting in poor streaming data query capabilities. Summary of the Invention

[0005] This application provides a data loading method and apparatus to overcome the problem of limited data range loading.

[0006] Firstly, this application provides a data loading method applied to a server, comprising:

[0007] Get precise and fuzzy search criteria;

[0008] Determine the first query code corresponding to the precise query condition and the second query code corresponding to the fuzzy query condition, wherein the first query code and the second query code are query codes corresponding to a preset database;

[0009] The target data is obtained by performing a query in the preset database according to the first query code and the second query code;

[0010] The target data is loaded into a preset flow table.

[0011] In one possible design, determining the first query code corresponding to the precise query condition includes:

[0012] Determine the calculation expression corresponding to the precise query condition, and the calculation expression is used to determine the position of the data to be queried in the preset database;

[0013] The calculation expression is converted into code in a preset format to obtain the first query code.

[0014] In one possible design, determining the computational expression corresponding to the precise query condition includes:

[0015] Obtain the syntax tree corresponding to the exact query condition, wherein the syntax tree includes multiple function nodes, and the function nodes are elements in the exact query condition;

[0016] Obtain the type of each function node in the syntax tree;

[0017] The computation expression is determined from the plurality of function nodes based on the type of each function node in the syntax tree and the number of function parameters corresponding to each function node.

[0018] In one possible design, determining the computation expression from the plurality of function nodes based on the type of each function node in the syntax tree and the number of function parameters corresponding to each function node includes:

[0019] Traverse each function node in the syntax tree, and perform a push operation on the first type of function node and a pop operation on the second type of function node until the traversal of each function node in the syntax tree is completed. Then, determine the child function node of the top element in the first stack as the computation expression.

[0020] The push operation includes: placing the function node of the first type into the first stack, and placing the number of function parameters corresponding to the function node of the first type into the second stack;

[0021] The pop operation includes: obtaining a second quantity N from the top of the second stack, and popping N function nodes from the first stack, where N is an integer greater than or equal to 1.

[0022] In one possible design, converting the computation expression into code of a preset format to obtain the first query code includes:

[0023] Determine the conversion function for the Java code;

[0024] The calculation expression is transformed by the transformation function to obtain the first query code, which is Java code.

[0025] In one possible design, determining the second query code corresponding to the fuzzy query condition includes:

[0026] Based on the fuzzy query conditions, at least one column condition and query information corresponding to each column condition are generated. The query information includes column identifiers in the preset database and column identifiers in the preset flow table.

[0027] The second query code is generated based on the at least one column condition.

[0028] In one possible design, the step of performing query processing in the preset database according to the first query code and performing query processing in the preset database according to the second query code to obtain the target data includes:

[0029] The identifier of the target data is determined based on the first query code and the second query code;

[0030] If the identifier of the target data is not found in the server's cache, the target data is queried from the preset database.

[0031] In one possible design, the query processing based on the first query code in the preset database includes:

[0032] Based on the first query code, the identifiers of multiple data to be queried are determined, and the identifiers of the data to be queried are stored in the query queue. The query queue is also used to store the identifiers of other data to be queried.

[0033] The query is performed in the preset database based on the identifier of the data to be queried in the query queue.

[0034] In one possible design, the step of performing a query in the preset database based on the identifier of the data to be queried in the query queue includes:

[0035] If the number of identifiers of the data to be queried in the queue is greater than or equal to M, then M identifiers of the data to be queried are obtained from the queue, and a query is performed in the preset database based on the M identifiers of the data to be queried.

[0036] If the number of identifiers of the data to be queried in the queue is less than M, and the time difference between the current time and the last time the preset database was queried is greater than or equal to a first threshold, then all identifiers of the data to be queried are obtained from the queue, and the query is performed in the preset database according to the identifiers of all the data to be queried.

[0037] In one possible design, the query processing based on the second query code in the preset database includes:

[0038] Based on the second query code, determine the identifier of the data to be queried corresponding to the second query code in the preset database;

[0039] The query is performed in the preset database based on the identifier of the data to be queried.

[0040] In one possible design, the preset flow table is a Flink flow table, and the preset database is an HBase database.

[0041] Secondly, this application provides a data loading apparatus, comprising:

[0042] The acquisition module is used to obtain precise query conditions and fuzzy query conditions;

[0043] The determining module is used to determine the first query code corresponding to the precise query condition and the second query code corresponding to the fuzzy query condition, wherein the first query code and the second query code are query codes corresponding to a preset database;

[0044] The processing module is used to perform query processing in the preset database according to the first query code, and to perform query processing in the preset database according to the second query code, to obtain target data;

[0045] The storage module is used to load the target data into a preset flow table.

[0046] In one possible design, the determining module is specifically used for:

[0047] Determine the calculation expression corresponding to the precise query condition, and the calculation expression is used to determine the position of the data to be queried in the preset database;

[0048] The calculation expression is converted into code in a preset format to obtain the first query code.

[0049] In one possible design, the determining module is specifically used for:

[0050] Obtain the syntax tree corresponding to the exact query condition, wherein the syntax tree includes multiple function nodes, and the function nodes are elements in the exact query condition;

[0051] Obtain the type of each function node in the syntax tree;

[0052] The computation expression is determined from the plurality of function nodes based on the type of each function node in the syntax tree and the number of function parameters corresponding to each function node.

[0053] In one possible design, the determining module is specifically used for:

[0054] Traverse each function node in the syntax tree, and perform a push operation on the first type of function node and a pop operation on the second type of function node until the traversal of each function node in the syntax tree is completed. Then, determine the child function node of the top element in the first stack as the computation expression.

[0055] The push operation includes: placing the function node of the first type into the first stack, and placing the number of function parameters corresponding to the function node of the first type into the second stack;

[0056] The pop operation includes: obtaining a second quantity N from the top of the second stack, and popping N function nodes from the first stack, where N is an integer greater than or equal to 1.

[0057] In one possible design, the determining module is specifically used for:

[0058] Determine the conversion function for the Java code;

[0059] The calculation expression is transformed by the transformation function to obtain the first query code, which is Java code.

[0060] In one possible design, the determining module is specifically used for:

[0061] Based on the fuzzy query conditions, at least one column condition and query information corresponding to each column condition are generated. The query information includes column identifiers in the preset database and column identifiers in the preset flow table.

[0062] The second query code is generated based on the at least one column condition.

[0063] In one possible design, the processing module is specifically used for:

[0064] The identifier of the target data is determined based on the first query code and the second query code;

[0065] If the identifier of the target data is not found in the server's cache, the target data is queried from the preset database.

[0066] In one possible design, the processing module is specifically used for:

[0067] Based on the first query code, the identifiers of multiple data to be queried are determined, and the identifiers of the data to be queried are stored in the query queue. The query queue is also used to store the identifiers of other data to be queried.

[0068] The query is performed in the preset database based on the identifier of the data to be queried in the query queue.

[0069] In one possible design, the processing module is specifically used for:

[0070] If the number of identifiers of the data to be queried in the queue is greater than or equal to M, then M identifiers of the data to be queried are obtained from the queue, and a query is performed in the preset database based on the M identifiers of the data to be queried.

[0071] If the number of identifiers of the data to be queried in the queue is less than M, and the time difference between the current time and the last time the preset database was queried is greater than or equal to a first threshold, then all identifiers of the data to be queried are obtained from the queue, and the query is performed in the preset database according to the identifiers of all the data to be queried.

[0072] In one possible design, the processing module is specifically used for:

[0073] Based on the second query code, determine the identifier of the data to be queried corresponding to the second query code in the preset database;

[0074] The query is performed in the preset database based on the identifier of the data to be queried.

[0075] In one possible design, the preset flow table is a Flink flow table, and the preset database is an HBase database.

[0076] Thirdly, this application provides a data loading device, comprising:

[0077] Memory, used to store programs;

[0078] A processor for executing the program stored in the memory, wherein, when the program is executed, the processor is configured to perform the method described in the first aspect above and any of the various possible designs of the first aspect.

[0079] Fourthly, this application provides a computer-readable storage medium including instructions that, when executed on a computer, cause the computer to perform the methods described in the first aspect above and any of the various possible designs of the first aspect.

[0080] This application provides a data loading method and apparatus. The method includes: obtaining precise query conditions and fuzzy query conditions; determining a first query code corresponding to the precise query conditions and a second query code corresponding to the fuzzy query conditions, wherein the first query code and the second query code are query codes corresponding to a preset database; performing query processing in the preset database according to the first query code and the second query code to obtain target data; and loading the target data into a preset flow table. The method supports data querying using both precise and fuzzy query conditions, which improves the streaming data querying capability of the preset flow table in the preset database. Attached Figure Description

[0081] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0082] Figure 1 This is a schematic diagram of an application scenario provided by an embodiment of this application;

[0083] Figure 2 The flow of the data loading method provided in the embodiments of this application Figure 1 ;

[0084] Figure 3 The flow of the data loading method provided in the embodiments of this application Figure 2 ;

[0085] Figure 4 This is a schematic diagram of the structure of the data loading device provided in the embodiments of this application;

[0086] Figure 5 This is a schematic diagram of the hardware structure of the data loading device provided in an embodiment of this application. Detailed Implementation

[0087] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0088] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0089] To facilitate understanding of the technical solution of this application, the relevant concepts involved in this application will be introduced first:

[0090] Apache Flink (hereinafter referred to as Flink) is a big data processing framework with advantages such as integrated streaming and batch data processing, multi-level application programming interfaces (APIs), input / output consistency guarantees, flexible deployment, large-scale computing, low latency, and high throughput. Streaming data is a sequence of data arriving sequentially, in large quantities, rapidly, and continuously. Batch data refers to a collection of data containing a certain amount of data.

[0091] Apache HBase (hereinafter referred to as HBase) is a column-oriented, scalable, distributed storage system with advantages such as high reliability, high performance, and low latency. HBase technology allows for the construction of large-scale storage clusters on various servers to store massive amounts of data and provide high-speed data queries. HBase uses a key-value format to store data, where each row of data is uniquely identified by a key, typically called the row key.

[0092] The open-source parser (Another Tool for Language Recognition, ANTLR) is used to automatically generate and visualize syntax trees from input code. It provides a processing framework for automatically constructing custom languages ​​based on syntax descriptions for languages ​​including Java, C++, and C#.

[0093] The following section introduces the background technology, prior art, and problems existing in the prior art involved in this application:

[0094] In business application data processing, due to the large volume of data but low information density, there's a tendency to focus on processing critical business data. In practice, this critical business data is often collected via messaging frameworks like Kafka and transmitted to the real-time stream processing system Flink for real-time data analysis. However, some ancillary attributes of this critical business data are stored as data information in the distributed storage system HBase. Therefore, loading streaming data from HBase into Flink streaming tables is a crucial operation before using Flink for data analysis.

[0095] Currently, in existing technologies related to data loading, Flink typically performs queries in HBase using exact query conditions and loads the retrieved data stream into a Flink streaming table. This HBase is only compatible with exact query conditions; that is, it can only retrieve the data stream corresponding to the exact query conditions from HBase.

[0096] However, existing technologies have limitations in loading streaming data from HBase into Flink streaming tables. This is because the streaming processing system Flink can only query streaming data in HBase using a single query method with precise query conditions, resulting in poor streaming data query capabilities.

[0097] Based on the aforementioned problems, this application proposes the following technical concept: Since query conditions include both precise and fuzzy query conditions, the precise and fuzzy query conditions are parsed separately to generate Java code, thus obtaining the Java code corresponding to the precise and fuzzy query conditions. Based on these Java codes, query processing is performed in HBase, resulting in streaming data, which is then loaded into a Flink streaming table. This support for data querying using both precise and fuzzy query conditions enhances the Flink streaming system's ability to query streaming data in HBase.

[0098] Below, in conjunction with Figure 1 This paper introduces the application scenarios of the embodiments of this application.

[0099] Figure 1 This is a schematic diagram illustrating an application scenario provided by an embodiment of this application. Please refer to [link / reference]. Figure 1 Server 10 includes a preset flow table and a preset database.

[0100] The predefined flow table is a distributed data stream processing framework or system. For example, it could be Apache Flink (hereinafter referred to as Flink). Flink executes arbitrary streaming data programs in a data-parallel and pipelined manner.

[0101] The default database is a distributed data storage system. For example, it could be Apache HBase (hereinafter referred to as HBase). HBase is a highly reliable, high-performance, column-oriented, scalable distributed storage system.

[0102] In a server, before a pre-defined stream table processes data, it typically loads streaming data from an external system. For example, a pre-defined stream table might load streaming data from a pre-defined database. The process of loading streaming data from a pre-defined database involves the pre-defined stream table querying data in HBase based on query conditions. If the data is found in HBase, the data corresponding to the query conditions is loaded into the Flink stream table.

[0103] The technical solutions shown in this application will now be described in detail through specific embodiments. It should be noted that the following embodiments may exist independently or in combination with each other; identical or similar content will not be repeated in different embodiments.

[0104] Based on the technical concept described above, the following section combines... Figure 2 The data loading method provided in this application will be described in detail with specific embodiments. Figure 2 The flow of the data loading method provided in the embodiments of this application Figure 1 .

[0105] like Figure 2 As shown, the method includes:

[0106] S201. Obtain precise query conditions and fuzzy query conditions.

[0107] Before introducing how to obtain precise and fuzzy query conditions, we will first explain the configuration of the preset flow table to achieve connection to the preset database.

[0108] In this embodiment, configuration information, such as query information, required for connecting a preset flow table to a preset database is recorded using a database table, and a configuration page is designed for maintenance personnel. This allows for batch updates of multiple query records via database operation statements if changes are needed; furthermore, the configuration page provides detailed instructions to guide maintenance personnel through the configuration process and reduces the difficulty of connecting the preset flow table to the preset database. Additionally, the preset database connection password can be encrypted and stored in the preset flow table to prevent plaintext password leakage.

[0109] The following example, using Flink as the default flow table and HBase as the default database, illustrates the main configuration information for connecting Flink to HBase. See Table 1 for details.

[0110] Table 1

[0111]

[0112]

[0113] It's important to note that ZooKeeper is a distributed, open-source distributed application coordination service and a crucial component of HBase. ZooKeeper provides consistency services for HBase, offering functionalities such as configuration maintenance, domain name service, distributed synchronization, and group services.

[0114] Additionally, it should be noted that in this embodiment, to enable Flink to connect to different versions of HBase, the corresponding HBase client archive (JavaArchive, Jar) needs to be used and the Application Programming Interface (API) automatically adapted when connecting to different versions of HBase. This avoids issues such as API incompatibility, data parsing errors, and unstable connections. Specifically, when submitting a Flink task, by configuring the Flink runtime environment, the Flink framework can automatically load the corresponding HBase version's Jar.

[0115] The following section explains the relevant precise and fuzzy search criteria.

[0116] In this embodiment, the goal is to load streaming data from a preset database into a preset stream table. After the preset stream table establishes a connection with the preset database, the preset stream table sends query conditions to the preset database. Data is then queried in the preset database based on these query conditions.

[0117] Below, using HBase as the default database, we'll first introduce HBase: In HBase, a row key is typically used to represent a row of data. The quality of the row key design directly determines HBase's read and write performance. Data in HBase is globally sorted according to the ASCII dictionary order of the row keys. When sorting row keys, the first byte of two row keys is compared. If they are different, the two first bytes are sorted according to ASCII dictionary order, and the row key corresponding to the first sorted byte is placed first. If they are the same, the second byte is compared, and so on... When comparing the Xth byte, the length of one of the row keys has been exceeded, and the shorter row key is placed first. For example, if there are 5 row keys: "012", "0", "123", "234", "3", the result after sorting by ASCII dictionary is: "0", "012", "123", "234", "3".

[0118] Generally, data is retrieved in HBase based on query conditions. There are two types of query conditions in HBase: exact query conditions and fuzzy query conditions. Specifically, the syntax for an exact query condition is `rowkey = `rowkeyExpr`, and the syntax for a fuzzy query condition is `rowkey like `rowkeyExpr` [and otherConditions]`. Here, `rowkeyExpr` is the expression used to calculate `rowkey`, and it supports common string operations such as string concatenation, reversal, and hashing. `otherConditions` refers to query conditions other than `rowkey` (hereinafter referred to as non-rowkey query conditions). Examples of non-rowkey query conditions include: `sex = 'male' and age = 20`.

[0119] The following describes one possible implementation method for obtaining precise and fuzzy query conditions.

[0120] In one possible implementation, the query conditions are obtained from the configuration information of the Flink connection to HBase. Based on the syntax of precise and fuzzy query conditions, it is determined whether the query conditions in the configuration information are precise or fuzzy. If the query conditions in the configuration information are precise, then the precise query conditions are obtained; if the query conditions in the configuration information are fuzzy, then the fuzzy query conditions are obtained.

[0121] S202. Determine the first query code corresponding to the precise query condition and the second query code corresponding to the fuzzy query condition. The first query code and the second query code are query codes corresponding to the preset database.

[0122] The system retrieves data from a preset database based on the query code. The first query code corresponds to exact search criteria, while the second query code corresponds to fuzzy search criteria.

[0123] It's important to note that HBase frequently uses methods like reversal, concatenation, and hashing to avoid data hotspots. Therefore, the `rowkeyExpr` in the query conditions supports common string manipulation functions such as `reverse`, `substring`, and `concat`. `reverse` reverses the string, `substring` concatenates multiple strings, and `concat` joins multiple strings into a single string. To implement these functionalities, a function parsing class is created for each function supported by `rowkeyExpr`. These parsing classes inherit a functional interface containing two methods. One method sets the function parameters, and the other generates Java code. For example, these two methods could be `setChildren` and `codeGen`, where `setChildren` sets the function parameters and `codeGen` generates the Java code.

[0124] In this embodiment, a first query code corresponding to the precise query condition is determined based on the precise query condition, and a second query code corresponding to the fuzzy query condition is determined based on the fuzzy query condition. The first query code and the second query code are query codes corresponding to a preset database.

[0125] It should be noted that precise query conditions only include rowkey query conditions, while fuzzy query conditions include both rowkey query conditions and non-rowkey query conditions.

[0126] Therefore, in the process of obtaining the first query code based on the precise query conditions, it is only necessary to parse the rowkeyExpr corresponding to the rowkey, and the code obtained after parsing is the first query code; in the process of obtaining the second query code based on the fuzzy query conditions, it is necessary to parse the rowkey query conditions and the non-rowkey query conditions in the fuzzy query conditions respectively. The query code obtained after parsing the rowkey query conditions and the query code obtained after parsing the non-rowkey query conditions together form the second query code.

[0127] Next, we will provide an example of one possible implementation of the query code that determines the rowkeyExpr based on the rowkeyExpr.

[0128] In one possible implementation, a standard Abstract Syntax Tree (AST) is generated based on the ANTLR parsing syntax definition file and rowkeyExpr. The ANTLR parsing syntax definition file describes the structural composition of various syntactic components of the language. The generated AST includes multiple functions, which may or may not contain subfunctions. Each subfunction includes an entry function node and at least one exit function node. Next, two empty stacks are created, for example, stack1 and stack2. Stack1 stores functions, and stack2 stores the number of subfunctions corresponding to each function. Then, the AST is traversed sequentially. If an entry function node is traversed, the function is first pushed onto stack1, and then the number of its subfunctions is pushed onto stack2. If an exit function node is traversed, stack2 is popped to obtain the number n subfunctions corresponding to the function. Then, n functions are popped from stack1. It is then determined whether all function nodes in the AST have been traversed; if not, the AST traversal continues. If the traversal is complete, the top function of stack1 is retrieved; this function is the expression function of rowkeyExpr. Next, the conversion method of the top function is called to obtain the query code corresponding to rowkeyExpr. The conversion method converts the top function into the corresponding query code. For example, the conversion method could be the codeGen method, and the query code type could be Java code.

[0129] Next, an example of one possible implementation of the query code that determines the query condition corresponding to the non-rowkey query condition will be provided.

[0130] In one possible implementation, a standard Abstract Syntax Tree (AST) is generated based on the ANTLR parsing syntax definition file and non-rowkey query conditions. A list `List` is created, and then the AST nodes are traversed. If a non-rowkey query condition node is encountered, a corresponding parsing object is constructed. This parsing object includes the preset database column name, comparison condition, and Flink column name, and is added to the list `List`. The format of the parsing object corresponding to a non-rowkey query condition can be, for example, `stream.presetstreamtablecolumnname comparisoncondition presetdatabasecolumnname`. For example, a non-rowkey query condition could be `stream.name1 = name`, where `name1` is the presetstream table column name, `name` is the preset database column name, and "=" is the comparison condition. It is then determined whether all non-rowkey query condition nodes in the AST have been traversed. If not, the AST traversal continues. If the traversal is complete, the query code for each non-rowkey query condition is generated based on the list `List`. Specifically, it iterates through each parsed object in the List, calls the corresponding transformation method for each parsed object, and obtains the query code corresponding to each parsed object based on the transformation method, that is, obtains the query code corresponding to each non-Rowkey query condition. For example, the transformation method is codeGen, which is used to generate Java code.

[0131] S203. Perform query processing in the preset database according to the first query code and the second query code to obtain the target data.

[0132] Next, an example of a possible implementation of query processing in a preset database based on the first query code will be provided.

[0133] In one possible implementation, a query identifier corresponding to the first query code in a preset database is calculated based on the first query code, and each query identifier obtained from the query is added to a query queue. Then, data query processing is performed in the preset database based on the query identifiers in the query queue, and the data obtained after the data query processing is the target data. For example, the rowkey corresponding to the first query code in HBase is calculated based on the first query code, and each rowkey is stored in a query queue. Then, data query processing is performed in HBase based on the rowkeys in the query queue, and the data obtained after the data query processing is the target data.

[0134] Next, an example of a possible implementation of query processing in a preset database based on the second query code will be provided.

[0135] In one possible implementation, based on the second query code, the corresponding query information in a preset database is calculated. Then, based on the query information, the data corresponding to the query information, i.e., the target data, is retrieved from the preset database. For example, based on the second query code, the rowkey prefix and non-rowkey query conditions corresponding to the second query code are retrieved from HBase. The query return columns are set according to the HBase dimension list in the configuration table; based on the rowkey prefix, non-rowkey query conditions, and the HBase dimension list, the data obtained by querying HBase using a Scan function is the target data.

[0136] S204. Load the target data into the preset flow table.

[0137] After obtaining the target data based on the above step S203, the target data is loaded into the preset flow table.

[0138] The data loading method provided in this application includes: obtaining precise query conditions and fuzzy query conditions; determining a first query code corresponding to the precise query conditions and a second query code corresponding to the fuzzy query conditions, wherein the first query code and the second query code are query codes corresponding to a preset database; performing query processing in the preset database according to the first query code and the second query code to obtain target data; and loading the target data into a preset flow table. The method allows for both precise and fuzzy queries, making the data loading method unrestricted by the query method and improving data query capabilities.

[0139] Based on the above embodiments, the data loading method provided in this application will be further described below with reference to a specific embodiment. Figure 3 To introduce, Figure 3 The flow of the data loading method provided in the embodiments of this application Figure 2 .

[0140] like Figure 3 As shown, the method includes:

[0141] S301. Obtain precise query conditions and fuzzy query conditions.

[0142] The specific implementation of step S301 is similar to that of step S201, and will not be described in detail here.

[0143] S302. Obtain the syntax tree corresponding to the exact query conditions. The syntax tree includes multiple function nodes, and the function nodes are the elements in the exact query conditions.

[0144] In an exact query, the expression for rowkey is rowkeyExpr.

[0145] In this embodiment, a standard abstract syntax tree is generated based on the ANTLR parsing syntax definition file and rowkeyExpr. The ANTLR parsing syntax definition file describes the structural composition of various syntactic components of the language.

[0146] The syntax tree corresponding to the exact query conditions includes multiple function nodes. Each function node is a key element in the exact query conditions.

[0147] S303. Obtain the type of each function node in the syntax tree.

[0148] S304. Traverse each function node in the syntax tree, and perform a push operation on the first type of function node and a pop operation on the second type of function node until the traversal of each function node in the syntax tree is completed. Then, determine the child function node of the top element in the first stack as the computation expression.

[0149] Next, steps S303 and S304 will be explained together.

[0150] In this embodiment, the node type of each function node in the syntax tree is obtained, wherein the node type of the function node includes a first type and a second type. Specifically, when the function node type is the first type, a push operation is performed on the function node, and a pop operation is performed on the corresponding function node of the second type.

[0151] In this embodiment, each function node in the syntax tree is traversed sequentially. When a function node of the first type is encountered, it is pushed onto the stack; when a function node of the second type is encountered, it is popped from the stack.

[0152] After traversing each function node of the syntax tree, the child function node of the top element in the first stack is determined as the expression to be evaluated.

[0153] The push operation includes: pushing the function node of the first type into the first stack, and pushing the number of function parameters corresponding to the function node of the first type into the second stack.

[0154] The pop operation includes: obtaining the second quantity N from the top of the second stack, and popping N function nodes from the first stack, where N is an integer greater than or equal to 1.

[0155] S305. Determine the conversion function for the Java code.

[0156] S306. The calculation expression is transformed using a transformation function to obtain the first query code, which is Java code.

[0157] Next, steps S305 and S306 will be explained together.

[0158] In this embodiment, a Java code conversion function is first determined. This conversion function is used to convert the computation expression into query code. For example, the conversion function could be `codeGen`. After determining the conversion function, the computation expression is converted using this function to obtain the converted first query code, where the first query code is the Java code corresponding to the computation expression.

[0159] S307. Based on the fuzzy query conditions, generate at least one column condition and query information corresponding to each column condition. The query information includes column identifiers in the preset database and column identifiers in the preset flow table.

[0160] In this implementation, when the query condition is a fuzzy query condition, the following describes a possible implementation method for generating query information based on the fuzzy query condition. The query information includes column identifiers from a preset database and column identifiers from a preset flow table.

[0161] It should be noted that fuzzy search conditions include rowkeyExpr and non-rowkey search conditions.

[0162] The specific implementation method for generating the Java code corresponding to rowkeyExpr based on the conversion function and rowkeyExpr is similar to that of steps S302 to S306, and will not be repeated here.

[0163] In this embodiment, a standard abstract syntax tree is generated based on the ANTLR parsing syntax definition file and non-rowkey query conditions. The ANTLR parsing syntax definition file describes the structural composition of various syntactic components of the language.

[0164] Next, we will explain one possible implementation method for generating query information based on the syntax tree corresponding to non-rowkey query conditions.

[0165] In one possible implementation, a list `List1` is created. Next, the syntax tree corresponding to non-rowkey query conditions is traversed sequentially. When a node is encountered that is a non-rowkey query condition node, at least one column condition and its corresponding query information are generated. For example, when traversing the syntax tree corresponding to non-rowkey query conditions, a parsing object is constructed for each node. This parsing object includes the preset database column name, comparison conditions, and Flink column name. This parsing object is then added to the list `List`. The format of the parsing object corresponding to a non-rowkey query condition can be, for example, `stream.presetstreamtablecolumnname comparisoncondition presetdatabasecolumnname`. For example, a non-rowkey query condition could be `stream.name1 = name`, where `name1` is the presetstream table column name, `name` is the preset database column name, and "=" represents the comparison condition.

[0166] S308. Generate a second query code based on at least one column condition.

[0167] The specific implementation of step S308 is similar to that of step S307, and will not be described in detail here.

[0168] S309. Determine the identifier of the target data based on the first query code and the second query code.

[0169] In this embodiment, we will first describe one possible implementation of determining the identifier of the target data based on the first query code and the second query code.

[0170] In one possible implementation, the identifier corresponding to the target data to be queried is calculated based on the first query code and the second query code. The identifier can be an HBase rowkey. For example, if the query condition is rowkey likestream.name and age = stream.age, a second query code is generated based on this query condition, and the corresponding stream table data is determined based on the generated second query code. For instance, if the second query code determines the corresponding stream table data as name = zhangsan and age = 10, then the rowkey corresponding to the query condition is [zhangsan, 10].

[0171] S310. If the identifier of the target data does not exist in the server's cache, then query the target data in the preset database.

[0172] In this embodiment, based on the identifier of the target data determined by the first query code and the second query code, it is determined whether the identifier of the target data exists in the server's cache.

[0173] If the identifier of the target data exists in the server's cache, then the target data is retrieved from the server's cache based on the identifier of the target data.

[0174] If the identifier of the target data does not exist in the server's cache, the target data will be queried from the preset database based on the identifier of the target data.

[0175] Next, if the identifier of the target data does not exist in the server's cache, the specific method for querying the target data in the preset database based on the identifier of the target data can be found in steps S311-S315.

[0176] Specifically, steps S311-S313 are the specific implementation methods for querying target data according to the first query code, and steps S314-S315 are the specific implementation methods for querying target data according to the second query code.

[0177] 311. Based on the first query code, determine the identifiers of multiple data to be queried, and store the identifiers of the data to be queried in the query queue. The query queue is also used to store the identifiers of other data to be queried.

[0178] Based on the above steps S302 to S306, the first query code corresponding to the precise query condition is obtained. Next, in this embodiment, according to the first query code, the identifiers of multiple data to be queried are determined, and the identifiers of the data to be queried are stored in the query queue. The query queue is also used to store the identifiers of other data to be queried.

[0179] For example, the HBase rowkey to be queried is calculated based on the first query code and stored in the queue to be queried. The system maintains a separate daemon thread to consume queue data and schedules HBase query tasks.

[0180] S312. If the number of identifiers of the data to be queried in the queue is greater than or equal to M, then obtain M identifiers of the data to be queried from the queue and perform query processing in the preset database based on the M identifiers of the data to be queried.

[0181] Where M is a positive integer greater than or equal to 1.

[0182] In one possible implementation, the identifiers of the data to be queried are processed based on the number of identifiers in the queue. Specifically, if the number of identifiers of the data to be queried in the queue is less than M, then the identifiers of those M data to be queried are not processed for the time being; if the number of identifiers of the data to be queried in the queue is greater than or equal to M, then the identifiers of those M data to be queried are processed. This helps to improve the system's query throughput and query efficiency.

[0183] The query process is performed in the preset database based on the identifiers of the M data to be queried.

[0184] The value of M can be set according to requirements, or the size of M can be determined according to the maximum asynchronous concurrency of the system, for example, you can refer to Formula 1.

[0185]

[0186] Here, parseInt() is the integer function, poolSize is the maximum asynchronous concurrency of the system, and β is a numerical value representing the size of the concurrency.

[0187] S313. If the number of identifiers of the data to be queried in the queue is less than M, and the time difference between the current time and the last time of querying the preset database is greater than or equal to the first threshold, then obtain all the identifiers of the data to be queried in the queue, and perform query processing in the preset database based on all the identifiers of the data to be queried.

[0188] The first threshold is a numerical value representing time.

[0189] In one possible implementation, the identifiers of the data to be queried are used for query processing based on the number of identifiers of the data to be queried in the query queue and the time difference between the current time and the last query time to the preset database. Specifically, when the number of identifiers of the data to be queried in the query queue is less than M, and the time difference between the current time and the last query time to the preset database is greater than or equal to a first threshold, then all identifiers of the data to be queried are obtained from the query queue, and query processing is performed in the preset database based on all identifiers of the data to be queried.

[0190] The size of the first threshold can be set according to requirements.

[0191] S314. Based on the second query code, determine the identifier of the data to be queried corresponding to the second query code in the preset database.

[0192] In this embodiment, the identifier in the preset database is calculated based on the second query code corresponding to the fuzzy query condition. For example, based on the second query condition, the rowkey prefix corresponding to the second query condition in HBase is calculated, and the non-rowkey query condition corresponding to the second query condition in HBase is also calculated. Furthermore, the queried data is filtered according to pre-configured preset return columns for HBase queries.

[0193] S315. Based on the identifier of the data to be queried, perform a query in the preset database.

[0194] In one possible implementation, based on the identifier of the data to be queried, a scan function is used to query data in a preset database, and the target data is then retrieved from a preset data repository. Here, the scan function is a query method used to retrieve data from the preset database.

[0195] S316. Load the target data into the preset flow table.

[0196] In one possible implementation, the target data is parsed and processed, such as compressed, and saved in a preset data format. This saved data is then loaded into a field of a preset flow table. For example, taking HBase as the preset database, HBase generally supports two main categories of data formats: multi-column HBase data formats and single-column HBase data formats. Single-column data formats include CSV, Avro, and Protobuf, among others. The format for saving the target data can be selected according to requirements and is not restricted. Using Flink as the preset flow table and CSV as the saved data format as an example, the target data in HBase is compressed, the processed data is saved in CSV format, and the saved data is loaded into a field of a Flink flow table.

[0197] The data loading method provided in this application includes: obtaining precise query conditions and fuzzy query conditions. The method involves: obtaining a syntax tree corresponding to the precise query conditions, where the syntax tree includes multiple function nodes, and each function node is an element in the precise query conditions; obtaining the type of each function node in the syntax tree; traversing each function node in the syntax tree, performing a push operation on function nodes of the first type and a pop operation on function nodes of the second type, until the traversal of all function nodes in the syntax tree is complete, and determining the child function node of the top element in the first stack as the computation expression; determining a conversion function for Java code; converting the computation expression using the conversion function to obtain a first query code, which is Java code; generating at least one column condition and query information corresponding to each column condition based on the fuzzy query conditions, where the query information includes column identifiers in a preset database and column identifiers in a preset flow table; generating a second query code based on at least one column condition; if the identifier of the target data does not exist in the server's cache, querying the target data in the preset database; determining the identifiers of multiple data to be queried based on the first query code, and storing the identifiers of the data to be queried in a query queue, which is also used to store identifiers of other data to be queried. If the number of identifiers for the data to be queried in the query queue is greater than or equal to M, then M identifiers for the data to be queried are retrieved from the query queue, and a query is performed in the preset database based on these M identifiers. If the number of identifiers for the data to be queried in the query queue is less than M, and the time difference between the current time and the last query time in the preset database is greater than or equal to a first threshold, then all identifiers for the data to be queried are retrieved from the query queue, and a query is performed in the preset database based on all identifiers. The identifier for the data to be queried corresponding to the second query code in the preset database is determined. A query is performed in the preset database based on the identifier of the data to be queried. The target data is loaded into the preset flow table.

[0198] Figure 4 This is a schematic diagram of the structure of the data loading device provided in an embodiment of this application. Figure 4 As shown, the device 400 includes: an acquisition module 401, a determination module 402, a processing module 403, and a storage module 404.

[0199] Module 401 is used to obtain precise query conditions and fuzzy query conditions;

[0200] The determining module 402 is used to determine the first query code corresponding to the precise query condition and the second query code corresponding to the fuzzy query condition, wherein the first query code and the second query code are query codes corresponding to a preset database;

[0201] The processing module 403 is used to perform query processing in the preset database according to the first query code and to perform query processing in the preset database according to the second query code to obtain target data;

[0202] Storage module 404 is used to load the target data into a preset flow table.

[0203] In one possible design, the determining module 402 is specifically used for:

[0204] Determine the calculation expression corresponding to the precise query condition, and the calculation expression is used to determine the position of the data to be queried in the preset database;

[0205] The calculation expression is converted into code in a preset format to obtain the first query code.

[0206] In one possible design, the determining module 402 is specifically used for:

[0207] Obtain the syntax tree corresponding to the exact query condition, wherein the syntax tree includes multiple function nodes, and the function nodes are elements in the exact query condition;

[0208] Obtain the type of each function node in the syntax tree;

[0209] The computation expression is determined from the plurality of function nodes based on the type of each function node in the syntax tree and the number of function parameters corresponding to each function node.

[0210] In one possible design, the determining module 402 is specifically used for:

[0211] Traverse each function node in the syntax tree, and perform a push operation on the first type of function node and a pop operation on the second type of function node until the traversal of each function node in the syntax tree is completed. Then, determine the child function node of the top element in the first stack as the computation expression.

[0212] The push operation includes: placing the function node of the first type into the first stack, and placing the number of function parameters corresponding to the function node of the first type into the second stack;

[0213] The pop operation includes: obtaining a second quantity N from the top of the second stack, and popping N function nodes from the first stack, where N is an integer greater than or equal to 1.

[0214] In one possible design, the determining module 402 is specifically used for:

[0215] Determine the conversion function for the Java code;

[0216] The calculation expression is transformed by the transformation function to obtain the first query code, which is Java code.

[0217] In one possible design, the determining module 402 is specifically used for:

[0218] Based on the fuzzy query conditions, at least one column condition and query information corresponding to each column condition are generated. The query information includes column identifiers in the preset database and column identifiers in the preset flow table.

[0219] The second query code is generated based on the at least one column condition.

[0220] In one possible design, the processing module 403 is specifically used for:

[0221] The identifier of the target data is determined based on the first query code and the second query code;

[0222] If the identifier of the target data is not found in the server's cache, the target data is queried from the preset database.

[0223] In one possible design, the processing module 403 is specifically used for:

[0224] Based on the first query code, the identifiers of multiple data to be queried are determined, and the identifiers of the data to be queried are stored in the query queue. The query queue is also used to store the identifiers of other data to be queried.

[0225] The query is performed in the preset database based on the identifier of the data to be queried in the query queue.

[0226] In one possible design, the processing module 403 is specifically used for:

[0227] If the number of identifiers of the data to be queried in the queue is greater than or equal to M, then M identifiers of the data to be queried are obtained from the queue, and a query is performed in the preset database based on the M identifiers of the data to be queried.

[0228] If the number of identifiers of the data to be queried in the queue is less than M, and the time difference between the current time and the last time the preset database was queried is greater than or equal to a first threshold, then all identifiers of the data to be queried are obtained from the queue, and the query is performed in the preset database according to the identifiers of all the data to be queried.

[0229] In one possible design, the processing module 403 is specifically used for:

[0230] Based on the second query code, determine the identifier of the data to be queried corresponding to the second query code in the preset database;

[0231] The query is performed in the preset database based on the identifier of the data to be queried.

[0232] In one possible design, the preset flow table is a Flink flow table, and the preset database is an HBase database.

[0233] The apparatus provided in this embodiment can be used to execute the technical solutions of the above method embodiments. Its implementation principle and technical effects are similar, and will not be described again here.

[0234] Figure 5 This is a schematic diagram of the hardware structure of the data loading device provided in the embodiments of this application, such as... Figure 5 As shown, the data loading device 500 of this embodiment includes: a processor 501 and a memory 502; wherein

[0235] Memory 502 is used to store instructions executed by the computer;

[0236] The processor 501 is used to execute computer execution instructions stored in the memory to implement the various steps of the data loading method in the above embodiments. For details, please refer to the relevant descriptions in the foregoing method embodiments.

[0237] Alternatively, the memory 502 can be either standalone or integrated with the processor 501.

[0238] When the memory 502 is set up independently, the data loading device also includes a bus 503 for connecting the memory 502 and the processor 501.

[0239] This application provides a computer-readable storage medium storing computer-executable instructions. When a processor executes the computer-executable instructions, it implements the data loading method executed by the data loading device described above.

[0240] This application also provides a computer program product, which includes a computer program stored in a readable storage medium. At least one processor of an electronic device can read the computer program from the readable storage medium, and the at least one processor executes the computer program to cause the electronic device to perform the solution provided in any of the above embodiments.

[0241] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices, or modules, and may be electrical, mechanical, or other forms.

[0242] The integrated modules implemented as software functional modules described above can be stored in a computer-readable storage medium. These software functional modules, stored in a storage medium, include several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute some steps of the methods described in the various embodiments of this application.

[0243] It should be understood that the aforementioned processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. A general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this invention can be directly manifested as execution by a hardware processor, or execution by a combination of hardware and software modules within the processor.

[0244] The memory may include high-speed RAM, and may also include non-volatile storage (NVM), such as at least one disk storage device, and may also be a USB flash drive, external hard drive, read-only memory, disk or optical disc, etc.

[0245] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, the buses shown in the accompanying drawings are not limited to a single bus or a single type of bus.

[0246] The aforementioned storage medium can be implemented from any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The storage medium can be any available medium accessible to general-purpose or special-purpose computers.

[0247] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0248] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

Claims

1. A data loading method, characterized in that, Applied to a server, the method includes: Obtain the query conditions; the query conditions include string manipulation functions; if the syntax of the query conditions is the syntax of an exact query condition, then it is determined to be an exact query condition; if the syntax of the query conditions is the syntax of a fuzzy query condition, then it is determined to be a fuzzy query condition. Determine the calculation expression corresponding to the precise query condition, the calculation expression is used to determine the position of the data to be queried in a preset database; convert the calculation expression into code of a preset format to obtain a first query code; and determine the second query code corresponding to the fuzzy query condition, the first query code and the second query code being Java query code corresponding to the preset database; The target data is obtained by performing a query in the preset database according to the first query code and the second query code; The target data is loaded into a preset flow table.

2. The method according to claim 1, characterized in that, Determining the calculation expression corresponding to the precise query condition includes: Obtain the syntax tree corresponding to the exact query condition, wherein the syntax tree includes multiple function nodes, and the function nodes are elements in the exact query condition; Obtain the type of each function node in the syntax tree; The computation expression is determined from the plurality of function nodes based on the type of each function node in the syntax tree and the number of function parameters corresponding to each function node.

3. The method according to claim 2, characterized in that, The step of determining the computation expression from the plurality of function nodes based on the type of each function node in the syntax tree and the number of function parameters corresponding to each function node includes: Traverse each function node in the syntax tree, and perform a push operation on the first type of function node and a pop operation on the second type of function node until the traversal of each function node in the syntax tree is completed. Then, determine the child function node of the top element in the first stack as the computation expression. The push operation includes: placing the function node of the first type into the first stack, and placing the number of function parameters corresponding to the function node of the first type into the second stack; The pop operation includes: obtaining a second quantity N from the top of the second stack, and popping N function nodes from the first stack, where N is an integer greater than or equal to 1.

4. The method according to any one of claims 1-3, characterized in that, The step of converting the calculation expression into code of a preset format to obtain the first query code includes: Determine the conversion function for the Java code; The calculation expression is transformed by the transformation function to obtain the first query code, which is Java code.

5. The method according to claim 1, characterized in that, Determining the second query code corresponding to the fuzzy query condition includes: Based on the fuzzy query conditions, at least one column condition and query information corresponding to each column condition are generated. The query information includes column identifiers in the preset database and column identifiers in the preset flow table. The second query code is generated based on the at least one column condition.

6. The method according to any one of claims 1-3, characterized in that, The step of performing query processing in the preset database according to the first query code and query processing in the preset database according to the second query code to obtain target data includes: The identifier of the target data is determined based on the first query code and the second query code; If the identifier of the target data is not found in the server's cache, the target data is queried from the preset database.

7. The method according to any one of claims 1-3, characterized in that, The query processing based on the first query code in the preset database includes: Based on the first query code, the identifiers of multiple data to be queried are determined, and the identifiers of the data to be queried are stored in the query queue. The query queue is also used to store the identifiers of other data to be queried. The query is performed in the preset database based on the identifier of the data to be queried in the query queue.

8. The method according to claim 7, characterized in that, The step of performing a query in the preset database based on the identifier of the data to be queried in the query queue includes: If the number of identifiers of the data to be queried in the queue is greater than or equal to M, then M identifiers of the data to be queried are obtained from the queue, and a query is performed in the preset database based on the M identifiers of the data to be queried. If the number of identifiers of the data to be queried in the queue is less than M, and the time difference between the current time and the last time the preset database was queried is greater than or equal to a first threshold, then all identifiers of the data to be queried are obtained from the queue, and the query is performed in the preset database according to the identifiers of all the data to be queried.

9. The method according to any one of claims 1-3, characterized in that, The step of performing a query in the preset database according to the second query code includes: Based on the second query code, determine the identifier of the data to be queried corresponding to the second query code in the preset database; The query is performed in the preset database based on the identifier of the data to be queried.

10. The method according to any one of claims 1-3, characterized in that, The preset flow table is a Flink flow table, and the preset database is an HBase database.

11. A data loading device, characterized in that, Applied to a server, the device includes: The acquisition module is used to acquire query conditions; the query conditions include string manipulation functions; if the syntax of the query conditions is the syntax of exact query conditions, then it is determined to be an exact query condition; if the syntax of the query conditions is the syntax of fuzzy query conditions, then it is determined to be a fuzzy query condition. The determination module is used to determine the calculation expression corresponding to the precise query condition, the calculation expression being used to determine the position of the data to be queried in a preset database; convert the calculation expression into code in a preset format to obtain a first query code; and determine the second query code corresponding to the fuzzy query condition, wherein the first query code and the second query code are Java query codes corresponding to the preset database. The processing module is used to perform query processing in the preset database according to the first query code, and to perform query processing in the preset database according to the second query code, to obtain target data; The storage module is used to load the target data into a preset flow table.

12. A data loading device, characterized in that, include: Memory, used to store programs; A processor for executing the program stored in the memory, wherein when the program is executed, the processor is configured to perform the method as described in any one of claims 1 to 10.

13. A computer-readable storage medium, characterized in that, Includes instructions that, when executed on a computer, cause the computer to perform the method as described in any one of claims 1 to 10.

Images