Data query method, device, system and storage medium

Abstract

The present invention relates to data processing in the field of financial technology or the medical field, and provides a data query method, device, system and storage medium. The method reads main system parameters and auxiliary system parameters, converts a data query script into a script syntax tree, traverses the script syntax tree, generates multiple script execution plans and execution sequence numbers, processes the multiple script execution plans according to the execution sequence numbers, obtains the target result, and during the processing of the multiple script execution plans, stores the obtained intermediate results in an auxiliary distributed file system. When it is monitored that the multiple script execution plans have completed processing, the target result is stored in the main distributed file system, which can solve the technical problem of low execution efficiency of data query tasks. In addition, the present invention also relates to blockchain technology, and the intermediate results can be stored in the blockchain.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a data query method, device, system and storage medium. Background Art

[0002] In the fields of fintech and healthcare, the query function of the HIVE data analysis system has emerged to facilitate user queries of financial or medical data. When executing data queries, the HIVE data analysis system typically stores the generated intermediate results in the HDFS distributed file system. When the HDFS distributed file system is heavily loaded, writing intermediate results further increases the load on the HDFS distributed file system, resulting in low query efficiency and reduced timeliness. Summary of the Invention

[0003] In view of the above, it is necessary to provide a data query method, device, system and storage medium that can solve the technical problem of low execution efficiency of data query tasks.

[0004] In one aspect, the present invention provides a data query method applied to a HIVE data analysis system, wherein the HIVE data analysis system runs a parameter configuration interface, and the data query method includes:

[0005] Reading multiple configuration files and initial system parameters in the parameter configuration interface, wherein the initial system parameters include main system parameters and auxiliary system parameters;

[0006] Convert the received data query script into a script syntax tree;

[0007] Traversing the script syntax tree to generate multiple script execution plans of the data query script and an execution sequence number for each script execution plan;

[0008] Processing the multiple script execution plans according to the execution sequence numbers to obtain a target result;

[0009] During the processing of the plurality of script execution plans, storing intermediate results obtained by processing the plurality of script execution plans in an auxiliary distributed file system corresponding to the auxiliary system parameters;

[0010] When it is monitored that the plurality of script execution plans are completed, the target results are stored in a primary distributed file system corresponding to the primary system parameters.

[0011] According to a preferred embodiment of the present invention, the reading of the initial system parameters in the multiple configuration files and the parameter configuration interface includes:

[0012] Obtaining the path directory corresponding to the parameter configuration interface;

[0013] Obtain the multiple configuration files from the path directory;

[0014] Reading information corresponding to the preset tags in the multiple configuration files as a first parameter, and reading information corresponding to the preset tags in the parameter configuration interface as a second parameter;

[0015] The first parameter and the second parameter are combined to obtain the initial system parameters.

[0016] According to a preferred embodiment of the present invention, converting the received data query script into a script syntax tree includes:

[0017] Preprocessing the data query script to obtain a target script;

[0018] Performing lexical analysis on the target script to obtain a grammar array;

[0019] The syntax array is parsed and corrected to obtain the script syntax tree.

[0020] According to a preferred embodiment of the present invention, performing lexical analysis on the target script to obtain a grammar array includes:

[0021] Scanning the target script for script characters;

[0022] Matching the script character with a preset lexical rule to obtain a character type corresponding to the script character;

[0023] The syntax array is generated according to the script character and the character type.

[0024] According to a preferred embodiment of the present invention, the step of performing grammatical analysis and correction on the grammatical array to obtain the script grammar tree includes:

[0025] Performing syntax analysis on the target script according to the syntax array to determine whether the syntax of the target script is correct;

[0026] If the syntax of the target script is correct, generating an initial syntax tree according to the code rules corresponding to the target script;

[0027] Creating a node object based on an initial node in the initial syntax tree;

[0028] The script syntax tree is generated according to the node object and the initial syntax tree.

[0029] According to a preferred embodiment of the present invention, processing the multiple script execution plans according to the execution sequence numbers to obtain the target result includes:

[0030] For the script execution plan whose execution sequence number is greater than the preset sequence number, identifying the target sequence number based on the execution sequence number;

[0031] Obtaining an intermediate result corresponding to the target sequence number from the auxiliary distributed file system as input data;

[0032] The script execution plan is processed based on the input data until the multiple script execution plans are completed and the target result is obtained.

[0033] According to a preferred embodiment of the present invention, storing the target result in a primary distributed file system corresponding to the primary system parameter includes:

[0034] identifying a source query database from the data query script;

[0035] Filtering the target system parameters corresponding to the source query database from the plurality of main system parameters;

[0036] Locating a primary distributed file system corresponding to the target system parameters as a target distributed file system;

[0037] Identifying a generation interface for the data query script;

[0038] Obtaining a data table corresponding to the generated interface;

[0039] Locating a table path of the data table in the target distributed file system;

[0040] The target result is written into the table path.

[0041] On the other hand, the present invention further provides a data query device that runs on a HIVE data analysis system. The HIVE data analysis system runs a parameter configuration interface. The data query device includes:

[0042] A reading unit, configured to read a plurality of configuration files and initial system parameters in the parameter configuration interface, wherein the initial system parameters include main system parameters and auxiliary system parameters;

[0043] A conversion unit, configured to convert a received data query script into a script syntax tree;

[0044] A generating unit, configured to traverse the script syntax tree to generate a plurality of script execution plans of the data query script and an execution sequence number for each script execution plan;

[0045] a processing unit, configured to process the plurality of script execution plans according to the execution sequence numbers to obtain a target result;

[0046] a storage unit, configured to store, during processing of the plurality of script execution plans, intermediate results obtained by processing the plurality of script execution plans in an auxiliary distributed file system corresponding to the auxiliary system parameters;

[0047] The storage unit is further configured to store the target result in a primary distributed file system corresponding to the primary system parameter when it is detected that the multiple script execution plans have completed processing.

[0048] On the other hand, the present invention also provides a HIVE data analysis system, comprising:

[0049] a memory storing computer-readable instructions; and

[0050] The processor executes the computer-readable instructions stored in the memory to implement the data query method.

[0051] On the other hand, the present invention further proposes a computer-readable storage medium, wherein the computer-readable storage medium stores computer-readable instructions, and the computer-readable instructions are executed by a processor in the HIVE data analysis system to implement the data query method.

[0052] It can be seen from the above technical solutions that this application can not only improve the comprehensiveness of parameter reading by reading parameters from the multiple configuration files and the parameter configuration interface, but also enable the HIVE data analysis system to identify the primary distributed file system and the auxiliary distributed file system, thereby avoiding the inability to accurately store the intermediate results and the target results due to the HIVE data analysis system. Furthermore, during the processing of the multiple script execution plans, this application can reduce the read, write and delete operations on the primary distributed file system by storing the intermediate results in the auxiliary distributed file system, so that the auxiliary distributed file system can be implemented as a diversion of the primary distributed file system, thereby improving the performance of the primary distributed file system, thereby improving the efficiency of generating the target results and ensuring the timeliness of data query tasks. This application is applicable to the field of financial technology. Through this application, it can assist financial enterprise staff in quickly querying financial data, thereby promoting the development of smart cities. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 This is an application scenario diagram of the data query method of the present invention.

[0054] Figure 2 It is a flow chart of a preferred embodiment of the data query method of the present invention.

[0055] Figure 3It is a schematic diagram of the structure of the script syntax tree in the present invention.

[0056] Figure 4 It is a functional module diagram of a preferred embodiment of the data query device of the present invention.

[0057] Figure 5 Schematic diagram of the structure of the HIVE data analysis system of the preferred embodiment of the data query method of the present invention. DETAILED DESCRIPTION

[0058] In order to make the objectives, technical solutions and advantages of the present invention more clear, the present invention is described in detail below with reference to the accompanying drawings and specific embodiments.

[0059] like Figure 1 FIG2 is an application scenario diagram of the data query method of the present invention. The data query method is applied to a HIVE data analysis system 1, which communicates with a primary distributed file system 2 and a secondary distributed file system 3. The HIVE data analysis system 1, the primary distributed file system 2, and the secondary distributed file system 3 all run on a distributed system 4, where there can be multiple primary distributed file systems 2.

[0060] The HIVE data analysis system 1 is based on the data warehouse tool in the distributed system 4 , and the HIVE data analysis system 1 can be used to extract, transform, load, etc. data.

[0061] The primary distributed file system 2 and the auxiliary distributed file system 3 are distributed file systems suitable for running on general-purpose hardware. They are highly fault-tolerant and designed to be deployed on low-cost hardware. They provide high-throughput access to application data.

[0062] The distributed system 4 provides high throughput access to application data. The distributed system 4 is suitable for running applications with very large data sets.

[0063] like Figure 2 FIG. 1 is a flow chart of a preferred embodiment of the data query method of the present invention. According to different requirements, the order of the steps in the flow chart can be changed, and some steps can be omitted.

[0064] The data query method can acquire and process relevant data based on artificial intelligence technology. Artificial Intelligence (AI) is the theory, method, technology, and application system that uses digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use knowledge to achieve optimal results.

[0065] Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interaction systems, and mechatronics. AI software technologies primarily encompass computer vision, robotics, biometrics, speech processing, natural language processing, and machine learning / deep learning.

[0066] The data query method is applied to one or more HIVE data analysis systems. The HIVE data analysis system is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored computer-readable instructions. Its hardware includes but is not limited to microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital signal processors (DSPs), embedded devices, etc.

[0067] The HIVE data analysis system can be any electronic product that can interact with a user, such as a personal computer, a tablet computer, a smart phone, a personal digital assistant (PDA), a game console, an interactive network television (IPTV), a smart wearable device, etc.

[0068] The HIVE data analysis system may include network devices and / or user devices. The network devices include, but are not limited to, a single network HIVE data analysis system, a HIVE data analysis system group consisting of multiple network HIVE data analysis systems, or a cloud based on cloud computing consisting of a large number of hosts or network HIVE data analysis systems.

[0069] The network where the HIVE data analysis system is located includes, but is not limited to: the Internet, wide area network, metropolitan area network, local area network, virtual private network (VPN), etc.

[0070] The HIVE data analysis system runs with a parameter configuration interface.

[0071] 101 , reading multiple configuration files and initial system parameters in the parameter configuration interface, where the initial system parameters include main system parameters and auxiliary system parameters.

[0072] In at least one embodiment of the present invention, the multiple configuration files may be files generated by an operation and maintenance user of a financial enterprise after completing system configuration on the parameter configuration interface. The multiple configuration files may include configurations for the primary system parameters and configurations for the auxiliary system parameters. The multiple configuration files may also be files generated by medical personnel after completing system configuration on the parameter configuration interface.

[0073] The parameter configuration interface may be an interface for the operation and maintenance user or the medical staff to configure the main system parameters or the auxiliary system parameters in the HIVE data analysis system.

[0074] The primary system parameter is used to indicate a primary distributed file system, and the secondary system parameter is used to indicate a secondary distributed file system.

[0075] In at least one embodiment of the present invention, the HIVE data analysis system reads multiple configuration files and initial system parameters in the parameter configuration interface, including:

[0076] Obtaining the path directory corresponding to the parameter configuration interface;

[0077] Obtain the multiple configuration files from the path directory;

[0078] Reading information corresponding to the preset tags in the multiple configuration files as a first parameter, and reading information corresponding to the preset tags in the parameter configuration interface as a second parameter;

[0079] The first parameter and the second parameter are combined to obtain the initial system parameters.

[0080] The path directory refers to the path where the generated file corresponding to the parameter configuration interface is stored, and the path directory supports changes.

[0081] The preset tags include, but are not limited to, dfs.nameservices tags, hive.exec.scratchdir tags, and the like.

[0082] For example, the first parameter read from the multiple configuration files is: "a, b", and the second parameter read from the parameter configuration interface is: "b, c", then after merging, the initial system parameters are "a, b, c".

[0083] By combining the first parameter and the second parameter, it is possible to avoid omission of the initial system parameters and failure to store data successfully.

[0084] 102, converting the received data query script into a script syntax tree.

[0085] In at least one embodiment of the present invention, the data query script may be a script generated by a user triggering a query for financial data or medical data. The financial data includes transaction data generated by electronic or online transactions, insurance data, etc. The medical data includes, but is not limited to, patient medical records.

[0086] The script syntax tree is an abstract representation of the data query script. The script syntax tree represents the syntax structure of the data query script in a tree-like form, and each node of the script syntax tree represents a structure in the data query script.

[0087] In at least one embodiment of the present invention, the HIVE data analysis system converting the received data query script into a script syntax tree includes:

[0088] Preprocessing the data query script to obtain a target script;

[0089] Performing lexical analysis on the target script to obtain a grammar array;

[0090] The syntax array is parsed and corrected to obtain the script syntax tree.

[0091] The target script is a data query script after removing irrelevant characters such as spaces. For example, the target script may be SELECT id,name FROM test WHERE id>2 LIMIT 2.

[0092] The syntax array includes script characters in the target script and category codes corresponding to the script characters. For example, the syntax array may be (SELECT, 24).

[0093] By preprocessing the data query script, irrelevant characters such as spaces in the data query script can be removed, which not only improves the script quality of the target script, but also avoids the influence of irrelevant characters on the script syntax tree. By performing syntax analysis and correction on the syntax array, syntax errors in the target script can be avoided, which may cause the script syntax tree to be unable to query data.

[0094] Specifically, the HIVE data analysis system performs lexical analysis on the target script to obtain a grammar array including:

[0095] Scanning the target script for script characters;

[0096] Matching the script character with a preset lexical rule to obtain a character type corresponding to the script character;

[0097] The syntax array is generated according to the script character and the character type.

[0098] The preset lexical rules are rules generated based on the scripting language of the data query script. Different scripting languages have different preset lexical rules. The scripting language may include JAVA language, etc.

[0099] By scanning the script characters, omission of the script characters can be avoided, thereby improving the comprehensiveness of the generation of the grammar array.

[0100] Specifically, the HIVE data analysis system performs syntax analysis and correction on the syntax array to obtain the script syntax tree including:

[0101] Performing syntax analysis on the target script according to the syntax array to determine whether the syntax of the target script is correct;

[0102] If the syntax of the target script is correct, generating an initial syntax tree according to the code rules corresponding to the target script;

[0103] Creating a node object based on an initial node in the initial syntax tree;

[0104] The script syntax tree is generated according to the node object and the initial syntax tree.

[0105] The code rules are rules constructed and generated by a scripting language.

[0106] like Figure 3 As shown, Figure 3 : This is a schematic diagram of the structure of the script syntax tree in the present invention. The script syntax tree includes a root node SELECT. The child nodes of the root node SELECT are selectList, FROM, WHERE and LIMIT. The leaf nodes of the root node SELECT are id, name, test, id, >, 2 and 2. Figure 3 The script syntax tree shown is an example only.

[0107] Through the above implementation, it is possible to ensure that the script syntax tree is generated based on a grammatically correct target script, thereby improving the accuracy of the script syntax tree.

[0108] In other embodiments, if the syntax of the target script is incorrect, the target script is identified based on the syntax array, and reminder information is obtained, and the reminder information is sent to the generation device of the data query script.

[0109] 103 , traverse the script syntax tree to generate multiple script execution plans of the data query script and an execution sequence number of each script execution plan.

[0110] In at least one embodiment of the present invention, the multiple script execution plans are based on logical execution plans obtained by parsing the script syntax tree.

[0111] The execution sequence number is used to indicate the execution order of the multiple script execution plans.

[0112] In at least one embodiment of the present invention, the HIVE data analysis system traverses the script syntax tree to generate multiple script execution plans for the data query script and an execution sequence number for each script execution plan, including:

[0113] Identify the root node and leaf nodes of the script syntax tree;

[0114] Splitting the script syntax tree based on the root node and the leaf nodes to obtain multiple execution operation trees;

[0115] Merging the multiple execution operation trees to obtain a target operation tree;

[0116] Performing transformation processing on the target operation tree to obtain the multiple script execution plans;

[0117] An execution sequence number of each script execution plan is generated based on the position of each script execution plan in the target script.

[0118] The root node refers to a node in the script syntax tree that has no parent node.

[0119] The leaf node refers to a node in the script syntax tree that has no child nodes.

[0120] Each execution operation tree refers to a tree formed by a node path between the root node and the leaf nodes.

[0121] The script syntax tree can be quickly split through the root node and the leaf node, thereby improving the generation efficiency of the multiple script execution plans.

[0122] 104. Process the multiple script execution plans according to the execution sequence numbers to obtain a target result.

[0123] It should be emphasized that in order to further ensure the privacy and security of the above target results, the above target results can also be stored in a node of a blockchain.

[0124] In at least one embodiment of the present invention, the target result refers to the query result corresponding to the data query script. For example, the target result may be the total transaction amount on a certain day, or the target result may be the diagnostic information of a user on a certain day.

[0125] In at least one embodiment of the present invention, the HIVE data analysis system processes the multiple script execution plans according to the execution sequence numbers to obtain target results including:

[0126] For the script execution plan whose execution sequence number is greater than the preset sequence number, identifying the target sequence number based on the execution sequence number;

[0127] Obtaining an intermediate result corresponding to the target sequence number from the auxiliary distributed file system as input data;

[0128] The script execution plan is processed based on the input data until the multiple script execution plans are completed and the target result is obtained.

[0129] The preset sequence number is usually set to 1.

[0130] The target sequence number refers to an adjacent sequence number that is smaller than the execution sequence number. For example, if the execution sequence number is 2, the target sequence number is 1; if the execution sequence number is 3, the target sequence number is 2.

[0131] By obtaining the input data from the auxiliary distributed file system and processing the multiple script execution plans, there is no need to read data from the primary distributed file system corresponding to the primary system parameters. Therefore, the load of the primary distributed file system can be reduced, thereby improving the execution efficiency of the multiple script execution plans.

[0132] In other embodiments, the HIVE data analysis system directly processes the script execution plan with the execution sequence number being the preset sequence number to obtain an intermediate result corresponding to the script execution plan with the preset sequence number.

[0133] 105 . During the processing of the multiple script execution plans, store intermediate results obtained by processing the multiple script execution plans in an auxiliary distributed file system corresponding to the auxiliary system parameters.

[0134] In at least one embodiment of the present invention, the HIVE data analysis system writes the plurality of intermediate results and the execution sequence numbers corresponding to the plurality of intermediate results into the auxiliary distributed file system.

[0135] Through the above implementation, the auxiliary distributed file system can achieve the effect of offloading traffic for the primary distributed file system.

[0136] 106. When it is detected that the multiple script execution plans have completed processing, the target results are stored in a primary distributed file system corresponding to the primary system parameters.

[0137] In at least one embodiment of the present invention, when it is detected that the multiple script execution plans have completed processing, the method further includes:

[0138] The intermediate result is deleted from the auxiliary distributed file system.

[0139] Through the above implementation, when the multiple script execution plans are completed, the multiple intermediate results can be released, thereby achieving the purpose of reducing the load of the auxiliary distributed file system.

[0140] In at least one embodiment of the present invention, the HIVE data analysis system storing the target result in a primary distributed file system corresponding to the primary system parameter includes:

[0141] identifying a source query database from the data query script;

[0142] Filtering the target system parameters corresponding to the source query database from the plurality of main system parameters;

[0143] Locating a primary distributed file system corresponding to the target system parameters as a target distributed file system;

[0144] Identifying a generation interface for the data query script;

[0145] Obtaining a data table corresponding to the generated interface;

[0146] Locating a table path of the data table in the target distributed file system;

[0147] The target result is written into the table path.

[0148] The source query database refers to an execution database of the data query script. For example, the source query database may be a database storing financial data or medical data.

[0149] The target distributed file system refers to a primary distributed file system storing the source query database.

[0150] The generation interface refers to the interface when the data query script is generated.

[0151] The data table refers to a table or partition that stores the target result.

[0152] Through the source query database and the generation interface, the table path can be accurately located, thereby improving the writing accuracy of the target result.

[0153] It can be seen from the above technical solutions that this application can not only improve the comprehensiveness of parameter reading by reading parameters from the multiple configuration files and the parameter configuration interface, but also enable the HIVE data analysis system to identify the primary distributed file system and the auxiliary distributed file system, thereby avoiding the inability to accurately store the intermediate results and the target results due to the HIVE data analysis system. Furthermore, during the processing of the multiple script execution plans, this application can reduce the read, write and delete operations on the primary distributed file system by storing the intermediate results in the auxiliary distributed file system, so that the auxiliary distributed file system can be implemented as a diversion of the primary distributed file system, thereby improving the performance of the primary distributed file system, thereby improving the efficiency of generating the target results and ensuring the timeliness of data query tasks. This application is applicable to the field of financial technology. Through this application, it can assist financial enterprise staff in quickly querying financial data, thereby promoting the development of smart cities.

[0154] like Figure 4 , which is a functional module diagram of a preferred embodiment of a data query device according to the present invention. The data query device 11 includes a reading unit 110, a conversion unit 111, a generation unit 112, a processing unit 113, a storage unit 114, and a deletion unit 115. As used herein, a module / unit refers to a series of computer-readable instruction segments that can be accessed by the processor 13 and perform a fixed function, and is stored in the memory 12. In this embodiment, the functions of each module / unit will be described in detail in subsequent embodiments.

[0155] The data query device runs on the HIVE data analysis system, and the HIVE data analysis system runs a parameter configuration interface.

[0156] The reading unit 110 reads a plurality of configuration files and initial system parameters in the parameter configuration interface, where the initial system parameters include main system parameters and auxiliary system parameters.

[0157] In at least one embodiment of the present invention, the multiple configuration files may be files generated by an operation and maintenance user of a financial enterprise after completing system configuration on the parameter configuration interface. The multiple configuration files may include configurations for the primary system parameters and configurations for the auxiliary system parameters. The multiple configuration files may also be files generated by medical personnel after completing system configuration on the parameter configuration interface.

[0158] The parameter configuration interface may be an interface for the operation and maintenance user or the medical staff to configure the main system parameters or the auxiliary system parameters in the HIVE data analysis system.

[0159] The primary system parameter is used to indicate a primary distributed file system, and the secondary system parameter is used to indicate a secondary distributed file system.

[0160] In at least one embodiment of the present invention, the reading unit 110 reads the initial system parameters in the multiple configuration files and the parameter configuration interface, including:

[0161] Obtaining the path directory corresponding to the parameter configuration interface;

[0162] Obtain the multiple configuration files from the path directory;

[0163] Reading information corresponding to the preset tags in the multiple configuration files as a first parameter, and reading information corresponding to the preset tags in the parameter configuration interface as a second parameter;

[0164] The first parameter and the second parameter are combined to obtain the initial system parameters.

[0165] The path directory refers to the path where the generated file corresponding to the parameter configuration interface is stored, and the path directory supports changes.

[0166] The preset tags include, but are not limited to, dfs.nameservices tags, hive.exec.scratchdir tags, and the like.

[0167] For example, the first parameter read from the multiple configuration files is: "a, b", and the second parameter read from the parameter configuration interface is: "b, c", then after merging, the initial system parameters are "a, b, c".

[0168] By combining the first parameter and the second parameter, it is possible to avoid omission of the initial system parameters and failure to store data successfully.

[0169] The conversion unit 111 converts the received data query script into a script syntax tree.

[0170] In at least one embodiment of the present invention, the data query script may be a script generated by a user triggering a query for financial data or medical data. The financial data includes transaction data generated by electronic or online transactions, insurance data, etc. The medical data includes, but is not limited to, patient medical records.

[0171] The script syntax tree is an abstract representation of the data query script. The script syntax tree represents the syntax structure of the data query script in a tree-like form, and each node of the script syntax tree represents a structure in the data query script.

[0172] In at least one embodiment of the present invention, the converting unit 111 converts the received data query script into a script syntax tree, including:

[0173] Preprocessing the data query script to obtain a target script;

[0174] Performing lexical analysis on the target script to obtain a grammar array;

[0175] The syntax array is parsed and corrected to obtain the script syntax tree.

[0176] The target script is a data query script after removing irrelevant characters such as spaces. For example, the target script may be SELECT id,name FROM test WHERE id>2 LIMIT 2.

[0177] The syntax array includes script characters in the target script and category codes corresponding to the script characters. For example, the syntax array may be (SELECT, 24).

[0178] By preprocessing the data query script, irrelevant characters such as spaces in the data query script can be removed, which not only improves the script quality of the target script, but also avoids the influence of irrelevant characters on the script syntax tree. By performing syntax analysis and correction on the syntax array, syntax errors in the target script can be avoided, which may cause the script syntax tree to be unable to query data.

[0179] Specifically, the conversion unit 111 performs lexical analysis on the target script to obtain a grammar array including:

[0180] Scanning the target script for script characters;

[0181] Matching the script character with a preset lexical rule to obtain a character type corresponding to the script character;

[0182] The syntax array is generated according to the script character and the character type.

[0183] The preset lexical rules are rules generated based on the scripting language of the data query script. Different scripting languages have different preset lexical rules. The scripting language may include JAVA language, etc.

[0184] By scanning the script characters, omission of the script characters can be avoided, thereby improving the comprehensiveness of the generation of the grammar array.

[0185] Specifically, the conversion unit 111 performs syntax analysis and correction on the syntax array to obtain the script syntax tree including:

[0186] Performing syntax analysis on the target script according to the syntax array to determine whether the syntax of the target script is correct;

[0187] If the syntax of the target script is correct, generating an initial syntax tree according to the code rules corresponding to the target script;

[0188] Creating a node object based on an initial node in the initial syntax tree;

[0189] The script syntax tree is generated according to the node object and the initial syntax tree.

[0190] The code rules are rules constructed and generated by a scripting language.

[0191] like Figure 3 As shown, Figure 3 : This is a schematic diagram of the structure of the script syntax tree in the present invention. The script syntax tree includes a root node SELECT. The child nodes of the root node SELECT are selectList, FROM, WHERE and LIMIT. The leaf nodes of the root node SELECT are id, name, test, id, >, 2 and 2. Figure 3 The script syntax tree shown is an example only.

[0192] Through the above implementation, it is possible to ensure that the script syntax tree is generated based on a grammatically correct target script, thereby improving the accuracy of the script syntax tree.

[0193] In other embodiments, if the syntax of the target script is incorrect, the target script is identified based on the syntax array, and reminder information is obtained, and the reminder information is sent to the generation device of the data query script.

[0194] The generating unit 112 traverses the script syntax tree to generate multiple script execution plans of the data query script and an execution sequence number of each script execution plan.

[0195] In at least one embodiment of the present invention, the multiple script execution plans are based on logical execution plans obtained by parsing the script syntax tree.

[0196] The execution sequence number is used to indicate the execution order of the multiple script execution plans.

[0197] In at least one embodiment of the present invention, the generating unit 112 traverses the script syntax tree to generate multiple script execution plans of the data query script and the execution sequence number of each script execution plan includes:

[0198] Identify the root node and leaf nodes of the script syntax tree;

[0199] Splitting the script syntax tree based on the root node and the leaf nodes to obtain multiple execution operation trees;

[0200] Merging the multiple execution operation trees to obtain a target operation tree;

[0201] Performing transformation processing on the target operation tree to obtain the multiple script execution plans;

[0202] An execution sequence number of each script execution plan is generated based on the position of each script execution plan in the target script.

[0203] The root node refers to a node in the script syntax tree that has no parent node.

[0204] The leaf node refers to a node in the script syntax tree that has no child nodes.

[0205] Each execution operation tree refers to a tree formed by a node path between the root node and the leaf nodes.

[0206] The script syntax tree can be quickly split through the root node and the leaf node, thereby improving the generation efficiency of the multiple script execution plans.

[0207] The processing unit 113 processes the multiple script execution plans according to the execution sequence numbers to obtain a target result.

[0208] It should be emphasized that in order to further ensure the privacy and security of the above target results, the above target results can also be stored in a node of a blockchain.

[0209] In at least one embodiment of the present invention, the target result refers to the query result corresponding to the data query script. For example, the target result may be the total transaction amount on a certain day, or the target result may be the diagnostic information of a user on a certain day.

[0210] In at least one embodiment of the present invention, the processing unit 113 processes the multiple script execution plans according to the execution sequence numbers to obtain the target result including:

[0211] For the script execution plan whose execution sequence number is greater than the preset sequence number, identifying the target sequence number based on the execution sequence number;

[0212] Obtaining an intermediate result corresponding to the target sequence number from the auxiliary distributed file system as input data;

[0213] The script execution plan is processed based on the input data until the multiple script execution plans are completed and the target result is obtained.

[0214] The preset sequence number is usually set to 1.

[0215] The target sequence number refers to an adjacent sequence number that is smaller than the execution sequence number. For example, if the execution sequence number is 2, the target sequence number is 1; if the execution sequence number is 3, the target sequence number is 2.

[0216] By obtaining the input data from the auxiliary distributed file system and processing the multiple script execution plans, there is no need to read data from the primary distributed file system corresponding to the primary system parameters. Therefore, the load of the primary distributed file system can be reduced, thereby improving the execution efficiency of the multiple script execution plans.

[0217] In other embodiments, the processing unit 113 directly processes the script execution plan with the execution sequence number being the preset sequence number to obtain an intermediate result corresponding to the script execution plan with the preset sequence number.

[0218] During the processing of the multiple script execution plans, the storage unit 114 stores intermediate results obtained by processing the multiple script execution plans in the auxiliary distributed file system corresponding to the auxiliary system parameters.

[0219] In at least one embodiment of the present invention, the storage unit 114 writes the plurality of intermediate results and the execution sequence numbers corresponding to the plurality of intermediate results into the auxiliary distributed file system.

[0220] Through the above implementation, the auxiliary distributed file system can achieve the effect of offloading traffic for the primary distributed file system.

[0221] When it is detected that the plurality of script execution plans have completed processing, the storage unit 114 stores the target results in the primary distributed file system corresponding to the primary system parameters.

[0222] In at least one embodiment of the present invention, when it is detected that the multiple script execution plans have completed processing, the deleting unit 115 deletes the intermediate results from the auxiliary distributed file system.

[0223] Through the above implementation, when the multiple script execution plans are completed, the multiple intermediate results can be released, thereby achieving the purpose of reducing the load of the auxiliary distributed file system.

[0224] In at least one embodiment of the present invention, the storage unit 114 storing the target result in the primary distributed file system corresponding to the primary system parameter includes:

[0225] identifying a source query database from the data query script;

[0226] Filtering the target system parameters corresponding to the source query database from the plurality of main system parameters;

[0227] Locating a primary distributed file system corresponding to the target system parameters as a target distributed file system;

[0228] Identifying a generation interface for the data query script;

[0229] Obtaining a data table corresponding to the generated interface;

[0230] Locating a table path of the data table in the target distributed file system;

[0231] The target result is written into the table path.

[0232] The source query database refers to an execution database of the data query script. For example, the source query database may be a database storing financial data.

[0233] The target distributed file system refers to a primary distributed file system storing the source query database.

[0234] The generation interface refers to the interface when the data query script is generated.

[0235] The data table refers to a table or partition that stores the target result.

[0236] Through the source query database and the generation interface, the table path can be accurately located, thereby improving the writing accuracy of the target result.

[0237] It can be seen from the above technical solutions that this application can not only improve the comprehensiveness of parameter reading by reading parameters from the multiple configuration files and the parameter configuration interface, but also enable the HIVE data analysis system to identify the primary distributed file system and the auxiliary distributed file system, thereby avoiding the inability to accurately store the intermediate results and the target results due to the HIVE data analysis system. Furthermore, during the processing of the multiple script execution plans, this application can reduce the read, write and delete operations on the primary distributed file system by storing the intermediate results in the auxiliary distributed file system, so that the auxiliary distributed file system can be implemented as a diversion of the primary distributed file system, thereby improving the performance of the primary distributed file system, thereby improving the efficiency of generating the target results and ensuring the timeliness of data query tasks. This application is applicable to the field of financial technology. Through this application, it can assist financial enterprise staff in quickly querying financial data, thereby promoting the development of smart cities.

[0238] like Figure 5 FIG. 1 is a schematic diagram of the structure of a HIVE data analysis system according to a preferred embodiment of the present invention for implementing a data query method.

[0239] In one embodiment of the present invention, the HIVE data analysis system 1 includes, but is not limited to, a memory 12 , a processor 13 , and computer-readable instructions stored in the memory 12 and executable on the processor 13 , such as a data query program.

[0240] Those skilled in the art will understand that the schematic diagram is merely an example of the HIVE data analysis system 1 and does not constitute a limitation of the HIVE data analysis system 1. The HIVE data analysis system 1 may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, the HIVE data analysis system 1 may also include input and output devices, network access devices, buses, etc.

[0241] The processor 13 may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor. The processor 13 is the computing core and control center of the HIVE data analysis system 1. It utilizes various interfaces and circuits to connect various components of the HIVE data analysis system 1 and executes the operating system of the HIVE data analysis system 1 as well as various installed applications and program codes.

[0242] Exemplarily, the computer-readable instructions may be divided into one or more modules / units, which are stored in the memory 12 and executed by the processor 13 to implement the present invention. The one or more modules / units may be a series of computer-readable instruction segments capable of performing specific functions, and the computer-readable instruction segments are used to describe the execution process of the computer-readable instructions in the HIVE data analysis system 1. For example, the computer-readable instructions may be divided into a reading unit 110, a conversion unit 111, a generation unit 112, a processing unit 113, a storage unit 114, and a deletion unit 115.

[0243] The memory 12 can be used to store the computer-readable instructions and / or modules. The processor 13 implements the various functions of the HIVE data analysis system 1 by running or executing the computer-readable instructions and / or modules stored in the memory 12 and accessing the data stored in the memory 12. The memory 12 can mainly include a program storage area and a data storage area. The program storage area can store an operating system and at least one application required for a function (such as a sound playback function, an image playback function, etc.); the data storage area can store data generated based on the use of the HIVE data analysis system. The memory 12 can include non-volatile and volatile memory, such as a hard disk, internal memory, a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, at least one disk storage device, a flash memory device, or other storage devices.

[0244] The memory 12 may be an external memory and / or an internal memory of the HIVE data analysis system 1. Furthermore, the memory 12 may be a physical memory, such as a memory stick, a TF card (Trans-flash Card), and the like.

[0245] If the modules / units integrated in the HIVE data analysis system 1 are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the present invention can also implement all or part of the processes in the above-mentioned method embodiments by instructing related hardware through computer-readable instructions. The computer-readable instructions can be stored in a computer-readable storage medium. When executed by a processor, the computer-readable instructions can implement the steps of each of the above-mentioned method embodiments.

[0246] The computer-readable instructions include computer-readable instruction codes, which may be in source code form, object code form, executable file, or some intermediate form. The computer-readable medium may include any entity or device capable of carrying the computer-readable instruction codes, a recording medium, a USB flash drive, a mobile hard drive, a magnetic disk, an optical disk, a computer memory, a read-only memory (ROM), or a random access memory (RAM).

[0247] Blockchain, as used in this article, refers to a novel application model for computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanisms, and encryption algorithms. Blockchain is essentially a decentralized database, a series of data blocks generated using cryptographic methods. Each block contains information about a batch of online transactions, which is used to verify the validity of the information (to prevent counterfeiting) and generate the next block. Blockchain can include the underlying blockchain platform, the platform product service layer, and the application service layer.

[0248] Combine Figure 2 The memory 12 in the HIVE data analysis system 1 stores computer-readable instructions to implement a data query method, and the processor 13 can execute the computer-readable instructions to implement:

[0249] Reading multiple configuration files and initial system parameters in the parameter configuration interface, wherein the initial system parameters include main system parameters and auxiliary system parameters;

[0250] Convert the received data query script into a script syntax tree;

[0251] Traversing the script syntax tree to generate multiple script execution plans of the data query script and an execution sequence number for each script execution plan;

[0252] Processing the multiple script execution plans according to the execution sequence numbers to obtain a target result;

[0253] During the processing of the plurality of script execution plans, storing intermediate results obtained by processing the plurality of script execution plans in an auxiliary distributed file system corresponding to the auxiliary system parameters;

[0254] When it is monitored that the plurality of script execution plans are completed, the target results are stored in a primary distributed file system corresponding to the primary system parameters.

[0255] Specifically, the specific implementation method of the processor 13 for the above computer readable instructions can refer to Figure 1 The description of the relevant steps in the corresponding embodiments will not be repeated here.

[0256] In the several embodiments provided herein, it should be understood that the disclosed systems, devices, and methods may be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the module division is merely a logical functional division, and actual implementation may employ other division methods.

[0257] The computer-readable storage medium stores computer-readable instructions, wherein the computer-readable instructions, when executed by the processor 13, are used to implement the following steps:

[0258] Reading multiple configuration files and initial system parameters in the parameter configuration interface, wherein the initial system parameters include main system parameters and auxiliary system parameters;

[0259] Convert the received data query script into a script syntax tree;

[0260] Traversing the script syntax tree to generate multiple script execution plans of the data query script and an execution sequence number for each script execution plan;

[0261] Processing the multiple script execution plans according to the execution sequence numbers to obtain a target result;

[0262] During the processing of the plurality of script execution plans, storing intermediate results obtained by processing the plurality of script execution plans in an auxiliary distributed file system corresponding to the auxiliary system parameters;

[0263] When it is monitored that the plurality of script execution plans are completed, the target results are stored in a primary distributed file system corresponding to the primary system parameters.

[0264] The modules described as separate components may or may not be physically separate, and the components shown as modules may or may not be physical units, that is, they may be located in one place or distributed across multiple network elements. Some or all of the modules may be selected to achieve the purpose of the solution of this embodiment according to actual needs.

[0265] In addition, the functional modules in various embodiments of the present invention may be integrated into a single processing unit, each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or hardware plus software functional modules.

[0266] Therefore, the embodiments should be considered in all respects as illustrative and non-restrictive, and the scope of the invention is defined by the appended claims rather than the foregoing description, and all changes that come within the meaning and range of equivalents of the claims are intended to be embraced therein. Any reference to a figure in a claim should not be construed as limiting the claim to which it relates.

[0267] Furthermore, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. The multiple units or devices described may also be implemented by a single unit or device through software or hardware. Terms such as first and second are used to indicate names and do not imply any particular order.

Claims

1. A data query method, applied to a HIVE data analysis system, wherein the HIVE data analysis system runs a parameter configuration interface, characterized in that: The data query method includes: Reading multiple configuration files and initial system parameters in the parameter configuration interface, wherein the initial system parameters include main system parameters and auxiliary system parameters; Convert the received data query script into a script syntax tree; Traversing the script syntax tree to generate multiple script execution plans of the data query script and an execution sequence number for each script execution plan; Processing the multiple script execution plans according to the execution sequence numbers to obtain a target result; During the processing of the plurality of script execution plans, storing intermediate results obtained by processing the plurality of script execution plans in an auxiliary distributed file system corresponding to the auxiliary system parameters; When it is monitored that the plurality of script execution plans are completed, the target results are stored in a primary distributed file system corresponding to the primary system parameters.

2. The data query method according to claim 1, wherein: The reading of the initial system parameters in the multiple configuration files and the parameter configuration interface includes: Obtaining the path directory corresponding to the parameter configuration interface; Obtain the multiple configuration files from the path directory; Reading information corresponding to the preset tags in the multiple configuration files as a first parameter, and reading information corresponding to the preset tags in the parameter configuration interface as a second parameter; The first parameter and the second parameter are combined to obtain the initial system parameters.

3. The data query method according to claim 1, wherein: Converting the received data query script into a script syntax tree includes: Preprocessing the data query script to obtain a target script; Performing lexical analysis on the target script to obtain a grammar array; The syntax array is parsed and corrected to obtain the script syntax tree.

4. The data query method according to claim 3, wherein: The lexical analysis of the target script is performed to obtain a grammar array including: Scanning the target script for script characters; Matching the script character with a preset lexical rule to obtain a character type corresponding to the script character; The syntax array is generated according to the script character and the character type.

5. The data query method according to claim 3, wherein: The grammatical analysis and correction of the grammatical array to obtain the script grammar tree includes: Performing syntax analysis on the target script according to the syntax array to determine whether the syntax of the target script is correct; If the syntax of the target script is correct, generating an initial syntax tree according to the code rules corresponding to the target script; Creating a node object based on an initial node in the initial syntax tree; The script syntax tree is generated according to the node object and the initial syntax tree.

6. The data query method according to claim 1, wherein: The processing of the multiple script execution plans according to the execution sequence numbers to obtain the target result includes: For the script execution plan whose execution sequence number is greater than the preset sequence number, identifying the target sequence number based on the execution sequence number; Obtaining an intermediate result corresponding to the target sequence number from the auxiliary distributed file system as input data; The script execution plan is processed based on the input data until the multiple script execution plans are completed and the target result is obtained.

7. The data query method according to claim 1, wherein: The storing the target result in the primary distributed file system corresponding to the primary system parameter includes: identifying a source query database from the data query script; Filtering the target system parameters corresponding to the source query database from the plurality of main system parameters; Locating a primary distributed file system corresponding to the target system parameters as a target distributed file system; Identifying a generation interface for the data query script; Obtaining a data table corresponding to the generated interface; Locating a table path of the data table in the target distributed file system; The target result is written into the table path.

8. A data query device, running on a HIVE data analysis system, wherein the HIVE data analysis system runs a parameter configuration interface, characterized in that: The data query device comprises: A reading unit, configured to read a plurality of configuration files and initial system parameters in the parameter configuration interface, wherein the initial system parameters include main system parameters and auxiliary system parameters; A conversion unit, configured to convert a received data query script into a script syntax tree; A generating unit, configured to traverse the script syntax tree to generate a plurality of script execution plans of the data query script and an execution sequence number for each script execution plan; a processing unit, configured to process the plurality of script execution plans according to the execution sequence numbers to obtain a target result; a storage unit, configured to store, during processing of the plurality of script execution plans, intermediate results obtained by processing the plurality of script execution plans in an auxiliary distributed file system corresponding to the auxiliary system parameters; The storage unit is further configured to store the target result in a primary distributed file system corresponding to the primary system parameter when it is detected that the multiple script execution plans have completed processing.

9. A HIVE data analysis system, characterized in that: The HIVE data analysis system includes: a memory storing computer-readable instructions; and A processor executes computer-readable instructions stored in the memory to implement the data query method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-readable instructions, and the computer-readable instructions are executed by a processor in the HIVE data analysis system to implement the data query method according to any one of claims 1 to 7.

Images