Artificial intelligence-based data processing method and apparatus, electronic device, and medium

Abstract

The application relates to the technical field of big data, and provides a data processing method and device based on artificial intelligence, electronic equipment and a medium.The method comprises the following steps: acquiring a first database table; parsing the first database table to obtain a parsing result; constructing a conversion strategy; converting each node of a syntax tree in the parsing result based on the conversion strategy to obtain a conversion result; when the conversion result is conversion success, checking a second database table associated with the conversion result; when it is determined that the second database table is qualified, executing a task in the second database table to obtain a data processing result.The first database table is parsed and converted to obtain the second database table, and the task is executed after the second database table is qualified, so that the fault tolerance of task execution is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of big data, and particularly relates to a data processing method and device based on artificial intelligence, electronic equipment and medium. BACKGROUND

[0002] With the development of the Spark computing engine, its superior memory computing performance is faster than the Hadoop native MapReduce engine computing, and the utilization rate of memory resources in the distributed cluster is higher, but in terms of the stability of the big data set operation, the MapReduce engine is relatively higher than Spark.

[0003] At present, the current Spark SQL syntax and Hive SQL syntax are not completely compatible, there are differences, and the fault tolerance of task execution cannot be ensured in the execution task process. SUMMARY

[0004] In view of the above, it is necessary to propose a data processing method and device based on artificial intelligence, electronic equipment and medium, by analyzing and converting the first database table to obtain the second database table, and executing the task after the second database table is verified, the fault tolerance of task execution is improved.

[0005] The first aspect of the present application provides a data processing method based on artificial intelligence, the method comprises:

[0006] In response to the received data processing request, the first database table is obtained;

[0007] The first database table is parsed to obtain a parsing result;

[0008] A conversion strategy is constructed, and each node of the syntax tree in the parsing result is converted based on the conversion strategy to obtain a conversion result;

[0009] When the conversion result is conversion success, the second database table associated with the conversion result is verified;

[0010] When it is determined that the second database table is verified, the task in the second database table is executed to obtain a data processing result.

[0011] Optionally, the execution of the task in the second database table to obtain the data processing result comprises:

[0012] When it is detected that any one of the tasks in the second database table fails to execute, the first database table is obtained, and the task in the first database table is executed;

[0013] When each task in the first database table is successfully executed, it is determined that the data processing result is the end of task execution.

[0014] When any one of the first database tables fails to execute a task, determine that the data processing result is a trigger exception task processing instruction.

[0015] Optionally, the parsing the first database table to obtain a parsing result comprises:

[0016] Obtaining a syntax rule set corresponding to the first database table;

[0017] Generating a syntax analyzer based on the syntax rule set;

[0018] Performing syntax analysis on a first query statement in the first database table using the syntax analyzer to obtain a corresponding syntax element;

[0019] Determining the node position of each syntax element and arranging each syntax element according to the node position to generate a corresponding syntax tree, and determining the syntax tree as the parsing result.

[0020] Optionally, the constructing a conversion strategy comprises:

[0021] Identifying the difference between the first syntax of the first database table and the second syntax of the second database table;

[0022] Constructing a conversion strategy based on the difference using a preset function.

[0023] Optionally, the converting each node of the syntax tree in the parsing result based on the conversion strategy to obtain a conversion result comprises:

[0024] Identifying the conversion strategy of each node of the syntax tree;

[0025] Converting the first query statement of each node based on the conversion strategy of each node;

[0026] When the conversion result is a second query statement of each node, determining that the conversion result is a conversion success, and associating the conversion result with the second database table, wherein the second database table is obtained by writing the second query statement of each node into a preset database table;

[0027] When the conversion result is not a second query statement of each node, determining that the conversion result is a conversion failure.

[0028] Optionally, the method further comprises:

[0029] When the conversion result is a conversion failure, adjusting the conversion strategy to perform secondary conversion on the first query statement of the failed node.

[0030] Optionally, the method further comprises:

[0031] When it is determined that the second database table check is unqualified, first failure information of a failed node of the check failure is obtained, and the first failure information is sent to a client;

[0032] A new conversion strategy reported by the client based on the first failure information is received, and a first query statement of the failed node is converted again based on the new conversion strategy;

[0033] When the conversion result is a second query statement of the failed node, it is determined that the conversion result is conversion success, and the conversion ends, or when the conversion result is not the second query statement of the failed node, it is determined that the conversion result is conversion failure, and the above steps are repeated until the first query statement of the failed node is converted successfully, and the conversion ends;

[0034] The second query statement of the failed node obtained by conversion is iteratively checked until the second database table check is qualified.

[0035] The second aspect of the application provides a data processing device based on artificial intelligence, the device comprising:

[0036] An acquisition module is configured to acquire a first database table in response to a received data processing request;

[0037] An analysis module is configured to analyze the first database table to obtain an analysis result;

[0038] A conversion module is configured to construct a conversion strategy, convert each node of a syntax tree in the analysis result based on the conversion strategy, and obtain a conversion result;

[0039] A check module is configured to check a second database table associated with the conversion result when the conversion result is conversion success;

[0040] An execution module is configured to execute a task in the second database table to obtain a data processing result when it is determined that the second database table check is qualified.

[0041] The third aspect of the application provides an electronic device, comprising a processor and a memory, wherein the processor is configured to implement the data processing method based on artificial intelligence when executing the computer program stored in the memory.

[0042] The fourth aspect of the application provides a computer readable storage medium, wherein the computer readable storage medium stores a computer program, and the computer program is executed by a processor to implement the data processing method based on artificial intelligence.

[0043] In summary, the data processing method, device, electronic equipment and medium based on artificial intelligence can promote the construction of smart city, and is applied to fields such as smart building, smart security, smart community, smart life and Internet of Things. The first database table is parsed to obtain a parsing result, which facilitates subsequent conversion of the first database table based on the parsing result, and improves the conversion efficiency of the first database table. A conversion strategy is constructed, each node of a syntax tree in the parsing result is converted based on the conversion strategy, and a conversion result is obtained. By constructing the conversion strategy in advance, each node in the syntax tree is converted into a second query statement according to the corresponding conversion strategy by calling the conversion strategy, without human intervention, automatic conversion is realized, and the data conversion efficiency is improved. When the conversion result is conversion success, the second database table associated with the conversion result is checked, and when the second database table passes the check, the tasks in the second database table are executed to obtain a data processing result. In the case of ensuring that the conversion result is successful and the check result is qualified, the tasks in the second database table are executed, and when any one of the tasks in the second database table fails, the tasks in the first database table can be executed, avoiding the phenomenon that any one of the tasks in the second database table fails to execute the tasks, and improving the fault tolerance of executing the tasks. BRIEF DESCRIPTION OF DRAWINGS

[0044] Figure 1 is a flowchart of the data processing method based on artificial intelligence provided by the embodiment one of the present application.

[0045] Figure 2 is a structure diagram of the data processing device based on artificial intelligence provided by the embodiment two of the present application.

[0046] Figure 3 is a structure diagram of the electronic equipment provided by the embodiment three of the present application. DETAILED DESCRIPTION

[0047] In order to more clearly understand the above-mentioned purposes, features and advantages of the present application, the present application will be described in detail below in combination with the drawings and specific embodiments. It should be noted that the embodiments of the present application and the features in the embodiments can be combined with each other without conflict.

[0048] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present application belongs. The terms used in the specification of the present application are only for the purpose of describing specific embodiments and are not intended to limit the present application.

[0049] Embodiment one

[0050] Figure 1is a flowchart of the data processing method based on artificial intelligence provided by the first embodiment of the present application.

[0051] In the present embodiment, the data processing method based on artificial intelligence can be applied in electronic devices. For electronic devices that need to perform data processing based on artificial intelligence, the function of data processing based on artificial intelligence provided by the method of the present application can be integrated directly on the electronic device, or run in the form of a software development kit (SDK) in the electronic device.

[0052] The embodiments of the present application can acquire and process related data based on artificial intelligence technology. Artificial intelligence (AI) is the use of digital computers or computer-controlled machines to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0053] The basic technology of artificial intelligence generally includes technologies such as sensors, special artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction system, mechatronics, etc. The software technology of artificial intelligence mainly includes computer vision technology, robot technology, biometric technology, speech processing technology, natural language processing technology, and machine learning, deep learning, etc.

[0054] As shown in Figure 1 The data processing method based on artificial intelligence specifically includes the following steps, and the order of the steps in the flowchart can be changed, and some steps can be omitted according to different requirements.

[0055] S11, in response to the received data processing request, acquiring a first database table.

[0056] In the present embodiment, structured query language (Structured Query Language, SQL) is a query and programming language for database, used for accessing data and querying, updating and managing relational databases. Because the design mechanisms of Hive platform and Spark platform are different, Hive uses Mapreduce mechanism to perform data operation, while Spark uses RDD mechanism to parse and execute SQL, and the speed of executing different SQL statements is different.

[0057] In the present embodiment, the data processing request is used to request to acquire a to-be-processed HiveSQL file, and the HiveSQL file is determined as the first database table.

[0058] S12, parsing the first database table to obtain a parsing result.

[0059] In this embodiment, the parsing result refers to a result obtained after parsing the syntax in the first database table.

[0060] In an optional embodiment, the parsing of the first database table to obtain a parsing result comprises:

[0061] obtaining a syntax rule set corresponding to the first database table;

[0062] generating a syntax analyzer based on the syntax rule set;

[0063] performing syntax analysis on a first query statement in the first database table by using the syntax analyzer to obtain a corresponding syntax element;

[0064] determining a node position of each syntax element and arranging each syntax element according to the node position to generate a corresponding syntax tree, and determining the syntax tree as the parsing result.

[0065] Specifically, the syntax analyzer can be an ANTLR syntax analyzer, which is a recognizer tool written in Java language and directly generates a parser of a target language from ANTLR syntax rules in a syntax rule set.

[0066] In this embodiment, by performing syntax analysis on the first query statement in the first database table, a syntax element in the first query statement is identified, for example, the syntax element can be a keyword (such as SELECT, FROM, WHERE), a syntax expression, etc. A syntax tree is generated according to the syntax element and a corresponding node position, and the syntax tree is obtained by deducing the first database table according to a syntax rule set corresponding to the first database table.

[0067] In this embodiment, by parsing the syntax structure of the first database table, subsequent conversion of the first database table based on the syntax structure is facilitated, and the conversion efficiency of the first database table is improved.

[0068] S13, constructing a conversion strategy, converting each node of the syntax tree in the parsing result based on the conversion strategy to obtain a conversion result.

[0069] In the embodiment, the conversion result contains a second database table, wherein the second database table is obtained by converting the syntax tree of the first database table using the constructed conversion strategy, and specifically, the second database table is a Spark SQL file, i.e., the Hive SQL file is converted into the Spark SQL file by using the constructed conversion strategy.

[0070] In the embodiment, Spark SQL is developed from Shark, and Shark converts the first database table, i.e., the Hive SQL file, into the Spark SQL file by syntax analysis in order to realize Hive compatibility.

[0071] In an optional embodiment, the constructed conversion strategy includes:

[0072] identifying the difference between the first syntax of the first database table and the second syntax of the second database table;

[0073] constructing the conversion strategy based on the difference by using a preset function.

[0074] In the embodiment, the preset function can be a UDF (user-defined function) function, and specifically, the UDF function is a user-defined function, which is a subprogram composed of one or more SQL statements and is used for code encapsulation and reuse. The conversion strategy is constructed by using the user-defined function based on the difference between the first syntax of the first database table and the second syntax of the second database table.

[0075] In an optional embodiment, the conversion of each node of the syntax tree in the parsing result based on the conversion strategy to obtain the conversion result includes:

[0076] identifying the conversion strategy of each node of the syntax tree;

[0077] converting the first query statement of each node based on the conversion strategy of each node;

[0078] when the conversion result is the second query statement of each node, determining that the conversion is successful, and associating the conversion result with the second database table, wherein the second database table is obtained by writing the second query statement of each node into a preset database table;

[0079] when the conversion result is not the second query statement of each node, determining that the conversion is unsuccessful.

[0080] The first query statement in the embodiment refers to a query statement in the first database table, and the second query statement is a query statement obtained after the first query statement is converted based on the conversion strategy.

[0081] In the embodiment, the second query statement of each node can be obtained only when the conversion is successful. Therefore, when the first query statement of any node is not converted into the second query statement, it is determined that the conversion fails in the process of converting the first query statement of each node based on the conversion strategy of each node.

[0082] In the embodiment, the conversion strategy can be constructed in advance. The first query statement of each node in the syntax tree is converted into the second query statement according to the corresponding conversion strategy by calling the conversion strategy. No human intervention is needed, automatic conversion is realized, and the data conversion efficiency is improved.

[0083] S14, when the conversion result is conversion success, verifying the second database table associated with the conversion result.

[0084] In the embodiment, in the process of converting the Hive SQL file into the Spark SQL file, the conversion may fail due to the fact that the conversion strategy does not meet the requirements. When the conversion fails, the conversion strategy is adjusted and iteratively converted according to the failure information of the conversion failure until the conversion succeeds. No re-conversion of all nodes is needed, and the conversion efficiency of the Hive SQL file into the Spark SQL file is improved.

[0085] In an optional embodiment, the verifying the second database table associated with the conversion result comprises:

[0086] Obtaining the second query statement in the second database table;

[0087] Extracting the target syntax of each second query statement;

[0088] Traversing the standard syntax library corresponding to the second database table based on the target syntax of each second query statement;

[0089] When the target syntax of each second query statement is traversed in the standard syntax library, it is determined that the second database table passes the verification;

[0090] When the target syntax of any second query statement is not traversed in the standard syntax library, it is determined that the second database table fails the verification.

[0091] In the embodiment, the standard syntax library refers to the SparkSQL syntax library, and contains a plurality of SparkSQL syntaxes. When the target syntax of each second query statement is not traversed in the standard syntax library, it is determined that the conversion fails, that is, the second database table obtained by the conversion is unqualified.

[0092] In other optional embodiments, before the second database table is verified, the method further includes:

[0093] Based on the constructed conversion strategy, a target operation template is obtained.

[0094] A mapping relationship of the second query statement of the second database table is written into a corresponding conversion strategy of the target operation template, and a conversion report is generated.

[0095] In the embodiment, the conversion report is generated based on the conversion result. According to the constructed conversion strategy, the corresponding target operation template is determined, and the mapping relationship of the second query statement of the second database table is put into the corresponding conversion strategy. Subsequently, the failure information of the failure node located in the conversion report can be used to obtain the transformation and optimization conversion strategy guide, and the data processing efficiency is improved.

[0096] S15, when it is determined that the second database table is qualified, the tasks in the second database table are executed, and a data processing result is obtained.

[0097] In the embodiment, the data processing result contains a task execution end and a trigger abnormal task processing instruction.

[0098] In an optional embodiment, the execution of the tasks in the second database table to obtain the data processing result includes:

[0099] When it is detected that any one of the tasks in the second database table fails to execute, the first database table is obtained, and the tasks in the first database table are executed.

[0100] When each task in the first database table is successfully executed, it is determined that the data processing result is a task execution end.

[0101] When any one of the tasks in the first database table fails to execute, it is determined that the data processing result is a trigger abnormal task processing instruction.

[0102] In the embodiment, when any one of the tasks in the second database table fails to execute, the tasks in the first database table can be switched to execute, which avoids the phenomenon that any one of the tasks in the second database table fails to execute, and improves the fault tolerance of the executed tasks.

[0103] In an optional embodiment, the executing the task in the second database table to obtain a data processing result further includes:

[0104] When it is detected that each task in the second database table is executed successfully, the task execution ends.

[0105] In other optional embodiments, the method further includes:

[0106] When it is determined that the second database table passes the verification, outputting a directed acyclic graph of the second database table.

[0107] In this embodiment, the second database table is output as a directed acyclic graph, intuitively determining each task in the second database table, and each task exists independently. In the subsequent execution of the task, the data processing efficiency is improved.

[0108] Further, the method further includes:

[0109] When it is determined that the second database table fails the verification, obtaining first failure information of a failed node that fails the verification, and sending the first failure information to a client;

[0110] Receiving a new conversion strategy reported by the client based on the first failure information, and performing secondary conversion on the first query statement of the failed node based on the new conversion strategy;

[0111] When the conversion result is the second query statement of the failed node, it is determined that the conversion result is conversion success, and the conversion ends, or when the conversion result is not the second query statement of the failed node, it is determined that the conversion result is conversion failure, and the above steps are repeated until the first query statement of the failed node is converted successfully, and the conversion ends;

[0112] Performing iterative verification on the second query statement of the failed node obtained by conversion until the second database table passes the verification.

[0113] In this embodiment, when the second database fails the verification, the conversion strategy is continuously adjusted until the second database table passes the verification, and the accuracy of the obtained second database table is improved.

[0114] Further, the method further includes:

[0115] When the conversion result is conversion failure, adjusting the conversion strategy to perform secondary conversion on the first query statement of the failed node.

[0116] In the embodiment, when the conversion fails, the failure information obtained by the conversion failure is sent to the client, the client adjusts the conversion strategy according to the failure reason, and reconverts the first query statement of the failed node based on the new conversion strategy after adjustment until the conversion succeeds.

[0117] In the embodiment, the conversion accuracy is improved by continuously optimizing the conversion strategy.

[0118] In an optional embodiment, when the conversion result is conversion failure, the second conversion of the first query statement of the failed node is performed by adjusting the conversion strategy, comprising:

[0119] Obtaining the second failure information of the failed node of the conversion failure, and sending the second failure information to the client;

[0120] Receiving the new conversion strategy reported by the client based on the second failure information, and performing the second conversion of the first query statement of the failed node based on the new conversion strategy;

[0121] When the conversion result is the second query statement of the failed node, it is determined that the conversion result is conversion success, and the conversion is ended;

[0122] When the conversion result is not the second query statement of the failed node, it is determined that the conversion result is conversion failure, and the above steps are repeated until the first query statement of the failed node is converted successfully.

[0123] In summary, the data processing method based on artificial intelligence in the embodiment can obtain the parsing result by parsing the first database table, which facilitates the subsequent conversion of the first database table based on the parsing result, and improves the conversion efficiency of the first database table. The conversion strategy is constructed, and each node of the syntax tree in the parsing result is converted based on the conversion strategy to obtain the conversion result. By pre-construction of the conversion strategy, the first query statement of each node in the syntax tree is converted into the second query statement according to the corresponding conversion strategy by calling the conversion strategy, without human intervention, realizing automatic conversion and improving data conversion efficiency. When the conversion result is conversion success, the second database table associated with the conversion result is checked, and when the second database table passes the check, the tasks in the second database table are executed to obtain the data processing result. In the case of ensuring the success of the conversion result and the eligibility of the check result, the tasks in the second database table are executed, and when any task in the second database table fails, the tasks in the first database table can be switched to execute the tasks, avoiding the phenomenon that any task in the second database table fails to execute the tasks, and improving the fault tolerance of executing the tasks.

[0124] Embodiment two

[0125] Figure 2 is a structural diagram of the data processing apparatus based on artificial intelligence provided in Embodiment Two of the present application.

[0126] In some embodiments, the data processing apparatus based on artificial intelligence 20 can include a plurality of functional modules composed of program code segments. The program code of each program segment in the data processing apparatus based on artificial intelligence 20 can be stored in the memory of the electronic device and executed by the at least one processor to perform the functions of data processing based on artificial intelligence (see Figure 1 Description).

[0127] In this embodiment, the data processing apparatus based on artificial intelligence 20 can be divided into a plurality of functional modules according to the functions it performs. The functional modules can include an acquisition module 201, an analysis module 202, a conversion module 203, a verification module 204, an execution module 205, an output module 206, and an adjustment module 207. The module referred to in the present application refers to a series of computer-readable instruction segments that can be executed by at least one processor and can complete a fixed function, which is stored in the memory. In this embodiment, the functions of each module will be described in detail in subsequent embodiments.

[0128] The acquisition module 201 is configured to acquire a first database table in response to a received data processing request.

[0129] In this embodiment, the structured query language (Structured Query Language, SQL) is a language for database query and programming, used to access data and query, update and manage relational databases. Since the design mechanisms of the Hive platform and the Spark platform are different, Hive uses the Mapreduce mechanism to perform data operations, while Spark uses the RDD mechanism to parse and execute SQL, and different SQL statements have different speeds.

[0130] In this embodiment, the data processing request is used to request to obtain a HiveSQL file to be processed, and the HiveSQL file is determined as the first database table.

[0131] The analysis module 202 is configured to analyze the first database table to obtain an analysis result.

[0132] In this embodiment, the analysis result refers to the result obtained after analyzing the syntax in the first database table.

[0133] In an optional embodiment, the analysis module 202 analyzes the first database table to obtain an analysis result, which includes:

[0134] obtaining a syntax rule set corresponding to the first database table;

[0135] generating a syntax analyzer based on the syntax rule set;

[0136] parsing a first query statement in the first database table using the syntax analyzer to obtain a corresponding syntax element;

[0137] determining a node position of each syntax element and arranging each syntax element according to the node position to generate a corresponding syntax tree, and determining the syntax tree as a parsing result.

[0138] Specifically, the syntax analyzer can be an ANTLR syntax analyzer, which is a recognizer tool written in Java language and directly generates a parser of a target language from ANTLR syntax rules in a syntax rule set.

[0139] In this embodiment, by parsing the first query statement in the first database table, a syntax element in the first query statement is recognized, for example, the syntax element can be a keyword (such as SELECT, FROM, WHERE), a syntax expression, etc. A syntax tree is generated according to the syntax element and a corresponding node position. The syntax tree is derived from the first database table according to the syntax rule set corresponding to the first database table.

[0140] In this embodiment, by parsing the syntax structure of the first database table, it is convenient to convert the first database table based on the syntax structure subsequently, and the conversion efficiency of the first database table is improved.

[0141] The conversion module 203 is configured to construct a conversion strategy, convert each node of the syntax tree in the parsing result based on the conversion strategy, and obtain a conversion result.

[0142] In this embodiment, the conversion result includes a second database table, wherein the second database table is obtained by converting the syntax tree of the first database table using the constructed conversion strategy. Specifically, the second database table refers to a Spark SQL file, that is, a Hive SQL file is converted into a Spark SQL file by the constructed conversion strategy.

[0143] In this embodiment, Spark SQL is developed from Shark. In order to realize Hive compatibility, Shark converts a first database table, that is, a Hive SQL file, into a Spark SQL file by syntax parsing.

[0144] In an optional embodiment, the conversion module 203 constructs the conversion strategy includes:

[0145] identifying the difference between the first syntax of the first database table and the second syntax of the second database table;

[0146] constructing the conversion strategy based on the difference by using a preset function.

[0147] In the embodiment, the preset function can be a UDF (user-defined function) function. Specifically, the UDF function is a user-defined function, which is a subprogram composed of one or more SQL statements, and is used for encapsulating code for reuse. The conversion strategy is constructed by using the user-defined function based on the difference between the first syntax of the first database table and the second syntax of the second database table.

[0148] In an optional embodiment, the conversion module 203 converts each node of the syntax tree in the parsing result based on the conversion strategy to obtain a conversion result includes:

[0149] identifying the conversion strategy of each node of the syntax tree;

[0150] converting the first query statement of each node based on the conversion strategy of each node;

[0151] when the conversion result is the second query statement of each node, determining that the conversion is successful, and associating the conversion result with the second database table, wherein the second database table is obtained by writing the second query statement of each node into a preset database table;

[0152] when the conversion result is not the second query statement of each node, determining that the conversion is failed.

[0153] In the embodiment, the first query statement refers to the query statement in the first database table, and the second query statement is the query statement obtained by converting the first query statement based on the conversion strategy.

[0154] In the embodiment, the second query statement of each node can be obtained only when the conversion is successful. Therefore, in the process of converting the first query statement of each node based on the conversion strategy of each node, when the first query statement of any node is not converted into the second query statement, it is determined that the conversion is failed.

[0155] In this embodiment, a conversion strategy can be pre-built. By calling the conversion strategy, the first query statement of each node in the syntax tree is converted into the second query statement according to the corresponding conversion strategy. No human intervention is required, which realizes automatic conversion and improves data conversion efficiency.

[0156] The verification module 204 is used to verify the second database table associated with the conversion result when the conversion result is successful.

[0157] In this embodiment, during the process of converting Hive SQL files to Spark SQL files, conversion failures may occur due to the conversion strategy not meeting the requirements. When a conversion fails, the conversion strategy is continuously adjusted based on the failure information to perform iterative conversion until the conversion is successful. This eliminates the need to reconvert all nodes, thus improving the conversion efficiency of converting Hive SQL files to Spark SQL files.

[0158] In an optional embodiment, the verification module 204 verifies the second database table associated with the conversion result, including:

[0159] Retrieve the second query statement from the second database table;

[0160] Extract the target syntax for each second query statement;

[0161] Based on the target syntax of each second query statement, the standard syntax library corresponding to the second database table is traversed.

[0162] When the target syntax of each second query statement is encountered during the traversal of the standard syntax library, it is determined that the second database table is valid.

[0163] If the target syntax of any second query statement is not found in the standard syntax library, the second database table is determined to be unqualified.

[0164] In this embodiment, the standard syntax library refers to the Spark SQL syntax library, which contains multiple Spark SQL syntaxes. When the target syntax of each second query statement is not found in the standard syntax library, the conversion is determined to be unsuccessful, that is, the second database table obtained by conversion is not valid.

[0165] In other optional embodiments, before verifying the second database table, a target computation template is obtained based on the constructed conversion strategy; the mapping relationship of the second query statement of the second database table is written into the corresponding conversion strategy of the target computation template, and a conversion report is generated.

[0166] In the embodiment, the conversion report is generated based on a conversion result, a corresponding target operation template is determined according to a constructed conversion strategy, and a mapping relationship of a second query statement of the second database table is put into the corresponding conversion strategy. Subsequently, a transformation and optimization conversion strategy guide can be quickly obtained according to failure information of a failed node positioned in the conversion report, and data processing efficiency is improved.

[0167] The execution module 205 is configured to execute tasks in the second database table to obtain a data processing result when it is determined that the second database table is qualified.

[0168] In the embodiment, the data processing result includes a task execution end and a trigger abnormal task processing instruction.

[0169] In an optional embodiment, the execution module 205 executes tasks in the second database table to obtain a data processing result, which includes:

[0170] When it is detected that any one of the tasks in the second database table fails to execute, the first database table is obtained, and tasks in the first database table are executed.

[0171] When each task in the first database table successfully executes, the data processing result is determined to be a task execution end.

[0172] When any one of the tasks in the first database table fails to execute, the data processing result is determined to be a trigger abnormal task processing instruction.

[0173] In the embodiment, when any one of the tasks in the second database table fails to execute, the first database table can be switched to execute tasks, which avoids the phenomenon that any one of the tasks in the second database table fails to execute, and improves the fault tolerance of executing tasks.

[0174] In an optional embodiment, the execution module 205 executes tasks in the second database table to obtain a data processing result, which further includes:

[0175] When it is detected that each task in the second database table successfully executes, the task execution ends.

[0176] In other optional embodiments, the output module 206 is configured to output a directed acyclic graph of the second database table when it is determined that the second database table is qualified.

[0177] In the embodiment, the second database table is output as a directed acyclic graph, which directly determines each task in the second database table, and each task exists independently. Subsequently, the tasks can be processed in parallel, and data processing efficiency is improved.

[0178] Further, when it is determined that the second database table check is not qualified, first failure information of a failed node of the check failure is acquired, the first failure information is sent to a client, a new conversion strategy reported by the client based on the first failure information is received, the first query statement of the failed node is converted again based on the new conversion strategy, when the conversion result is a second query statement of the failed node, it is determined that the conversion result is conversion success and the conversion is ended, or when the conversion result is not the second query statement of the failed node, it is determined that the conversion result is conversion failure, the above steps are repeated until the first query statement of the failed node is converted successfully and the conversion is ended, and the second query statement of the failed node converted is iteratively checked until the second database table check is qualified.

[0179] In the embodiment, when the second database check is not qualified, the conversion strategy is continuously adjusted until the second database table check is qualified, and the accuracy of the obtained second database table is improved.

[0180] The adjusting module 207 is configured to, when the conversion result is conversion failure, adjust the conversion strategy to convert the first query statement of the failed node again.

[0181] In the embodiment, when the conversion is failed, the failure information obtained by the conversion failure is sent to the client, the client adjusts the conversion strategy again according to the failure reason, and converts the first query statement of the failed node again based on the new conversion strategy after the adjustment until the conversion is successful.

[0182] In the embodiment, the conversion strategy is continuously optimized, and the accuracy of data conversion is improved.

[0183] In an optional embodiment, when the conversion result is conversion failure, the adjusting module 207 adjusts the conversion strategy to convert the first query statement of the failed node again, including:

[0184] Second failure information of the failed node of the conversion failure is acquired, and the second failure information is sent to the client.

[0185] The new conversion strategy reported by the client based on the second failure information is received, and the first query statement of the failed node is converted again based on the new conversion strategy.

[0186] When the conversion result is a second query statement of the failed node, it is determined that the conversion result is conversion success and the conversion is ended.

[0187] When the conversion result is not the second query statement of the failed node, it is determined that the conversion result is conversion failure, and the above steps are repeated until the first query statement of the failed node is converted successfully.

[0188] In summary, the data processing device based on artificial intelligence provided in the embodiment can obtain an analysis result by analyzing the first database table, facilitate subsequent conversion of the first database table based on the analysis result, and improve the conversion efficiency of the first database table. A conversion strategy is constructed, and each node of a syntax tree in the analysis result is converted based on the conversion strategy to obtain a conversion result. By constructing the conversion strategy in advance and calling the conversion strategy, the first query statement of each node in the syntax tree is converted into a second query statement according to the corresponding conversion strategy, without human intervention, automatic conversion is realized, and the data conversion efficiency is improved. When the conversion result is conversion success, the second database table associated with the conversion result is checked, and when the second database table passes the check, a task in the second database table is executed to obtain a data processing result. In the case where the conversion result is successful and the check result is qualified, the task in the second database table is executed. When any one of the tasks in the second database table fails, the task in the first database table can be executed, avoiding the phenomenon that any one of the tasks in the second database table fails and the task cannot be executed, and improving the fault tolerance of the executed task.

[0189] Embodiment three

[0190] Referring to Figure 3 Fig. 3 shows a structural schematic diagram of an electronic device provided in the third embodiment of the present application. In the preferred embodiment of the present application, the electronic device 3 comprises a memory 31, at least one processor 32, at least one communication bus 33, and a transceiver 34.

[0191] Those skilled in the art should understand that Figure 3 The structure of the electronic device shown in the figure does not constitute a limitation of the embodiment of the present application. It can be a bus structure or a star structure. The electronic device 3 can also include more or fewer other hardware or software, or different component arrangements.

[0192] In some embodiments, the electronic device 3 is an electronic device capable of automatically performing numerical calculation and / or information processing according to pre-set or stored instructions. Its hardware includes but is not limited to microprocessors, application-specific integrated circuits, programmable gate arrays, digital processors, and embedded devices. The electronic device 3 can also include a client device, which includes but is not limited to any electronic product that can interact with a client through a keyboard, a mouse, a remote control, a touchpad, or a voice control device, such as a personal computer, a tablet computer, a smartphone, a digital camera, etc.

[0193] It should be noted that the electronic device 3 is only an example, and other existing or future electronic products that can be adapted to the present application should also be included in the protection scope of the present application and are hereby incorporated by reference.

[0194] In some embodiments, the memory 31 is configured to store program codes and various data, such as the artificial intelligence-based data processing apparatus 20 installed in the electronic device 3, and to realize high-speed and automatic access to programs or data during the operation of the electronic device 3. The memory 31 includes a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), a one-time programmable read-only memory (OTPROM), an electrically-erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disk memories, a magnetic disk memory, a magnetic tape memory, or any other computer-readable medium capable of carrying or storing data.

[0195] In some embodiments, the at least one processor 32 can be composed of integrated circuits, such as a single packaged integrated circuit, or a plurality of packaged integrated circuits with the same or different functions, including one or more combinations of central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and various control chips. The at least one processor 32 is the control unit of the electronic device 3, which connects various components of the entire electronic device 3 through various interfaces and lines, and executes or runs programs or modules stored in the memory 31 and calls data stored in the memory 31 to perform various functions and process data of the electronic device 3.

[0196] In some embodiments, the at least one communication bus 33 is configured to realize the connection and communication between the memory 31 and the at least one processor 32.

[0197] Although not shown, the electronic device 3 can further include a power supply (such as a battery) to supply power to the various components, and optionally, the power supply can be logically connected to the at least one processor 32 via a power management device, so that the power management device can perform functions such as managing charging, discharging, and power consumption management. The power supply can also include one or more DC or AC power sources, recharging means, power failure detection circuitry, power converters or inverters, power status indicators, and any other components. The electronic device 3 can also include various sensors, Bluetooth modules, Wi-Fi modules, and the like, which are not described here.

[0198] It should be understood that the embodiments are only for illustration and are not limited in scope by the structure.

[0199] The integrated units in the form of software function modules described above can be stored in a computer-readable storage medium. The software function modules described above are stored in a storage medium and include a plurality of instructions for causing a computer device (which can be a personal computer, an electronic device, or a network device, etc.) or a processor to execute part of the method described in each embodiment of the present application.

[0200] In further embodiments, in combination Figure 2 , the at least one processor 32 can execute the operating device of the electronic device 3 and various application programs installed (such as the artificial intelligence-based data processing device 20 described above), program codes, etc., such as the various modules described above.

[0201] The memory 31 stores program codes, and the at least one processor 32 can call the program codes stored in the memory 31 to perform related functions. For example, Figure 2 The various modules described in the above are program codes stored in the memory 31 and executed by the at least one processor 32, so as to achieve the functions of the various modules to achieve the purpose of artificial intelligence-based data processing.

[0202] For example, the program codes can be divided into one or more modules / units, which are stored in the memory 31 and executed by the processor 32 to complete the present application. The one or more modules / units can be a series of computer-readable instruction segments that can complete a specific function, which are used to describe the execution process of the program codes in the electronic device 3. For example, the program codes can be divided into an acquisition module 201, an analysis module 202, a conversion module 203, a verification module 204, an execution module 205, an output module 206, and an adjustment module 207.

[0203] In one embodiment of the present application, the memory 31 stores a plurality of computer readable instructions which are executed by the at least one processor 32 to implement the function of artificial intelligence based data processing.

[0204] Specifically, the specific implementation method of the at least one processor 32 to the above instructions can refer to Figure 1 The description of the related steps in the corresponding embodiments will not be repeated here.

[0205] In several embodiments provided by the present application, it should be understood that the disclosed apparatus and method can be implemented by other means. For example, the above-described apparatus embodiment is only illustrative, for example, the division of the modules is only a logical function division, and actual implementation can have another division manner.

[0206] The modules described as separate components can or can not be physically separated, and the components displayed as modules can or can not be physical units, which can be located in one place or distributed to multiple network units. Part or all of the modules can be selected according to actual needs to achieve the purpose of the embodiment.

[0207] In addition, each functional module in each embodiment of the present application can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit. The above integrated unit can be realized in the form of hardware or in the form of hardware plus software functional module.

[0208] It is obvious for those skilled in the art that the present application is not limited to the details of the above exemplary embodiments, and the present application can be implemented in other specific forms without departing from the spirit or essential characteristics of the present application. Therefore, the embodiments should be regarded as exemplary and non-limiting, and the scope of the present application is defined by the appended claims rather than the above description, and all changes falling within the meaning and scope of the equivalent elements of the claims are intended to be included in the present application. Any reference signs in the claims should not be regarded as limiting the claims. In addition, it is obvious that the word "comprising" does not exclude other units or, singular does not exclude plural. The plurality of units or devices stated in the present application can also be realized by one unit or device through software or hardware. The words first, second, etc. are used to indicate names, and do not indicate any specific order.

[0209] Finally, it should be noted that the above examples are merely intended to illustrate the technical solutions of the present application and not to limit the present application. Although the present application has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present application can be modified or equivalently replaced without departing from the spirit and scope of the technical solutions of the present application.

Claims

1. A data processing method based on artificial intelligence, characterized in that, The method includes: In response to the received data processing request, retrieve the first database table; Parse the first database table to obtain the parsing result; A transformation strategy is constructed, and each node of the syntax tree in the parsed result is transformed based on the transformation strategy to obtain a transformation result. This includes: identifying the transformation strategy for each node of the syntax tree; transforming the first query statement of each node based on the transformation strategy for each node; when the transformation results in the second query statement of each node, the transformation result is determined to be successful, and the transformation result is associated with a second database table, wherein the second database table is obtained by writing the second query statement of each node into a preset database table; when the transformation result is not the second query statement of each node, the transformation result is determined to be a transformation failure; when the transformation result is a transformation failure, the transformation strategy is adjusted to perform a secondary transformation on the first query statement of the failed node. When the conversion result is successful, verify the second database table associated with the conversion result; When the second database table is determined to be valid, the directed acyclic graph of the second database table is output, the tasks in the second database table are executed, and the data processing results are obtained.

2. The data processing method based on artificial intelligence as described in claim 1, characterized in that, The execution of the task in the second database table to obtain the data processing results includes: When any task in the second database table fails to execute, the first database table is retrieved, and the task in the first database table is executed. When each task in the first database table is executed successfully, the data processing result is determined as the end of task execution; When any task in the first database table fails to execute, the data processing result is determined to trigger an exception handling instruction.

3. The data processing method based on artificial intelligence as described in claim 1, characterized in that, The parsing of the first database table yields the following results: Obtain the syntax rule set corresponding to the first database table; A parser is generated based on the set of grammar rules; The parser is used to parse the first query statement in the first database table to obtain the corresponding syntax elements; Determine the node position of each syntax element, and lay out each syntax element according to the node position to generate the corresponding syntax tree, and determine the syntax tree as the parsing result.

4. The data processing method based on artificial intelligence as described in claim 1, characterized in that, The construction and transformation strategy includes: Identify the differences between the first syntax of the first database table and the second syntax of the second database table; Based on the aforementioned differences, a preset function is used to construct a transformation strategy.

5. The data processing method based on artificial intelligence as described in claim 1, characterized in that, The method further includes: When it is determined that the second database table fails the verification, the first failure information of the failed node is obtained and sent to the client. Receive a new conversion strategy reported by the client based on the first failure information, and perform a secondary conversion on the first query statement of the failed node based on the new conversion strategy; When the converted result is the second query statement of the failed node, the conversion result is determined to be successful and the conversion ends. Alternatively, when the converted result is not the second query statement of the failed node, the conversion result is determined to be failed. The above steps are repeated until the first query statement of the failed node is successfully converted and the conversion ends. The second query statement of the failed node obtained from the transformation is iteratively verified until the second database table passes the verification.

6. A data processing device based on artificial intelligence, characterized in that, The device includes: The acquisition module is used to acquire the first database table in response to the received data processing request; The parsing module is used to parse the first database table and obtain the parsing result; A conversion module is used to construct a conversion strategy and convert each node of the syntax tree in the parsed result based on the conversion strategy to obtain a conversion result. This includes: identifying the conversion strategy for each node of the syntax tree; converting the first query statement of each node based on the conversion strategy for each node; when the conversion results in the second query statement of each node, determining the conversion result as successful and associating the conversion result with a second database table, wherein the second database table is obtained by writing the second query statement of each node into a preset database table; when the conversion results in a result other than the second query statement of each node, determining the conversion result as a conversion failure; when the conversion result is a conversion failure, adjusting the conversion strategy to perform a secondary conversion on the first query statement of the failed node. The verification module is used to verify the second database table associated with the conversion result when the conversion result is successful. The execution module is used to output a directed acyclic graph of the second database table when it is determined that the second database table has passed the verification, execute the tasks in the second database table, and obtain the data processing results.

7. An electronic device, characterized in that, The electronic device includes a processor and a memory, wherein the processor is configured to implement the artificial intelligence-based data processing method as described in any one of claims 1 to 5 when executing a computer program stored in the memory.

8. A computer-readable storage medium storing a computer program thereon, characterized in that, When the computer program is executed by the processor, it implements the artificial intelligence-based data processing method as described in any one of claims 1 to 5.

Images