Code text processing method and device, electronic equipment and computer storage medium

Abstract

The present disclosure provides a code text processing method and device, electronic equipment and computer storage medium, which can be applied to the field of big data technology. The method comprises: acquiring a first table creation code text and a second table creation code text, the first table creation code text being used to construct an index table based on an intermediate data table in a data warehouse, and the second table creation code text being used to construct an intermediate data table in the data warehouse based on a source data table in a source database; screening target construction data from the first table creation code text, the target construction data comprising at least one intermediate data table and M target screening conditions; determining P first execution statements from the second table creation code text according to the at least one intermediate data table; updating the P first execution statements according to a connection mode of the M target screening conditions to obtain Q second execution statements after updating; and generating a third table creation code text according to the Q second execution statements and the first table creation code text.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of big data, and more particularly to a code text processing method and device, electronic equipment and computer storage medium. BACKGROUND

[0002] An enterprise can manage a large amount of enterprise data, such as transaction data, system data, etc., through a data middle platform architecture. The data management scheme of the related art is as follows: first, intermediate data tables are generated in a data warehouse (Mppdb) through multi-layer processing based on source data of a data lake (Hadoop), and then an index table required by a business is generated according to the intermediate data tables in the data warehouse.

[0003] With the continuous development of big data technology, the data management scheme is upgraded to: generating an index table required by a business directly from a data lake.

[0004] In the process of implementing the above-mentioned inventive concept, the inventors have found that the prior art at least has the following technical problems: the generation of the index table required by the business directly from the data lake by the developer coding implementation results in low generation efficiency and high generation cost. SUMMARY

[0005] In view of the above problems, the present disclosure provides a code text processing method, device, electronic equipment and computer storage medium.

[0006] According to a first aspect of the present disclosure, a code text processing method is provided, comprising:

[0007] obtaining a first table creation code text and a second table creation code text, wherein the first table creation code text is used to construct an index table based on intermediate data tables in a data warehouse, and the second table creation code text is used to construct intermediate data tables in the data warehouse based on source data tables in a source database;

[0008] filtering target construction data from the first table creation code text, wherein the target construction data includes at least one intermediate data table and M target filtering conditions, the target filtering conditions are used to filter data in the index table, and M≥1;

[0009] determining P first execution statements from the second table creation code text according to the at least one intermediate data table, wherein the first execution statement includes an execution statement for performing a query operation and an execution statement for performing an insertion operation, and P≥1;

[0010] updating the P first execution statements according to a connection mode of the M target filtering conditions to obtain Q second execution statements after updating, and P≥Q≥1; and

[0011] According to the Q second execution statements and the first table creation code text, third table creation code text is created, and the third table creation code text is used to construct a target table based on a source data table in a source database.

[0012] According to an embodiment of the present disclosure, wherein the target construction data is filtered from the first table creation code text, comprising:

[0013] A first expression is invoked to filter at least one intermediate data table from the first table creation code text.

[0014] A second expression is invoked to obtain M target filtering conditions written in a target writing mode from the first table creation code text.

[0015] According to an embodiment of the present disclosure, wherein the second expression comprises a first sub-expression and a second sub-expression, and the target writing mode comprises a correlation writing mode and a condition writing mode.

[0016] The second expression is invoked to obtain M target filtering conditions written in a target writing mode from the first table creation code text, comprising:

[0017] The first sub-expression is invoked to obtain a first string set written in a correlation writing mode from the first table creation code text.

[0018] The second sub-expression is invoked to obtain a second string set written in a condition writing mode from the first table creation code text.

[0019] In response to detecting that the first string set and / or the second string set is non-empty, M target filtering conditions are obtained from the first string set and / or the second string set.

[0020] According to an embodiment of the present disclosure, further comprising:

[0021] A third expression is invoked to perform a nested relationship judgment on the first table creation code text to obtain a nested relationship judgment result.

[0022] In response to the nested relationship judgment result representing that there is a nested relationship in the first table creation code text, the target construction data is filtered from the first table creation code text in an order from an outermost nested layer to an innermost nested layer.

[0023] According to an embodiment of the present disclosure, wherein the P first execution statements are determined from the second table creation code text according to the at least one intermediate data table, comprising:

[0024] According to the table name of the intermediate data table, fourth table creation code text comprising the at least one intermediate data table is filtered from the second table creation code text.

[0025] Call the deletion subroutine to delete the code comment information in the fourth table creation code text, to obtain a fifth table creation code text; and

[0026] Call the fourth expression to process the fifth table creation code text, to obtain intermediate construction data, wherein the intermediate construction data includes P first execution statements.

[0027] According to an embodiment of the present disclosure, wherein the P first execution statements are updated according to the connection mode of the M target filtering conditions to obtain Q second execution statements after updating, including:

[0028] In response to detecting that the connection mode of the M target filtering conditions is AND and the connection mode, N operation conditions and N field values are parsed from the P first execution statements, wherein the field value, the field name corresponding to the field value, and the operation condition jointly constitute a complete filtering condition, N≥M≥1;

[0029] According to the M target filtering conditions, the N operation conditions, the N field values, and the field name corresponding to the field value, Q third execution statements are filtered from the P first execution statements; and

[0030] Through the target updating mode, the execution statements for performing the insertion operation in the Q third execution statements are updated to obtain the Q second execution statements after updating.

[0031] According to an embodiment of the present disclosure, wherein the N operation conditions and the N field values are parsed from the P first execution statements, including:

[0032] Call the fifth expression to obtain the nested subquery statements contained in the P first execution statements to form a nested subquery statement set;

[0033] In response to detecting that the nested subquery statement set is not empty, for each nested subquery statement in the nested subquery statement set, determine the operation condition and the field value; and

[0034] In response to detecting that the nested subquery statement set is empty, for each first execution statement, determine the operation condition and the field value.

[0035] According to an embodiment of the present disclosure, wherein for each first execution statement, determining the operation condition and the field value includes:

[0036] Call the sixth expression to determine the source data table and the position data from the first execution statement, wherein the position data is used to represent the position of the positioning word of the source data table;

[0037] Based on the position information, the field value is parsed before the positioning word;

[0038] The seventh expression is called to obtain an operation condition from the first execution statement, where the operation condition includes a comparison operation condition and a set operation condition.

[0039] According to an embodiment of the present disclosure, wherein the Q third execution statements are screened from the P first execution statements according to the M target screening conditions, the N operation conditions, the N field values and the field names corresponding to the field values, includes:

[0040] For each first execution statement,

[0041] In response to the existence of an intersection between the data set determined based on the field value in the first execution statement, the field name corresponding to the field value and the operation condition, and the data set determined according to the target screening condition, the first execution statement is determined as the third execution statement.

[0042] According to an embodiment of the present disclosure, wherein the execution statements for performing the insert operation in the Q third execution statements are updated by a target update mode to obtain the updated Q second execution statements, includes:

[0043] The format of the execution statement for performing the insert operation in the third execution statement is updated to obtain a fourth execution statement;

[0044] A field name corresponding to the field value is added after the field value in the fourth execution statement to obtain the second execution statement; or the original field name in the fourth execution statement is updated to the field name corresponding to the field value to obtain the second execution statement.

[0045] According to an embodiment of the present disclosure, wherein the third table creation code text is generated according to the Q second execution statements and the second table creation code text, includes:

[0046] For each second execution statement, in response to determining that the source data table in the second execution statement belongs to the source data table, the second execution statement is used to replace the intermediate data table in the first table creation code text to generate the third table creation code text.

[0047] The second aspect of the present disclosure provides a code text processing apparatus, including:

[0048] The obtaining module is configured to obtain a first table creation code text and a second table creation code text, where the first table creation code text is used to construct an index table based on an intermediate data table in a data warehouse, and the second table creation code text is used to construct the intermediate data table in the data warehouse based on a source data table in a source database;

[0049] The screening module is configured to screen target construction data from the first table creation code text, where the target construction data includes at least one intermediate data table and M target screening conditions, the target screening conditions are used to screen data in the index table, and M≥1.

[0050] determining, from the second table creation code text, P first execution statements from the second table creation code text according to the at least one intermediate data table, wherein the first execution statements include execution statements for performing the query operation and execution statements for performing the insert operation, and P≥1;

[0051] updating, according to a connection manner of the M target filtering conditions, the P first execution statements to obtain Q second execution statements after updating, and P≥Q≥1; and

[0052] generating, according to the Q second execution statements and the first table creation code text, a third table creation code text, wherein the third table creation code text is used to construct the index table based on a source data table in the source database.

[0053] A third aspect of the present disclosure provides an electronic device, comprising: one or more processors; a memory for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors execute the above-mentioned code text processing method.

[0054] A fourth aspect of the present disclosure further provides a computer-readable storage medium having stored executable instructions, which, when executed by a processor, cause the processor to execute the above-mentioned code text processing method.

[0055] A fifth aspect of the present disclosure further provides a computer program product comprising a computer program, which, when executed by a processor, implements the above-mentioned code text processing method.

[0056] In the embodiments of the present disclosure, since the second execution statements are obtained from the second table creation code text, and the second table creation code text can create the intermediate data table based on the source data table of the source database, after replacing the intermediate data table in the first table creation code text with the second execution statements, the index table can be created based on the source data table of the source database, without introducing the data warehouse. In the above process, the existing first table creation code text and the second table creation code text are used, the third table creation code text is obtained by modifying the first table creation code text, without manual recoding or manual transcription cost, which can reduce the generation cost of the index table and improve the generation efficiency. BRIEF DESCRIPTION OF DRAWINGS

[0057] The above and other objects, features and advantages of the present disclosure will become more apparent from the following description of embodiments of the present disclosure, taken in conjunction with the accompanying drawings, in which:

[0058] Figure 1 The application scenario of the code text processing method according to the embodiments of the present disclosure is schematically shown;

[0059] Figure 2 The following schematically shows a flow chart of a code text processing method according to an embodiment of the present disclosure;

[0060] Figure 3 The following schematically illustrates an application scenario of data management according to a specific embodiment of the present disclosure;

[0061] Figure 4 Schematically shows a flow chart of a method for screening target construction data according to an embodiment of the present disclosure;

[0062] Figure 5 Schematically shows a flowchart of a method for determining a first execution statement according to an embodiment of the present disclosure;

[0063] Figure 6 The following schematically shows a flow chart of a code text processing method according to a specific embodiment of the present disclosure;

[0064] Figure 7 Schematically shows a structural block diagram of a code text processing device according to an embodiment of the present disclosure; and

[0065] Figure 8 The block diagram schematically shows an electronic device suitable for the code text processing method according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0066] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are merely exemplary and are not intended to limit the scope of the present disclosure. In the detailed description below, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of the embodiments of the present disclosure. However, it is apparent that one or more embodiments may also be implemented without these specific details. In addition, in the following description, descriptions of well-known structures and technologies are omitted to avoid unnecessary confusion of the concepts of the present disclosure.

[0067] The terms used herein are only for describing specific embodiments and are not intended to limit the present disclosure. The terms "comprise," "include," etc. used herein indicate the presence of the features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0068] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification and should not be interpreted in an idealized or overly rigid manner.

[0069] In the case of using expressions such as "at least one of A, B, and C, etc.", it is generally intended to include any of A, B, and C alone, A and B together, A and C together, B and C together, and A, B, and C together, etc.

[0070] In the technical solutions of the present disclosure, the collection, storage, use, processing, transmission, provision, disclosure, and application of data such as user personal information comply with relevant laws and regulations, necessary security measures are taken, and do not violate public order and good customs.

[0071] The data management scheme of the related art is to first establish ten major themes, summary layer tables, and market layer tables in a data warehouse (Mppdb) based on source data of a data lake (Hadoop), and then obtain an index table containing a specific field from the data warehouse through code text. With the continuous development of big data technology, it is necessary to remove the data warehouse in the data management scheme. Therefore, it is necessary to directly establish and store the index table from the data lake based on the Hive engine and Hadoop.

[0072] In the above data management scenario, the table creation code of the data warehouse is developed using the Perl language, the table creation code of the index table is implemented using the Structured Query Language (SQL), and the table creation code of the data lake is Hive SQL. The table creation code of the data lake cannot be used to directly create the index table. Therefore, the developer can imitate the table creation logic and code statements of the ten major themes, summary layer tables, and market layer tables, as well as the table creation logic and code statements of the index table, to compile code statements for directly establishing the index table from the data lake.

[0073] However, the Perl script that needs to be translated and transcribed reaches tens of thousands, and the developer coding to directly generate the index table required by the business from the data lake will have technical problems of low generation efficiency and high generation cost.

[0074] An embodiment of the present disclosure provides a code text processing method, including: obtaining a first table creation code text and a second table creation code text, wherein the first table creation code text is used to build an indicator table based on an intermediate data table in a data warehouse, and the second table creation code text is used to build an intermediate data table in the data warehouse based on a source data table in a source database; filtering target construction data from the first table creation code text, wherein the target construction data includes at least one intermediate data table and M target filtering conditions, and the target filtering conditions are used to filter data in the indicator table, M≥1; determining P first execution statements from the second table creation code text based on at least one intermediate data table, wherein the first execution statements include execution statements for executing query operations and for executing insert operations, P≥1; updating the P first execution statements according to a connection method of the M target filtering conditions to obtain Q updated second execution statements, P≥Q≥1; and generating a third table creation code text based on the Q second execution statements and the first table creation code text, wherein the third table creation code text is used to build an indicator table based on the source data table in the source database.

[0075] Figure 1 The application scenario of the code text processing method according to the embodiment of the present disclosure is schematically shown.

[0076] like Figure 1 As shown, the application scenario 100 according to this embodiment may include a first terminal device 101, a second terminal device 102, a third terminal device 103, a network 104, and a server 105. The network 104 is used as a medium for providing a communication link between the first terminal device 101, the second terminal device 102, the third terminal device 103, and the server 105. The network 104 may include various connection types, such as wired or wireless communication links or optical fiber cables.

[0077] The user may use at least one of the first terminal device 101, the second terminal device 102, and the third terminal device 103 to interact with the server 105 via the network 104 to receive or send messages, etc. Various communication client applications may be installed on the first terminal device 101, the second terminal device 102, and the third terminal device 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc. (only as examples).

[0078] The first terminal device 101 , the second terminal device 102 , and the third terminal device 103 may be various electronic devices having display screens and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, and the like.

[0079] The server 105 can be a server providing various services, such as a background management server (only as an example) providing support for a website browsed by a user using the first terminal device 101, the second terminal device 102, and the third terminal device 103. The background management server can perform analysis and the like on received user requests and the like, and feed back the processing results (such as a webpage, information, or data, or the like, obtained or generated according to the user request) to the terminal device.

[0080] For example, the user can input first table creation code text and second table creation code text using the first terminal device 101, the second terminal device 102, and the third terminal device 103. The server 105 obtains the first table creation code text and the second table creation code text, wherein the first table creation code text is used to construct an index table based on an intermediate data table in a data warehouse, and the second table creation code text is used to construct the intermediate data table in the data warehouse based on a source data table in a source database; filters target construction data from the first table creation code text, wherein the target construction data includes at least one intermediate data table and M target filtering conditions, and the target filtering conditions are used to filter data in the index table, and M≥1; determines P first execution statements from the second table creation code text according to the at least one intermediate data table, wherein the first execution statement includes an execution statement used to perform a query operation and an execution statement used to perform an insertion operation, and P≥1; updates the P first execution statements according to a connection mode of the M target filtering conditions to obtain Q second execution statements after updating, and P≥Q≥1; and generates third table creation code text according to the Q second execution statements and the first table creation code text, wherein the third table creation code text is used to construct the index table based on the source data table in the source database.

[0081] It should be noted that the code text processing method provided by the embodiments of the present disclosure can generally be executed by the server 105. Correspondingly, the code text processing apparatus provided by the embodiments of the present disclosure can generally be arranged in the server 105. The code text processing method provided by the embodiments of the present disclosure can also be executed by a server or a server cluster different from the server 105 and capable of communicating with the first terminal device 101, the second terminal device 102, the third terminal device 103, and / or the server 105. Correspondingly, the code text processing apparatus provided by the embodiments of the present disclosure can also be arranged in a server or a server cluster different from the server 105 and capable of communicating with the first terminal device 101, the second terminal device 102, the third terminal device 103, and / or the server 105.

[0082] It should be understood that Figure 1 The number of terminal devices, networks, and servers in the system 100 is only illustrative. According to the implementation needs, there can be any number of terminal devices, networks, and servers.

[0083] The following will be based on Figure 1 The described scenario, by Figures 2 to 6 The code text processing method of the disclosed embodiment is described in detail.

[0084] Figure 2 The flowchart of the code text processing method according to the embodiments of the present disclosure is schematically shown.

[0085] As Figure 2 shown, the method 200 includes operations S210-S250.

[0086] In operation S210, a first table creation code text and a second table creation code text are acquired, wherein the first table creation code text is used to construct an index table based on an intermediate data table in a data warehouse, and the second table creation code text is used to construct the intermediate data table in the data warehouse based on a source data table in a source database.

[0087] According to the embodiments of the present disclosure, the source database can be understood as a data lake in the field of big data technology, that is, a big data architecture for carrying enterprise data such as Hadoop.

[0088] According to the embodiments of the present disclosure, the first table creation code text includes a SQL script written using a SQL language, and the second table creation code text includes a Perl script developed using a Perl language.

[0089] According to the embodiments of the present disclosure, the index table is a data table composed of data including predetermined fields or a predetermined data interval obtained according to actual business requirements.

[0090] For example, the index table can be a data table including only an account field, an account opening time, and an account opening location field, and the index table can also be a data table composed of data with an account opening time of a specific year.

[0091] In operation S220, target construction data is filtered from the first table creation code text, wherein the target construction data includes at least one intermediate data table and M target filtering conditions, the target filtering conditions are used to filter data in the index table, and M≥1.

[0092] According to the embodiments of the present disclosure, the data warehouse can include thousands of intermediate data tables, and when creating the index table according to the intermediate data tables of the data warehouse, the index table needs to be created according to the filtering conditions and the source intermediate data table. Therefore, the first table creation code text used to create the index table can include at least one intermediate data and a plurality of target filtering conditions corresponding to the at least one intermediate data table.

[0093] For example, in the case that the first table creation code text only exists one intermediate data table, the M target filter conditions are only related to the intermediate data table; in the case that the first table creation code text includes two or more intermediate data tables, M1 target filter conditions are related to the first intermediate data table, M2 target filter conditions are related to the second intermediate data table, and so on.

[0094] In operation S230, P first execution statements are determined from the second table creation code text according to the at least one intermediate data table, wherein the first execution statements include execution statements for performing a query operation and execution statements for performing an insertion operation, and P≥1.

[0095] According to an embodiment of the present disclosure, since the intermediate data tables in the first table creation code text are created by the second creation code text, for each intermediate data table, at least one first execution statement for creating the intermediate data table can be determined from the second table creation code text.

[0096] According to an embodiment of the present disclosure, the execution statement for performing the query operation can be a select statement, and the execution statement for performing the insertion operation can be an insert statement.

[0097] In operation S240, the P first execution statements are updated according to the connection mode of the M target filter conditions, to obtain Q second execution statements after updating, and P≥Q≥1.

[0098] According to an embodiment of the present disclosure, there can be multiple filter conditions in the same first execution statement, and the connection mode includes and connection mode, that is, and connection; or connection mode, that is, or connection. The and / or connection mode will affect the data range of the filter, and thus the P first execution statements are updated according to the connection mode of the M target filter conditions to obtain the Q second execution statements.

[0099] According to an embodiment of the present disclosure, updating the P first execution statements according to the connection mode of the M target filter conditions can delete irrelevant filter conditions in the first execution statement by and to obtain the second execution statement. Therefore, the number of the second execution statements is less than or equal to the number of the first execution statements.

[0100] For example, the first execution statement can be “…A and B…”. Wherein A represents a first target filter condition for filtering an index table A’, and B represents a second target filter condition for filtering an index table B’. In the case that only the index table A’ is constructed and the index table B’ is not constructed, since the and connection associates the index table B’ with the index table A’, the second execution statement can be obtained by deleting the statement of constructing the index table B’ in the first execution statement.

[0101] In operation S250, a third table creation code text is created according to the Q second execution statements and the first table creation code text, where the third table creation code text is used to build the index table based on the source data table in the source database.

[0102] According to an embodiment of the present disclosure, after determining the Q second execution statements, the intermediate data table involved in the first table creation code text can be replaced by the second execution statements to obtain the third table creation code text.

[0103] In an embodiment of the present disclosure, since the second execution statements are obtained from the second table creation code text, and the second table creation code text can create the intermediate data table based on the source data table of the source database, after replacing the intermediate data table in the first table creation code text with the second execution statements, the index table can be created based on the source data table of the source database without introducing the data warehouse. In the above process, the existing first table creation code text and the second table creation code text are used to modify the first table creation code text to obtain the third table creation code text, without manual recoding or manual transcription cost, which can reduce the generation cost of the index table and improve the generation efficiency.

[0104] Figure 3 An application scenario of data management according to an embodiment of the present disclosure is schematically shown.

[0105] As shown in Figure 3 , the application scenario 300 of data management shows the change from the original data management mode to the data management mode obtained based on the code text processing method according to an embodiment of the present disclosure.

[0106] In the Figure 3 , the original data management mode needs to establish a relationship between each data lake and the temporary area table in the data warehouse. For example, a comparison relationship is established between data lake 1 and temporary area table 1, a comparison relationship is established between data lake 2 and temporary area table 12, a comparison relationship is established between data lake 3 and temporary area table 3, a comparison relationship is established between data lake 4 and temporary area table 5, a comparison relationship is established between data lake 5 and temporary area table 15, and a comparison relationship is established between data lake 6 and temporary area table 6. In the data warehouse, a plurality of temporary area tables are aggregated into a certain topic in the top ten topics, and then one or more topics are aggregated into a summary layer table. For example, temporary area table 1 and temporary area table 2 are aggregated into top ten topic 1, temporary area table 3 and temporary area table 4 are aggregated into top ten topic 2, and temporary area table 5 and temporary area table 6 are aggregated into top ten topic 3. Top ten topic 1 and top ten topic 2 are aggregated into summary layer table 1. Summary layer table 1 and top ten topic 3 are aggregated into final market layer table 1. When creating an index table in business, the index table is created according to the market layer table 1.

[0107] In actual application scenarios, the establishment of indicator tables can only use Data Lake 1, Data Lake 4, and Data Lake 6. The data tables in other data lakes and data warehouses are all irrelevant code texts, which will lead to complex data processing steps and involve many redundant data tables.

[0108] In the data management method derived from the code text processing method according to the disclosed embodiment, since the code text for creating the third table is derived from the code text for creating the first table and the second execution statement, the generated indicator table is identical to the indicator table of the original data management method, and the replaced second execution statement retains the code logic of the data warehouse, that is, the logic of topic aggregation 1 is identical to that of summary layer table 1. Furthermore, by locating the first execution statement through the intermediate data table and updating the first execution statement to obtain the second execution statement, irrelevant Data Lake 2, Data Lake 3, and Data Lake 5 can be excluded, automatically eliminating redundant sub-tables.

[0109] Figure 4 The flowchart of the method for screening target construction data according to an embodiment of the present disclosure is schematically shown.

[0110] like Figure 4 As shown, the method 400 for screening target construction data of this embodiment includes operations S421 to S422, which can be used as a specific embodiment of operation S220.

[0111] In operation S421 , a first expression is called to filter out at least one intermediate data table from the first table creation code text.

[0112] In operation S422 , the second expression is called to obtain M target screening conditions written in the target writing mode from the first table creation code text.

[0113] According to an embodiment of the present disclosure, the first expression can be understood as a screening function or code text in the form of an expression. By calling the first expression, the execution logic defined by the first expression can be implemented to obtain a corresponding execution result. For example, the expression is a regular expression. The forms of other expressions in the present disclosure are similar to those of the first expression and are not repeated here.

[0114] According to an embodiment of the present disclosure, a process of calling a first expression to filter at least one intermediate data table from a first table creation code text includes: using the first expression to traverse and read the first table creation code text, and then obtaining the at least one intermediate data table. For example, the first expression can be r"\s+\(?\s*([0-9a-zA-Z_]+)\s+(T[0-9]+)\s+".

[0115] According to an embodiment of the present disclosure, the manner of calling other expressions in the present disclosure is similar to the manner of calling the first expression, that is, by traversing to read the corresponding table creation code text or execution statement to obtain the corresponding execution result, which will not be described herein again.

[0116] According to an embodiment of the present disclosure, the at least one intermediate table obtained by calling the first expression can be understood as a source table in the data warehouse, such as {table1, table2,...}.

[0117] According to an embodiment of the present disclosure, the second expression includes a first sub-expression and a second sub-expression, and the target writing mode includes an association writing mode and a condition writing mode. The M target filtering conditions written in the target writing mode are obtained from the first table creation code text by calling the second expression, including: calling the first sub-expression to obtain a first string set written in the association writing mode from the first table creation code text; calling the second sub-expression to obtain a second string set written in the condition writing mode from the first table creation code text; and in response to detecting that the first string set and / or the second string set is non-empty, obtaining the M target filtering conditions from the first string set and / or the second string set.

[0118] According to an embodiment of the present disclosure, the association writing mode can be a filtering condition attached by an association condition (on) in the association mode (join). The condition writing mode can be a filtering condition written by using a where clause.

[0119] For example, “user<0” after the where in “… where user<0” is a filtering condition.

[0120] In an embodiment of the present disclosure, the filtering condition written in the association writing mode is filtered out by calling the first sub-expression to form the first string set, and the filtering condition written in the condition writing mode is filtered out by calling the second sub-expression to form the second string set. The first sub-expression can be r"^(?:(?!\sSELECT\s).)+ON\s+([^;]+)\s*;?$" and the second sub-expression can be r"^(?:(?!\sSELECT\s).)+WHERE\s+([^;]+)\s*;?$".

[0121] According to an embodiment of the present disclosure, when the filtering condition written in the association writing mode can be filtered out in the first table creation code text, the first string set is non-empty; otherwise, the first string set is empty. When the filtering condition written in the condition writing mode can be filtered out in the first table creation code text, the second string set is non-empty; otherwise, the second string set is empty.

[0122] According to an embodiment of the present disclosure, in response to detecting that the first string set is non-empty, at least one filtering condition in the first string set is taken as a target filtering condition; in response to detecting that the second string set is non-empty, at least one filtering condition in the second string set is taken as a target filtering condition. Wherein, if the first string set and the second string set are both non-empty, the M target filtering conditions are derived from the first string set and the second string set. If the first string set or the second string set is non-empty, the M target filtering conditions are derived from the first string set or the second string set only. If the first string set and the second string set are both empty, there is no target filtering condition.

[0123] According to an embodiment of the present disclosure, the target filtering condition includes a field name, an operator and a field value. After determining the M target filtering conditions, a and comparison operator in each target filtering condition is determined by calling a first condition expression; and a and set operator in each target filtering condition is determined by calling a second condition expression.

[0124] According to an embodiment of the present disclosure, the and comparison operator includes greater than, less than, equal to, not equal to, etc.; and the and set operator includes belongs to, does not belong to.

[0125] According to an embodiment of the present disclosure, the first condition expression can be represented as: r"(where|and)\s+((T[0-9]+)\.(\w+)\s*([=><]{1,2})\s*([\w\-\'\"]+))[\s;]*". The second condition expression can be represented as: r"(where|and)\s+((T[0-9]+)\.([\w]+)\s*(IN|NOT\s+IN)\s*\(\s*([\w'\",\s]+)\s*\))".

[0126] Since the coding habits of different developers are different, the writing methods used by the code are also different. In an embodiment of the present disclosure, by obtaining filtering conditions of different writing modes and filtering comparison and set operators, it is possible to adapt to the first table creation code text of different writing habits, ensure the integrity of the filtering conditions, and avoid missing the filtering conditions.

[0127] According to an embodiment of the present disclosure, the method further comprises: calling a third expression to perform nested relationship judgment on the first table creation code text to obtain a nested relationship judgment result; and in response to the nested relationship judgment result representing that there is a nested relationship in the first table creation code text, constructing data from the first table creation code text in an order from the outermost nested layer to the innermost nested layer.

[0128] According to an embodiment of the present disclosure, since one SQL statement can be nested with another structured query statement (SQL statement), data query and acquisition are implemented. Therefore, by calling the third expression to determine the nested relationship, the statement relationship in the first table creation code text can be accurately split, so that the SQL statements are analyzed layer by layer, and the intermediate data table and the target filtering condition are screened.

[0129] According to an embodiment of the present disclosure, the third expression can be: r"(FROM|JOIN)\s+\(\s*([^;]+)\)\s*[a-z\s]*\s*(JOIN|ON)".

[0130] According to an embodiment of the present disclosure, in response to the nested relationship determination result representing that there is a nested relationship in the first table creation code text, the first table creation code text is split to distinguish an outer part and an inner part. Then, the target filtering condition is matched for the SQL statement in the outermost nested layer, that is, the target filtering condition written in the target writing mode is acquired from the SQL statement in the outermost nested layer by calling the third expression. The same method is used to recursively match the target filtering condition for the SQL statement in the inner part matched by the third expression, until there is no nested statement in the SQL statement in the innermost nested layer.

[0131] According to an embodiment of the present disclosure, in the process of screening the target to build data from the first table creation code text in the order from the outermost nested layer to the innermost nested layer, the target filtering condition matched in each nested layer is saved. The above-mentioned target filtering condition can be written in the association writing mode and / or the condition writing mode.

[0132] In an embodiment of the present disclosure, by calling the third expression, the target filtering condition of the nested SQL statement can be recursively screened and the target construction data can be acquired, so that the target filtering condition of the nested SQL statement can be avoided.

[0133] Figure 5 A flowchart of a method for determining a first execution statement according to an embodiment of the present disclosure is schematically shown.

[0134] As shown in FIG. 5, the method 500 for determining a first execution statement according to an embodiment of the present disclosure includes operations S531-S533, which can be one specific embodiment of operation S230. Figure 5

[0135] In operation S531, according to the table name of the intermediate data table, a fourth table creation code text including at least one intermediate data table is screened from the second table creation code text.

[0136] ​In operation S532, a deletion subprogram is invoked to delete the code comment information in the fourth table creation code text, to obtain a fifth table creation code text.

[0137] In operation S533, the fourth expression is invoked to process the fifth table creation code text, to obtain intermediate construction data, wherein the intermediate construction data comprises P first execution statements.

[0138] According to an embodiment of the present disclosure, since the second table creation code text comprises code text for creating an intermediate data table, after determining the intermediate data table from the first table creation code text, the fourth table creation code text comprising at least one intermediate data table can be filtered from the second table creation code text based on the at least one intermediate data table obtained by filtering.

[0139] According to an embodiment of the present disclosure, the table name of the intermediate data table is recorded synchronously when the intermediate data table is created, so that the fourth table creation code text can be obtained by filtering the code text comprising the table name of the at least one intermediate data table from the second table creation code text based on the table name.

[0140] According to an embodiment of the present disclosure, the code comment information added by the comment symbol is included when the second table creation code text is written. In order to further delete irrelevant code or irrelevant symbols in the fourth table creation code text, the code comment information in the fourth table creation code text can be deleted by invoking a deletion subprogram, to obtain a fifth table creation code text.

[0141] According to an embodiment of the present disclosure, the deletion subprogram can be a subprogram encapsulated, or a simple code input by an editor. For example, sub(r" / \*.*\* / ","",pls_contents).

[0142] According to an embodiment of the present disclosure, after obtaining the fifth table creation code text, the fourth expression is invoked to process the fifth table creation code text, to obtain intermediate construction data, wherein the intermediate construction data comprises P first execution statements.

[0143] According to an embodiment of the present disclosure, the fourth expression can be r"INSERT\s+INTO\s+([\w${}.]+)\s+\(([\w\s,]+)\)\s*(SELECT\s+[^;]*);", which is used to filter all statements for performing an insertion operation and a query operation from the second table creation code text. It can be understood that the present disclosure is not limited to the above embodiment, and as long as all statements for performing an insertion operation and a query operation can be obtained, the fourth expression is not limited to the above. In addition, the embodiment of the present disclosure is not limited to the specific form of other expressions, as long as the execution logic corresponding to the expression can be performed.

[0144] According to an embodiment of the present disclosure, the execution statement screened for performing the insert operation can be an insert statement.

[0145] For example, the first execution statement screened is in the form of: "Insert into tablel (fields) select Value1 From Source1 Where Condition1; Insert into table2 (fields) select Value2 From Source2 Where Condition2;...".

[0146] According to an embodiment of the present disclosure, in the process of screening the first execution statement, the table names of the temporary tables in the first execution statement, such as tablel and table2, are also recorded synchronously.

[0147] According to an embodiment of the present disclosure, in the process from the second table creation code text to the fourth table creation code text, since the corresponding script is determined only according to the intermediate data table, the program code irrelevant to the intermediate data table can be deleted from the second table creation code text, and the data amount of the processing code text is reduced.

[0148] According to an embodiment of the present disclosure, wherein the P first execution statements are updated according to the connection mode of the M target filter conditions to obtain the updated Q second execution statements, including: in response to detecting that the connection mode of the M target filter conditions is the AND connection mode, N operation conditions and N field values are parsed from the P first execution statements, wherein the field value, the field name corresponding to the field value, and the operation condition jointly constitute a complete filter condition, and N≥M≥1; Q third execution statements are screened from the P first execution statements according to the M target filter conditions, the N operation conditions, the N field values, and the field names corresponding to the field values; and execution statements for performing the insert operation in the Q third execution statements are updated by a target update mode to obtain the updated Q second execution statements.

[0149] According to an embodiment of the present disclosure, for the second table creation code text, the field name, the operation condition, and the field value can constitute a complete filter condition. However, in the second table creation code text, the field name, the operation condition, and the field value can be separated by other symbols or strings. For the target filter condition obtained from the first table creation text, the field name, the operator, and the field value in the target filter condition are connected. The operator is similar to the operation condition.

[0150] According to an embodiment of the present disclosure, each first execution statement can include at least one field name, at least one operation condition, and at least one field value.

[0151] According to an embodiment of the present disclosure, the target filter condition is compared with the field name, the operation condition and the field value obtained from the second table creation code text, and the execution statement not involved in the target filter condition is deleted from the first execution statement to obtain a third execution statement.

[0152] According to an embodiment of the present disclosure, and the connection mode is an and connection mode. The detection of whether the connection mode of the M target filter conditions is an and connection mode includes: converting all the strings in the first string set and / or the second string set matched by the first sub-expression and / or the second sub-expression into lowercase, and replacing all the target strings in the third expression with a predetermined uppercase letter. For example, the target string is \s, which is replaced with the predetermined uppercase letter "O". The count is initialized as 0, and the target filter conditions are looped one by one. When "(" is encountered, count = count + 1; when ")" is encountered, count = count - 1; when "O" is encountered and count = 0, False is returned, indicating that the target filter condition is not connected by "and". When all the loops are executed without returning False, it is determined that the target filter condition is connected by "and".

[0153] According to an embodiment of the present disclosure, further comprising: in response to detecting that the connection mode of the M target filter conditions has a non-and connection mode, taking the first execution statement as a third execution statement, P = Q; and updating the execution statements for performing the insertion operation in the Q third execution statements by the target update mode to obtain updated Q second execution statements.

[0154] It should be noted that in the case that the first string set is empty, and the second string set is empty or the target filter conditions in the second string set are not all connected by "and", the connection mode of the target filter conditions is a non-and connection mode.

[0155] In an embodiment of the present disclosure, by judging whether the SQL script is an and connection mode, different transcription strategies are executed to reduce the waste of unnecessary code resources.

[0156] According to an embodiment of the present disclosure, after calling the fourth expression to process the fifth table creation code text to obtain the intermediate construction data, further comprising: calling a field name matching expression to obtain all the field names and annotations of the temporary table in the first execution statement.

[0157] According to an embodiment of the present disclosure, the field name matching expression can be: r"\s*(\w+)\s* / \*\s*([\u4e00-\u9fa5\w]+)\s*\* / ". Since the code annotation information has been deleted by calling the deletion subprogram, at this time, the matched annotation is a Chinese annotation.

[0158] According to an embodiment of the present disclosure, the calling field name matching expression can also not find a corresponding field name, and if not found, it is skipped (equivalent to not replacing the temporary table). The next temporary table corresponding to the perl script is continued to be matched.

[0159] According to an embodiment of the present disclosure, the N operation conditions and the N field values are parsed from the P first execution statements, including: calling a fifth expression to obtain nested subquery statements contained in the P first execution statements to form a nested subquery statement set; in response to detecting that the nested subquery statement set is not empty, determining the operation condition and the field value for each nested subquery statement in the nested subquery statement set; and in response to detecting that the nested subquery statement set is empty, determining the operation condition and the field value for each first execution statement.

[0160] According to an embodiment of the present disclosure, the fifth expression is used to obtain nested subquery statements contained in the P first execution statements to form a nested subquery statement set. If the nested subquery statement set is empty, it indicates that the second table creation code text does not include nested subqueries, and the first execution statement can be directly parsed to obtain the operation condition and the field value. If the nested subquery statement set is not empty, it indicates that the second table creation code text includes nested subqueries, and the first execution statement needs to be recursively parsed. In the recursive parsing process, the operation condition and the field value are determined for each nested subquery statement at each level, and at this time, the processing of each nested subquery statement at each level is completed, and the processing of the first execution statement is also completed.

[0161] According to an embodiment of the present disclosure, the fifth expression can be r"FROM\s+\(\s*(SELECT\s+[^;]+)\)\s*([A-Z]{1}[0-9]+)".

[0162] Embodiments of the present disclosure avoid missing operation conditions and field values in the parsing process by introducing the nested judgment of the second table creation code text, that is, the perl script.

[0163] According to an embodiment of the present disclosure, determining the operation condition and the field value for each first execution statement includes: calling a sixth expression to determine a source data table and position data from the first execution statement, wherein the position data is used to represent the position of a positioning word of the source data table; based on the position information, the field value is parsed before the positioning word; and calling a seventh expression to obtain the operation condition from the first execution statement, wherein the operation condition includes a comparison operation condition and a set operation condition.

[0164] According to an embodiment of the present disclosure, the data of the temporary table in the first execution statement can be obtained from a plurality of source data tables.

[0165] According to an embodiment of the present disclosure, the sixth expression can be: r"FROM\s+([\w\.${}]+)\s+", and the locating word can be "from".

[0166] According to an embodiment of the present disclosure, the specific operation of parsing the field value can be: using the sixth expression of the pattern to match the data table of origin in the first execution statement, obtaining the position of the locating word "from" in the first execution statement, and then parsing the field value "value" before the position of the first execution statement. The specific parsing method includes: initializing count = 0, and then looping through the string between select and from by character, count = count + 1 when "(" is encountered, and count = count - 1 when ")" is encountered; when count is equal to 0 and the character is ",", it indicates that the current intercepted string is a value, which is stored in the list, and then continue to judge until the end, and store all values. It should be noted that the field value value here is in a one-to-one correspondence with the field name matched by the field name matching expression described above.

[0167] According to an embodiment of the present disclosure, the operation condition includes a comparison operation condition and a set operation condition. The comparison operation condition has the same expression form as the comparison operator described above, and includes greater than, less than, equal to, not equal to, etc. The set operation condition has the same expression form as the set operator described above, and includes belongs to and does not belong to.

[0168] According to an embodiment of the present disclosure, the seventh expression can include a third sub-expression and a fourth sub-expression. The third sub-expression is used to obtain the comparison operation condition, and the fourth sub-expression is used to obtain the set operation condition.

[0169] As a specific embodiment, the third sub-expression can be: r"(where|and)\s*([\w.]+)\s*([=><]{1,2}|LIKE|NOT\s+LIKE)\s*([\w%'\"]+)\s*". The fourth sub-expression can be: r"(where|and)\s*([\w.]+)\s*(IN|NOT\s+IN)\s*\(\s*([\w'\",\s]+)\s*\)\s*".

[0170] According to an embodiment of the present disclosure, after obtaining the comparison operation condition and the set operation condition, the comparison operation condition, the set operation condition and the entire first execution statement are stored.

[0171] According to an embodiment of the present disclosure, similar to the first table creation code text, the second table creation code text can also have nested subqueries, which can be parsed using a recursive method.

[0172] According to an embodiment of the present disclosure, in response to detecting that the set of nested subquery statements is non-empty, for each nested subquery statement in the set of nested subquery statements, an operation condition and a field value are determined. Specifically, the method of determining the operation condition and the field value for each nested subquery statement in the set of nested subquery statements is similar to the method of determining the operation condition and the field value for the first execution statement, except that the method based on recursion is parsed layer by layer.

[0173] For example, if the current first execution statement is matched to the nested subquery statement sub_sql, the first execution statement is split into three parts, sql_before, sub_sql and sql_back, according to the position of sub_sql, and the two parts sql_before and sql_back are stored. Recursive parsing is performed on sub_sql, specifically: similar to the method for the first execution statement, the sixth expression is called to determine the source data table and the position of from from the nested subquery statement sub_sql; then the string between select and from is parsed and stored. Then, similar to the method for the first execution statement, the seventh expression is called to parse and store the comparison operation condition and the set operation condition for the string after the sub_sql statement. At the end of the recursion, the value, the operation condition, sql_before and sql_back obtained by parsing each layer are obtained, and the parsing of the first execution statement including the nested subquery is completed.

[0174] According to an embodiment of the present disclosure, from P first execution statements, Q third execution statements are screened according to M target filtering conditions, N operation conditions, N field values and field names corresponding to the field values, comprising: for each first execution statement, in response to the existence of an intersection between a data set determined based on the field value in the first execution statement, the field name corresponding to the field value and the operation condition, and a data set determined according to the target filtering condition, the first execution statement is determined as a third execution statement.

[0175] For example, the form of the target filtering condition is "field operator value", where field represents the field name, operator represents the operator, and value represents the field value. For example, the target filtering condition is "x>0", "x" is the field, ">" is the operator, and "0" is the value. The operation condition parsed from the first execution statement sql_i in the second table creation code text is condition_i, and the field value is value_i. If there is no intersection between the data set determined by the target filtering condition in the SQL script and the data set determined by sql_i in the perl script, the current sql_i in the perl script is removed, otherwise it is stored.

[0176] According to an embodiment of the present disclosure, the data set selected by sql_i in the perl script is determined by value_i and condition_i, where value_i is in one-to-one correspondence with the field name, and is an equal relationship.

[0177] According to an embodiment of the present disclosure, for the convenience of description, the field value parsed from the first execution statement is taken as a first field value, the field name corresponding to the first field value is taken as a first field name, and the field value in the target filtering condition is taken as a second field value, and the field name is taken as a second field name.

[0178] According to an embodiment of the present disclosure, the method for determining whether there is an intersection between the data set determined based on the field value in the first execution statement, the field name corresponding to the field value, and the operation condition, and the data set determined according to the target filtering condition, comprises: replacing the first field name in the target filtering condition with the first field value to obtain a new target filtering condition, and determining whether there is an intersection according to the operation result returned by the new target filtering condition.

[0179] For example, if the return is true, it indicates that the first field name belongs to the data set filtered by the target filtering condition; otherwise, the first field name does not belong to the data set filtered by the target filtering condition, there is no intersection, and the current first execution statement is deleted.

[0180] For example, the first field value value_i of sql_i is 1, the first field name is replaced in the target filtering condition "x>0" to obtain "1>0", which is true, indicating that the data set of the current sql_i has an intersection with the data set filtered by the target filtering condition, and the current first execution statement sql_i is retained.

[0181] According to an embodiment of the present disclosure, after the first field name in the target filtering condition is replaced with the first field value to obtain a new target filtering condition, and whether there is an intersection is determined according to the operation result returned by the new filtering condition, it further comprises: in the case where the first field name and the second field name are determined to be the same, replacing the operator in the target filtering condition with the operation condition to obtain a new target filtering condition; and determining whether there is an intersection according to the operation result returned by the new filtering condition.

[0182] For example, the operation condition operator_i in the first execution statement is <, and the new target filtering condition after replacement is "x<0".

[0183] According to an embodiment of the present disclosure, in the case where it is determined that there is an intersection, the first execution statement is retained as a third execution statement; and in the case where it is determined that there is no intersection, the current first execution statement is deleted.

[0184] Embodiments of the present disclosure determine whether there is an intersection between the data sets determined by the perl script and the sql script, perform secondary screening, and can screen irrelevant data tables.

[0185] According to an embodiment of the present disclosure, the format of the execution statement for performing the insert operation in the third execution statement is updated to obtain a fourth execution statement, including: updating the format of the execution statement for performing the insert operation in the third execution statement to obtain the fourth execution statement; adding a field name corresponding to the field value after the field value in the fourth execution statement to obtain the second execution statement; or updating the original field name in the fourth execution statement to the field name corresponding to the field value to obtain the second execution statement.

[0186] According to an embodiment of the present disclosure, updating the format of the execution statement for performing the insert operation in the third execution statement to obtain the fourth execution statement includes: performing union on the third execution statement to obtain the fourth execution statement, and replacing the insert statement in the perl script with the union statement to establish a temporary sub-table.

[0187] According to an embodiment of the present disclosure, since the fourth execution statement in the perl script does not necessarily add a field name to all fields, even if it is added, it is inconsistent with the corresponding field_i. Therefore, the original field name in the fourth execution statement is updated to the field name corresponding to the field value to obtain the second execution statement.

[0188] For example, a corresponding field name field_i is added after each value_i in the fourth execution statement before union, or the original field name is replaced with field_i.

[0189] Embodiments of the present disclosure can ensure that the format and field name of the transcribed statement are consistent with the sql script, and improve the transcription accuracy.

[0190] According to an embodiment of the present disclosure, the third table creation code text is generated according to the Q second execution statements and the first table creation code text, including:

[0191] For each second execution statement, in response to determining that the source data table in the second execution statement belongs to the source data table, the second execution statement is used to replace the intermediate data table in the first table creation code text to generate the third table creation code text.

[0192] For example, the original first table creation code text is "select t1.x t2.y from tabel1 t1 left join table2 t2 on....", and the generated third table creation code text is "select t1.x t2.y from(select...from source1 Union select...from source2)t1 left join(select...from source3 Union select...from source4)t2 on....". That is, the source data table tabel1 is replaced by the second execution statement "(select...from source1 Union select...from source2)", and the source data table tabel2 is replaced by the second execution statement "(select...from source3 Union select...from source4)".

[0193] According to an embodiment of the present disclosure, the operation of determining that the source data table in the second execution statement belongs to the source data table: calling a data table detection expression to determine whether the source data table in the second execution statement belongs to the source data table.

[0194] According to an embodiment of the present disclosure, the data table detection expression can be: r"\s+\(?\s*(\$\{[\w]+\}\.([0-9a-zA-Z_]+))\s+(T[0-9]+)\s+".

[0195] According to an embodiment of the present disclosure, the data table detection expression can be: r"\s+\(?\s*(\$\{[\w]+\}\.([0-9a-zA-Z_]+))\s+(T[0-9]+)\s+".

[0196] According to an embodiment of the present disclosure, if there is an intermediate data table derived from the data warehouse, the second execution statement is taken as a new first execution statement, and the steps S220-S250 are repeated to perform recursion until all intermediate data tables are source data tables in the data lake.

[0197] In actual use, there is a special case that the sub-tables of the generated intermediate data table contain the table itself, at this time, an infinite recursion condition occurs. Therefore, in the case where the sub-tables of the intermediate data table include the above-mentioned intermediate data table itself, only the other sub-tables in the intermediate database are executed by the above-mentioned operations S220-S250, and the sub-tables in the intermediate data table belonging to the intermediate data table itself are not replaced.

[0198] Figure 6 A flowchart of a code text processing method according to an embodiment of the present disclosure is shown.

[0199] As shown in FIG. 6, the code text processing method 600 according to an embodiment includes operations S601-S610. Figure 6

[0200] In operation S601, a first table creation code text and a second table creation code text are obtained.

[0201] In operation S602, all intermediate data tables and all target filter conditions of all layers are filtered from the first table creation code text.

[0202] In operation S603, a plurality of first execution statements are matched from the second table creation text according to the table names of all intermediate data tables. Specifically, each intermediate data table corresponds to at least one first execution statement.

[0203] In operation S604, whether the connection mode of the target filter condition is AND connection mode or OR connection mode when there is no associated writing mode.

[0204] Specifically, when the connection mode of the target filter condition is AND connection mode or OR connection mode when there is no associated writing mode, operation S605 is entered; otherwise, operation S607 is entered. Specifically, when there is no associated writing mode, the target filter condition can only include a condition writing mode. At this time, whether the connection mode of the condition writing mode is AND connection mode or OR connection mode is determined to determine whether the target filter condition is AND connection mode or OR connection mode. When there is no associated writing mode and no condition writing mode, the corresponding connection mode is not AND connection mode or OR connection mode. When there is no associated writing mode and the connection mode of the condition writing mode has a non-AND connection mode or OR connection mode, the corresponding connection mode is also not AND connection mode or OR connection mode. When there is an associated writing mode, the default is non-AND connection mode or OR connection mode.

[0205] In operation S605, the operation conditions and field values of the plurality of first execution statements are recursively parsed.

[0206] In operation S606, the operation conditions and field values of the plurality of first execution statements are compared with the corresponding target filter conditions, and the first execution statements without intersection are deleted to obtain a plurality of third execution statements.

[0207] In operation S607, the field names in the plurality of third execution statements obtained by filtering are merged using union syntax to obtain a plurality of second execution statements. Specifically, when the result of the judgment in operation S604 is no, the first execution statement can be used as the third execution statement, and operation S607 is entered. ​

[0208] In operation S608, the corresponding source data table in the first table creation code text is replaced by the second execution statement.

[0209] In operation S609, whether the data table in the replaced second execution statement is a source data table from the data lake. Specifically, when the data table in the replaced second execution statement is a source data table from the data lake, operation S610 is entered to end. When the data table in the replaced second execution statement is not a source data table from the data lake, the replaced third table creation code text or the second execution statement is taken as a new first table creation code text, and the iteration processing is continued in operation S602.

[0210] In operation S610, the process ends.

[0211] Figure 7 A structural block diagram of a code text processing apparatus according to an embodiment of the present disclosure is schematically shown.

[0212] As shown in Figure 7 the code text processing apparatus 700 of this embodiment includes an acquisition module 710, a screening module 720, a determination module 730, an updating module 740, and a generation module 750.

[0213] The acquisition module 710 is configured to acquire a first table creation code text and a second table creation code text, where the first table creation code text is used to construct an index table based on an intermediate data table in a data warehouse, and the second table creation code text is used to construct the intermediate data table in the data warehouse based on a source data table in a source database. In an embodiment, the acquisition module 710 can be configured to perform operation S210 described above, and details are not repeated here.

[0214] The screening module 720 is configured to screen target construction data from the first table creation code text, where the target construction data includes at least one intermediate data table and M target screening conditions, the target screening conditions are used to screen data in the index table, and M≥1. In an embodiment, the determination module 730 can be configured to perform operation S220 described above, and details are not repeated here.

[0215] The determination module 730 is configured to determine P first execution statements from the second table creation code text according to the at least one intermediate data table, where the first execution statement includes an execution statement for performing a query operation and an execution statement for performing an insertion operation, and P≥1. In an embodiment, the determination module 730 can be configured to perform operation S230 described above, and details are not repeated here.

[0216] The updating module 740 is configured to update the P first execution statements according to a connection mode of the M target filtering conditions, to obtain Q second execution statements after updating, and P≥Q≥1. In an embodiment, the updating module 740 can be configured to perform the operation S240 described above, and details are not described herein again.

[0217] The generating module 750 is configured to generate third table creation code text according to the Q second execution statements and the first table creation code text, and the third table creation code text is used to construct the index table based on the source data table in the source database. In an embodiment, the generating module 750 can be configured to perform the operation S250 described above, and details are not described herein again.

[0218] According to an embodiment of the present disclosure, the filtering module 720 includes a first filtering sub-module and a second filtering sub-module.

[0219] The first filtering sub-module is configured to call the first expression to filter at least one intermediate data table from the first table creation code text. In an embodiment, the first determining sub-module can be configured to perform the operation S421 described above, and details are not described herein again.

[0220] The second filtering sub-module is configured to call the second expression to obtain the M target filtering conditions written in the target writing mode from the first table creation code text. In an embodiment, the second determining sub-module can be configured to perform the operation S422 described above, and details are not described herein again.

[0221] According to an embodiment of the present disclosure, the second expression includes a first sub-expression and a second sub-expression, and the target writing mode includes a correlation writing mode and a condition writing mode.

[0222] According to an embodiment of the present disclosure, the second filtering sub-module includes a first filtering unit, a second filtering unit and a third filtering unit.

[0223] The first filtering unit calls the first sub-expression to obtain a first string set written in the correlation writing mode from the first table creation code text.

[0224] The second filtering unit calls the second sub-expression to obtain a second string set written in the condition writing mode from the first table creation code text.

[0225] The third filtering unit obtains the M target filtering conditions from the first string set and / or the second string set in response to detecting that the first string set and / or the second string set is not empty.

[0226] According to an embodiment of the present disclosure, the code text processing apparatus 700 further includes a calling module and a recursive parsing module.

[0227] The calling module is configured to call the third expression to determine the nesting relationship of the first table creation code text, and obtain a nesting relationship determination result.

[0228] The recursive parsing module is configured to, in response to the nesting relationship determination result indicating that the nesting relationship exists in the first table creation code text, filter target construction data from the first table creation code text in an order from an outermost nesting layer to an innermost nesting layer.

[0229] According to an embodiment of the present disclosure, the determining module 730 includes a first determining sub-module, a second determining sub-module, and a third determining sub-module.

[0230] The first determining sub-module is configured to filter, according to the table name of the intermediate data table, fourth table creation code text including at least one intermediate data table from the second table creation code text. In an embodiment, the first determining sub-module can be configured to perform the operation S431 described above, and details are not repeated here.

[0231] The second determining sub-module is configured to call the root calling deletion sub-program to delete code comment information in the fourth table creation code text, and obtain fifth table creation code text. In an embodiment, the second determining sub-module can be configured to perform the operation S432 described above, and details are not repeated here.

[0232] The third determining sub-module is configured to call the fourth expression to process the fifth table creation code text, and obtain intermediate construction data, wherein the intermediate construction data includes P first execution statements. In an embodiment, the third determining sub-module can be configured to perform the operation S433 described above, and details are not repeated here.

[0233] According to an embodiment of the present disclosure, the updating module 740 includes a first updating sub-module, a second updating sub-module, and a third updating sub-module.

[0234] The first updating sub-module is configured to, in response to detecting that the connection mode of the M target filtering conditions is AND, parse N operation conditions and N field values from the P first execution statements, wherein the field value, the field name corresponding to the field value, and the operation condition together constitute a complete filtering condition, and N≥M≥1.

[0235] The second updating sub-module is configured to filter Q third execution statements from the P first execution statements according to the M target filtering conditions, the N operation conditions, the N field values, and the field names corresponding to the field values.

[0236] The third updating sub-module is configured to update, by a target updating mode, an execution statement for performing an insertion operation in the Q third execution statements, and obtain updated Q second execution statements.

[0237] According to an embodiment of the present disclosure, the first updating submodule comprises a calling unit, a first nested processing unit and a second nested processing unit.

[0238] The calling unit is configured to call the fifth expression, obtain P first execution statements containing nested subquery statements, and form a set of nested subquery statements.

[0239] The first nested processing unit is configured to, in response to detecting that the set of nested subquery statements is not empty, determine, for each nested subquery statement in the set of nested subquery statements, an operation condition and a field value.

[0240] The second nested processing unit is configured to, in response to detecting that the set of nested subquery statements is empty, determine, for each first execution statement, an operation condition and a field value.

[0241] According to an embodiment of the present disclosure, the second nested processing unit comprises a first calling subunit, a parsing unit and a second calling subunit.

[0242] The first calling subunit is configured to call the sixth expression, and determine, from the first execution statement, a source data table and position data, wherein the position data is used to represent the position of a positioning word of the source data table.

[0243] The parsing unit is configured to parse, based on the position information, the field value before the positioning word.

[0244] The second calling subunit is configured to call the seventh expression, and obtain, from the first execution statement, an operation condition, wherein the operation condition comprises a comparison operation condition and a set operation condition.

[0245] According to an embodiment of the present disclosure, the second updating submodule comprises a first updating unit.

[0246] The first updating unit is configured to, for each first execution statement, in response to determining, based on a field value in the first execution statement, a field name corresponding to the field value and an operation condition, that there is an intersection between a data set determined according to the target filtering condition and a data set determined based on the field value, the field name corresponding to the field value and the operation condition, determine the first execution statement as a third execution statement.

[0247] According to an embodiment of the present disclosure, the second updating submodule comprises a second updating unit and a third updating unit.

[0248] The second updating unit is configured to update the format of an execution statement in the third execution statement used to perform an insertion operation, to obtain a fourth execution statement.

[0249] The third updating unit is configured to add, after a field value in the fourth execution statement, a field name corresponding to the field value, to obtain the second execution statement; or update an original field name in the fourth execution statement to the field name corresponding to the field value, to obtain the second execution statement.

[0250] According to an embodiment of the present disclosure, the generating module 750 comprises a generating unit configured to, for each second execution statement, in response to determining that the origin data table in the second execution statement belongs to the source data table, replace the intermediate data table in the first table creation code text with the second execution statement, to generate a third table creation code text.

[0251] According to an embodiment of the present disclosure, any of the modules of the obtaining module 710, the screening module 720, the determining module 730, the updating module 740 and the generating module 750 can be combined in one module, or any of the modules can be split into multiple modules. Alternatively, at least part of the functions of one or more of the modules can be combined with at least part of the functions of other modules, and implemented in one module.

[0252] According to an embodiment of the present disclosure, at least one of the obtaining module 710, the screening module 720, the determining module 730, the updating module 740 and the generating module 750 can be implemented at least in part as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on chip, a system on board, a system on package, an application specific integrated circuit (ASIC), or any other reasonable manner of hardware or firmware that can be integrated or packaged with a circuit, or implemented in any one of software, hardware and firmware or in a proper combination of any of them. Alternatively, at least one of the obtaining module 710, the screening module 720, the determining module 730, the updating module 740 and the generating module 750 can be implemented at least in part as a computer program module that can perform corresponding functions when the computer program module is run.

[0253] Figure 8 A block diagram of an electronic device suitable for the code text processing method according to an embodiment of the present disclosure is schematically shown.

[0254] As shown in Figure 8 The electronic device 800 according to an embodiment of the present disclosure comprises a processor 801 that can perform various appropriate actions and processes according to programs stored in a read-only memory (ROM) 802 or loaded from a storage portion 808 into a random access memory (RAM) 803. The processor 801 can comprise, for example, a general-purpose microprocessor (such as a CPU), an instruction set processor and / or a related chipset, and / or a special-purpose microprocessor (such as an application specific integrated circuit (ASIC)), and the like. The processor 801 can further comprise an on-board memory for cache use. The processor 801 can comprise a single processing unit or multiple processing units for performing different actions of the method processes according to embodiments of the present disclosure.

[0255] In the RAM 803, various programs and data required for the operation of the electronic device 800 are stored. The processor 801, the ROM 802, and the RAM 803 are connected to each other via the bus 804. The processor 801 performs various operations of the method flow according to the embodiments of the present disclosure by executing the programs in the ROM 802 and / or the RAM 803. It should be noted that the programs can also be stored in one or more memories other than the ROM 802 and the RAM 803. The processor 801 can also perform various operations of the method flow according to the embodiments of the present disclosure by executing the programs stored in the one or more memories.

[0256] According to an embodiment of the present disclosure, the electronic device 800 can further include an input / output (I / O) interface 805, which is also connected to the bus 804. The electronic device 800 can further include one or more of the following components connected to the input / output I / O interface 805: an input part 806 including a keyboard, a mouse, and the like; an output part 807 including a display such as a cathode ray tube (CRT), a liquid crystal display (LCD), and the like, and a speaker, and the like; a storage part 808 including a hard disk, and the like; and a communication part 809 including a network interface card such as a LAN card, a modem, and the like. The communication part 809 performs communication processing via a network such as the Internet. A drive 810 is also connected to the I / O interface 805 as necessary. A removable medium 811 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, and the like is mounted on the drive 810 as necessary, so that a computer program read therefrom is installed in the storage part 808 as necessary.

[0257] The present disclosure also provides a computer readable storage medium, which can be included in the device / apparatus / system described in the above embodiments; or can exist separately without being assembled into the device / apparatus / system. The above computer readable storage medium carries one or more programs, when the one or more programs are executed, the method according to the embodiments of the present disclosure is implemented.

[0258] According to an embodiment of the present disclosure, the computer readable storage medium can be a nonvolatile computer readable storage medium, for example, can include, but is not limited to, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a Read Only Memory (ROM), an Erasable Programmable Read Only Memory (EPROM or Flash memory), a portable compact disc read only memory (CD-ROM), an optical storage, a magnetic storage, or any suitable combination of the foregoing. In the present disclosure, a computer readable storage medium can be any tangible medium that contains or stores a program that can be used by or in connection with an instruction execution system, apparatus, or device. For example, according to an embodiment of the present disclosure, the computer readable storage medium can include the ROM 802 and / or the RAM 803 described above and / or one or more memory other than the ROM 802 and the RAM 803.

[0259] Embodiments of the present disclosure also include a computer program product that includes a computer program containing program codes for executing the methods shown in the flowcharts. When the computer program product is run in a computer system, the program codes are used to make the computer system implement the above-mentioned methods provided by the embodiments of the present disclosure.

[0260] The above-mentioned functions defined in the system / device of the embodiments of the present disclosure are performed when the computer program is executed by the processor 801. According to an embodiment of the present disclosure, the above-mentioned system, device, module, unit, etc. can be implemented by computer program modules.

[0261] In one embodiment, the computer program can rely on a tangible storage medium such as an optical storage medium, a magnetic storage medium, etc. In another embodiment, the computer program can also be transmitted, distributed, and downloaded in the form of a signal on a network medium and installed and downloaded through the communication part 809 and / or installed from the detachable medium 811. The program codes contained in the computer program can be transmitted by any appropriate network medium, including but not limited to wireless, wired, etc., or any suitable combination of the foregoing.

[0262] In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 809 and / or installed from the detachable medium 811. When the computer program is executed by the processor 801, the above-mentioned functions defined in the system of the embodiments of the present disclosure are performed. According to an embodiment of the present disclosure, the above-mentioned system, device, apparatus, module, unit, etc. can be implemented by computer program modules.

[0263] According to embodiments of the present disclosure, program code of the computer programs provided by embodiments of the present disclosure can be written in any combination of one or more programming languages, and specifically, these computer programs can be implemented using a high-level procedural and / or object-oriented programming language, and / or an assembly / machine language. The programming language includes, but is not limited to, a programming language such as Java, C++, Python, "C" language, or a similar programming language. The program code can be executed entirely on a user computing device, partially on a user device, partially on a remote computing device, or entirely on a remote computing device or server. In the case involving a remote computing device, the remote computing device can be connected to the user computing device through any kind of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (for example, connected through the Internet by using an Internet service provider).

[0264] The flow diagrams and the block diagrams in the drawings are illustrations of possible architectures, functions, and operations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flow diagrams or block diagrams can represent a module, a segment, or a portion of code, which comprises one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions noted in the blocks can occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and / or flow diagrams, and combinations thereof, can be implemented by special purpose hardware-based systems that perform the specified functions or operations, or combinations of special purpose hardware and computer instructions.

[0265] Those skilled in the art can understand that the features described in various embodiments of the present disclosure and / or claims can be combined or / and integrated, even if such combinations or integrations are not explicitly described in the present disclosure. In particular, the features described in various embodiments of the present disclosure and / or claims can be combined and / or integrated in various combinations, without departing from the spirit and teachings of the present disclosure. All such combinations and / or integrations are within the scope of the present disclosure.

[0266] The specific embodiments described above are intended to be illustrative of the present disclosure and its best mode known to the inventors at the time of filing the application. It is understood that the present disclosure is not limited to the specific embodiments described above, but includes any and all alternatives falling within the scope of the present disclosure.

Claims

1. A method for processing code texts, comprising: obtaining a first table creation code text and a second table creation code text, wherein the first table creation code text is used to build a metric table based on intermediate data tables in a data warehouse, and the second table creation code text is used to build the intermediate data tables in the data warehouse based on source data tables in a source database; screening target construction data from the first table creation code text, wherein the target construction data comprises at least one intermediate data table and M target screening conditions, the target screening conditions are used to screen data in the metric table, and M≥1; determining P first execution statements from the second table creation code text according to at least one intermediate data table, wherein the first execution statements comprise execution statements for performing query operations and for performing insertion operations, and P≥1; updating P first execution statements according to a connection mode of M target screening conditions to obtain updated Q second execution statements, P≥Q≥1; and generating a third table creation code text according to Q second execution statements and the first table creation code text, wherein the third table creation code text is used to build the metric table based on the source data tables in the source database.

2. The method of claim 1, wherein, The screening of the target construction data from the first table creation code text comprises: calling a first expression to screen at least one intermediate data table from the first table creation code text; calling a second expression to obtain M target screening conditions written in a target writing mode from the first table creation code text.

3. The method of claim 2, wherein, The second expression comprises a first sub-expression and a second sub-expression, and the target writing mode comprises a correlation writing mode and a condition writing mode. The calling of the second expression to obtain M target screening conditions written in a target writing mode from the first table creation code text comprises: calling the first sub-expression to obtain a first string set written in a correlation writing mode from the first table creation code text; calling the second sub-expression to obtain a second string set written in a condition writing mode from the first table creation code text; in response to detecting that the first string set and / or the second string set is non-empty, obtaining M target screening conditions from the first string set and / or the second string set.

4. The method of any one of claims 1-3, further comprising: calling a third expression to perform a nested relationship judgment on the first table creation code text to obtain a nested relationship judgment result; in response to the nested relationship judgment result representing that there is a nested relationship in the first table creation code text, screening the target construction data from the first table creation code text in order from an outermost nested layer to an innermost nested layer.

5. The method of claim 1, wherein, The determination of P first execution statements from the second table creation code text according to at least one intermediate data table comprises: filtering, from the second table creation code text, fourth table creation code text including at least one of the intermediate data tables according to the table name of the intermediate data table; calling a deletion subprogram to delete code comment information in the fourth table creation code text to obtain fifth table creation code text; and calling a fourth expression to process the fifth table creation code text to obtain intermediate construction data, wherein the intermediate construction data includes P first execution statements.

6. The method of claim 5, wherein, The updating of the P first execution statements according to the connection mode of the M target filtering conditions includes: in response to detecting that the connection mode of the M target filtering conditions is and the connection mode, N operation conditions and N field values are parsed from the P first execution statements, wherein the field values, the field names corresponding to the field values, and the operation conditions together constitute a complete filtering condition, N≥M≥1; filtering, from the P first execution statements, Q third execution statements according to the M target filtering conditions, the N operation conditions, the N field values, and the field names corresponding to the field values; and updating, through a target updating mode, execution statements for performing an insertion operation in the Q third execution statements to obtain the Q second execution statements after updating.

7. The method of claim 6, wherein, The parsing of the N operation conditions and the N field values from the P first execution statements includes: calling a fifth expression to obtain nested subquery statements included in the P first execution statements to form a nested subquery statement set; in response to detecting that the nested subquery statement set is not empty, determining the operation condition and the field value for each nested subquery statement in the nested subquery statement set; and in response to detecting that the nested subquery statement set is empty, determining the operation condition and the field value for each first execution statement.

8. The method of claim 7, wherein, The determination of the operation condition and the field value for each first execution statement includes: calling a sixth expression to determine a source data table and position data from the first execution statement, wherein the position data is used to represent the position of a positioning word of the source data table; based on the position information, the field value is parsed before the positioning word; calling a seventh expression to obtain the operation condition from the first execution statement, wherein the operation condition includes a comparison operation condition and a set operation condition.

9. The method of claim 6, wherein, The filtering of the Q third execution statements from the P first execution statements according to the M target filtering conditions, the N operation conditions, the N field values, and the field names corresponding to the field values includes: for each first execution statement, in response to a data set determined based on the field value in the first execution statement, the field name corresponding to the field value, and the operation condition and a data set determined according to the target filtering condition having an intersection, the first execution statement is determined as the third execution statement.

10. The method of claim 6, wherein, The updating the execution statements for performing the insert operation in the third execution statements by the target updating mode comprises: updating a format of the execution statement for performing the insert operation in the third execution statement to obtain a fourth execution statement; adding a field name corresponding to the field value after the field value in the fourth execution statement to obtain the second execution statement, or updating an original field name in the fourth execution statement to the field name corresponding to the field value to obtain the second execution statement.

11. The method of claim 1, wherein, The generating the third table creation code text according to the Q second execution statements and the first table creation code text comprises: for each second execution statement, in response to determining that a source data table in the second execution statement belongs to the source data table, replacing the intermediate data table in the first table creation code text with the second execution statement to generate the third table creation code text.

12. A code text processing apparatus, comprising: an obtaining module configured to obtain a first table creation code text and a second table creation code text, wherein the first table creation code text is used to construct an index table based on an intermediate data table in a data warehouse, and the second table creation code text is used to construct the intermediate data table in the data warehouse based on a source data table in a source database; a screening module configured to screen target construction data from the first table creation code text, wherein the target construction data comprises at least one intermediate data table and M target screening conditions, the target screening conditions are used to screen data in the index table, and M≥1; a determining module configured to determine P first execution statements from the second table creation code text according to at least one intermediate data table, wherein the first execution statements comprise execution statements for performing a query operation and execution statements for performing an insert operation, and P≥1; an updating module configured to update P first execution statements according to a connection mode of M target screening conditions to obtain updated Q second execution statements, P≥Q≥1; and a generating module configured to generate a third table creation code text according to Q second execution statements and the first table creation code text, wherein the third table creation code text is used to construct the index table based on the source data table in the source database.

13. An electronic device, comprising: one or more processors; a storage device for storing one or more programs, wherein the one or more programs, when executed by the one or more processors, enable the one or more processors to perform the method according to any one of claims 1-11.

14. A computer-readable storage medium having stored thereon executable instructions that, when executed by a processor, cause the processor to perform the method according to any one of claims 1-11.

15. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1-11.

Images