Data processing method and device, electronic equipment and storage medium

Abstract

The invention discloses a data processing method and device, electronic equipment and a storage medium. Comprising the steps that to-be-processed data is acquired, a target data set is constructed based on the to-be-processed data, and the target data set comprises multiple rows of statements; statement identification information corresponding to each line of statements in the target data set is determined, and the statement identification information comprises statement start identification information and statement end identification information; on the basis of the statement starting identification information and the statement ending identification information corresponding to each line of statements, carrying out blocking processing on the multiple lines of statements in the target data set to obtain at least one statement block; and for each statement block, extracting at least one table name in the statement block, and constructing a table association relationship based on the at least one table name in the statement block. According to the method, the statement start identification information and the statement end identification information of each statement are determined, the multiple lines of statements are subjected to blocking processing, so that the table association relationship in each statement block is determined, and the accuracy of determining the table association relationship is improved.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a data processing method, device, electronic equipment and storage medium. Background Art

[0002] In the core systems of financial institutions, there are still a large number of packages that use Oracle database as the main business processing procedures. In the transformation of information innovation, financial institutions hope to replace the database in parallel, but do not want to make a lot of modifications to the JAVA application system. When a centralized database is converted to a distributed database, table sharding design is involved, and the specific design needs to be determined based on the current business characteristics. Strategies cannot be adopted arbitrarily, otherwise it will bring risks to system performance. The obvious characteristics of these systems are that there are a large number of large tables and multi-table associations, and the associated fields cannot be guaranteed to be the same. The core systems of financial institutions are often likely to have thousands or even tens of thousands of tables. Each table as a multi-node replication table will lead to a sharp increase in storage overhead, which will bring severe challenges to the storage and cost of a single host. Therefore, we need to find out the association relationship between the tables. Based on this relationship, the large table can be split into different groups, so that different groups are distributed in different shards. Accurately identifying the table association relationship in the source code data is an urgent problem to be solved. Summary of the invention

[0003] The present invention provides a data processing method, device, electronic device and storage medium to solve the problem of poor accuracy of association relationships in an automatic recognition table.

[0004] According to one aspect of the present invention, there is provided a data processing method, comprising:

[0005] Acquire the data to be processed, and construct a target data set based on the data to be processed, wherein the target data set includes multiple lines of statements;

[0006] Determine the statement identification information corresponding to each line of the statement in the target data set, wherein the statement identification information includes statement start identification information and statement end identification information;

[0007] Based on the statement start identification information and the statement end identification information respectively corresponding to each line of the statement, multiple lines of statements in the target data set are processed into blocks to obtain at least one statement block, wherein the statement block includes at least one line of statement;

[0008] For each statement block, at least one table name in the statement block is extracted, and a table association relationship is established based on the at least one table name in the statement block.

[0009] Optionally, constructing a target data set based on the data to be processed includes: obtaining preset filtering conditions, filtering the data to be processed based on the preset filtering conditions to obtain filtered data, and constructing an initial data set based on the filtered data, wherein the preset filtering conditions include a regular expression that matches a keyword in at least one function parameter and a regular expression that matches a preset character; eliminating set type statements in the initial data set to obtain a target data set, wherein the set type statements include set characters.

[0010] Optionally, filtering the data to be processed based on preset filtering conditions to obtain filtered data includes: constructing corresponding data filtering operation statements based on one or more regular expressions in the preset filtering conditions, and processing the data to be processed based on the data filtering operation statements to obtain filtered data.

[0011] Optionally, determining statement identification information corresponding to each line of statements in the target data set includes: obtaining a preset keyword set, and setting statement start identification information for each line of statements in the target data set based on the preset keyword set; obtaining a preset character set, and setting statement end identification information for each line of statements in the target data set based on the preset character set.

[0012] Optionally, setting statement start identification information for each line of statement in the target data set based on a preset keyword set includes:

[0013] At least one regular expression is constructed based on one or more items in a preset keyword set, and a statement start position marking operation statement is constructed based on at least one regular expression corresponding to the preset keyword set; the statement start position marking operation statement is executed to determine statement start identification information of each line of statements in a target data set.

[0014] Optionally, statement end identification information is set for each line of statements in the target data set based on a preset character set, including: constructing at least one regular expression based on one or more items in the preset character set, constructing a statement end position mark operation statement based on at least one regular expression corresponding to the preset character set, wherein the preset character set includes an end keyword and an end symbol; executing the statement end position mark operation statement to determine the statement end identification information of each line of statements in the target data set.

[0015] Optionally, the target data set also includes statement line numbers corresponding to each line of statements; based on the statement start identification information and statement end identification information corresponding to each line of statements, multiple lines of statements in the target data set are divided into blocks to obtain at least one statement block, including: traversing the statement start identification information and statement end identification information corresponding to each line of statements in ascending order of the statement line numbers of each line of statements; determining the statement line with the statement start identification information as the start identifier as the start statement line; determining the end statement line that is located after the start statement line and has the statement end identification information as the end identifier, and treating the statements between the start statement line and the end statement line as a statement block, wherein the end statement line is the statement line with the smallest line number interval between it and the start statement line.

[0016] According to another aspect of the present invention, there is provided a data processing device, comprising:

[0017] A target data set determination module is used to obtain the data to be processed and construct a target data set based on the data to be processed, wherein the target data set includes multiple lines of statements;

[0018] A statement identification information determination module is used to determine the statement identification information corresponding to each line of statement in the target data set, wherein the statement identification information includes statement start identification information and statement end identification information;

[0019] A statement block determination module, used for performing block processing on multiple lines of statements in a target data set based on statement start identification information and statement end identification information respectively corresponding to each line of statement, to obtain at least one statement block, wherein a statement block includes at least one line of statement;

[0020] The table association relationship building module is used to extract at least one table name in each sentence block, and build a table association relationship based on the at least one table name in the sentence block.

[0021] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0022] at least one processor; and

[0023] a memory communicatively connected to at least one processor; wherein,

[0024] The memory stores a computer program that can be executed by at least one processor. The computer program is executed by at least one processor so that the at least one processor can execute the data processing method of any embodiment of the present invention.

[0025] According to another aspect of the present invention, a computer-readable storage medium is provided. The computer-readable storage medium stores computer instructions, and the computer instructions are used to implement the data processing method of any embodiment of the present invention when executed by a processor.

[0026] The technical solution of the embodiment of the present invention is to obtain data to be processed, build a target data set based on the data to be processed, wherein the target data set includes multiple lines of statements; determine statement identification information corresponding to each line of statement in the target data set, wherein the statement identification information includes statement start identification information and statement end identification information; based on the statement start identification information and statement end identification information corresponding to each line of statement, the multiple lines of statements in the target data set are processed in blocks to obtain at least one statement block, wherein the statement block includes at least one line of statements; for each statement block, extract at least one table name in the statement block, and build a table association relationship based on at least one table name in the statement block. This solution processes the data to be processed to obtain the corresponding target data set, identifies each statement in the target data set, determines the statement start identification information and statement end of each statement, and then performs statement block processing according to the statement start identification information and statement end of each statement, and then identifies the table name in the statement block to accurately obtain the table with an associated relationship, thereby solving the problem of being unable to accurately determine the data table with an associated relationship in the data to be processed, improving the accuracy of determining the table name in the data to be processed and the accuracy of determining the table association relationship, thereby helping to improve the work efficiency of converting a centralized database to a distributed database.

[0027] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0029] Figure 1 is a flow chart of a data processing method provided by Embodiment 1 of the present invention;

[0030] Figure 2 is a flow chart of a data processing method provided by Embodiment 2 of the present invention;

[0031] Figure 3 is a structural schematic diagram of a data processing device provided in Embodiment 3 of the present invention;

[0032] Figure 4 It is a structural schematic diagram of an electronic device for implementing the data processing method of an embodiment of the present invention. DETAILED DESCRIPTION

[0033] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0034] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0035] Embodiment 1

[0036] Figure 1 1 is a flowchart of a data processing method provided by Embodiment 1 of the present invention. This embodiment is applicable to the case of determining table names and / or table association relationships. The method can be executed by a data processing device. The data processing device can be implemented in the form of hardware and / or software. The data processing device can be configured in electronic devices such as computers and servers. Figure 1 As shown, the method includes:

[0037] S110: Acquire the data to be processed, and construct a target data set based on the data to be processed, wherein the target data set includes multiple lines of statements.

[0038] Among them, the data to be processed can be specifically understood as source code data containing a data table name. In this embodiment, the data to be processed can be program source codes corresponding to multiple PACKAGEs in the dictionary view SCHEMA in the database resources. The DBA_SOURCE view in the database can be queried through a view query statement in the database to obtain static source code resources under at least one SCHEMA in the DBA_SOURCE view. The obtained static source code is used as the data to be processed. The corresponding query conditions can be set according to the actual situation to query part or all of the source code resources under one or more SCHEMAs. The target data set can be specifically understood as a data set constructed from multiple necessary field information extracted from the data to be processed. The multiple necessary fields include but are not limited to PACKAGE package name, type, statement line text and statement line number. For example, the PACKAGE name represents the package name, and the source code represented by the type belongs to the source code of the definition package or the source code in the package body. For example, the field data corresponding to the type is "PACKAGE BODY", which means that the code statement representing the current line belongs to the source code in the package body, the statement line text refers to each line of statement text in the source code, and the statement line number refers to the line number in the program source code of the PACKAGE where the statement line text is located. For example, a temporary data table can be created to store multiple field information in the target data set to facilitate subsequent information query.

[0039] Specifically, by retrieving the DBA_SOURCE view in the database, the source code of type PACKAGE BODY under the target user is obtained, and then the preset field information is extracted from the source code. The preset field information includes but is not limited to the PACKAGE package name, type, statement line text and statement line number. The target data set is constructed according to the extracted preset field information. By setting the filtering conditions and screening conditions, the code statements in each line of the source code are preliminarily filtered and screened according to the filtering conditions and screening conditions to filter out unnecessary information and screen out necessary field information to obtain the target data set for subsequent table name recognition processing. Exemplarily, the unnecessary screening information includes but is not limited to the keyword information in the comment line statement and the function actual parameter. The keyword information here includes but is not limited to SELECT, UPDATE, DELETE and MERGE, and the keyword information can be modified according to the actual filtering requirements. The necessary field information includes but is not limited to the package name, type, statement line text and statement line number, which is used for subsequent group query to accurately identify the table names with associated relationships in each PACKAGE, and the necessary field information can be modified according to the actual filtering and screening requirements.

[0040] Optionally, constructing a target data set based on the data to be processed includes: obtaining preset filtering conditions, filtering the data to be processed based on the preset filtering conditions to obtain filtered data, and constructing an initial data set based on the filtered data, wherein the preset filtering conditions include a regular expression that matches a keyword in at least one function parameter and a regular expression that matches a preset character; eliminating set type statements in the initial data set to obtain a target data set, wherein the set type statements include set characters.

[0041] Among them, the preset filtering condition can be specifically understood as a condition pre-set for filtering out interference data, and the preset filtering condition includes but is not limited to a regular expression matching a keyword in at least one function parameter and a regular expression matching a preset character. It should be noted that the keyword in the function parameter interferes with the subsequent determination of the start position and the end position of the statement. For example, in log printing functions such as DEBUG and PRINT, the statements containing the keywords in the function parameters can be filtered out first, and the corresponding regular expression is specifically constructed through one or more keywords in the function parameters to filter out the statements containing the keywords in the function parameters. The keywords of the function parameters in the source code include but are not limited to SELECT, UPDATE, DELETE and MERGE. According to the filtering requirements of the keywords of the function parameters, one or more corresponding regular expressions are constructed to filter out multiple keywords in the parameters of different functions as completely as possible, thereby improving the accuracy of the filtering process; the regular expression matching the preset characters specifically represents the expression constructed according to the preset characters and the function, which is an expression for filtering out the preset characters, wherein the preset characters include but are not limited to the comment characters and the INTO characters, and the comment characters include "-", " / *" and "* / ". The setting character specifically represents the start comment character and the comment end character, and the setting character statement represents the statement containing the start comment character and the comment end character, and is used to further remove the statement containing the comment information.

[0042] Specifically, in the process of determining the target data set, the corresponding preset filtering conditions are matched according to the data filtering requirements, and then the data to be processed is filtered according to the preset filtering conditions. Optionally, the data to be processed is filtered based on the preset filtering conditions to obtain the filtered data, including: constructing a corresponding data filtering operation statement based on one or more regular expressions in the preset filtering conditions, and processing the data to be processed based on the data filtering operation statement to obtain the filtered data. Specifically, one or more regular expressions in the preset filtering conditions are constructed into a corresponding data filtering operation statement according to the construction rules of the query statement, and then the data filtering operation statement is executed to filter the data to be processed to obtain the filtered data, and the initial data set is constructed through the field information in the filtered data. Further, the set type statements in the initial data set are eliminated to obtain the target data set, wherein the set type statements include set characters. By identifying whether there are set characters in each line of statement text in the initial data set, the statement text containing the set characters is eliminated from the initial data set to obtain the target data set that does not contain comment statements. In this embodiment, during the process of filtering the data to be processed, the set characters can be identified and marked for each code statement text, and similarly processed by setting corresponding identification and marking statements, corresponding identifiers can be set for each line of statement text in the initial data set. For example, if there is a start comment character in the statement text, the marker is set to 1, if there is an end comment character, the marker is set to -1, and if there is no start comment character and comment end character, the marker is set to 0, and then a corresponding filtering statement is created according to the marker, and the filtering statement is executed to eliminate the set type statements contained in the initial data set to obtain the target data set that does not contain the set type statements.

[0043] In this embodiment, the data to be processed is filtered and eliminated to filter out comment statements and statements containing keywords in function parameters, and then the setting type statements are further eliminated. This method of filtering and screening data multiple times is implemented to eliminate as much interference information as possible, such as comment information in the data to be processed and keywords in function parameters, thereby providing a reliable data basis for subsequent accurate identification of table names, and helping to improve the accuracy of table name recognition and extraction.

[0044] S120: Determine statement identification information corresponding to each line of statement in the target data set, wherein the statement identification information includes statement start identification information and statement end identification information.

[0045] Among them, the statement identification information can be specifically understood as the identification information set to characterize whether each line of statements is the statement start line or the statement end line of the statement block. The statement identification information includes statement start identification information and statement end identification information. Specifically, the characteristic information of the statement block start position and the statement block end position can be determined according to the database program development rules, and then the corresponding identification judgment statement is set according to the characteristic information to determine the statement identification information corresponding to each line of statements. Exemplarily, the starting position of the statement block of the source code in the PACKAGE package body starts with a keyword, where the keywords include but are not limited to SELECT, UPDATE, DELETE, CURSOR...FOR and FOR...IN, and the corresponding regular expression can be constructed based on the above keywords to determine whether each line of statements is at the beginning of the statement block; the characteristic information of the end position of the statement block includes a statement end symbol and a statement end condition, and the statement end condition is a condition determined by the characteristics of the statement end position. Exemplarily, the statement end condition can end with a LOOP string, can contain an end symbol ";" or a UNION string, and the previous line of statements includes keywords such as SELECT, INSERT, UPDATE, DELETE, MERGE, FROM, WHERE, GROUP BY, ORDER BY, HAVING and AND, and the previous line of statements does not contain FOR...UPDATE, etc., and the statement end condition that needs to be satisfied in the judgment statement line is specifically set according to the actual situation. A data column corresponding to the statement identification information can be added to the target data set, and preferably, two columns of field data can be added to store the statement start identification information and the statement end identification information respectively.

[0046] Specifically, by calling the pre-built start identification judgment statement corresponding to the statement start identification information and the end identification judgment statement corresponding to the statement end identification information, each line of statement text in the target data set is judged to obtain the statement identification information of each line of statement text. For each line of statement text, if the result returned by the start identification judgment statement is true after the line of statement text is processed, the statement start identification information of the line of statement is set to true, otherwise it is set to false or empty. If the result returned by the end identification judgment statement is true after the line of statement text is processed, the statement end identification information of the line of statement is set to true, otherwise it is set to false or empty. After traversing each line of statement in the target data set and processing according to the above judgment method, the statement identification information corresponding to each line of statement is marked.

[0047] In this embodiment, by identifying each line of statements in the target data set, statement identification information corresponding to each line of statements is obtained, including statement start identification information and statement end identification information, thereby providing a reliable data basis for subsequent rapid and accurate determination of statement blocks and improving the accuracy of splitting statement blocks.

[0048] S130. Based on the statement start identification information and the statement end identification information corresponding to each line of statement, multiple lines of statements in the target data set are processed into blocks to obtain at least one statement block, wherein the statement block includes at least one line of statement.

[0049] In this embodiment, the statement start identification information includes a start marker and a non-start marker. The start marker can be set to 1 to represent the start line of the corresponding statement behavior statement block, and the non-start marker can be set to 0 or empty to represent that the corresponding statement line is not the start line of the statement block. The statement end identification information includes an end marker and a non-end marker. The end marker can be set to 1 to represent the end line of the corresponding statement behavior statement block, and the non-start marker can be set to 0 or empty to represent that the corresponding statement line is not the end line of the statement block.

[0050] Specifically, multiple lines of statements in a target data set are processed in blocks by identifying the statement start identification information and the statement end identification information corresponding to each line of statements. The target data set can be first arranged in descending order according to the row number, and then the statement start identification information and the statement end identification information corresponding to each line of statements are identified in turn. When the first statement whose statement start identification information satisfies a preset statement start identification is identified, it is determined to be the first start statement, and the next line of statements is identified. If the statement start identification information and the statement end identification information of the line of statements do not satisfy the corresponding preset statement start identification and preset statement end identification, the next line of statements is identified. If a statement line whose statement end identification information satisfies a preset statement end identification is identified, the statement line is determined as the first end statement corresponding to the first start statement. Exemplarily, if the statement start identification information of any statement line is identified as 1, the statement line is determined to be the statement start line, and the next statement line is identified. When a statement whose statement end identification information satisfies a preset statement end identification is identified for the first time, the currently identified statement line is determined to be the statement end line, and the statements between the previously identified start statement line are regarded as a statement block. According to the above method, the statements between each start statement line and the next statement end line are determined to be a statement block, and the statement block includes at least one statement line.

[0051] In this embodiment, multiple lines of statements in the target data set are divided into blocks using the statement start identification information and statement end identification information corresponding to each line of statements. This allows for accurate acquisition of multiple statement blocks corresponding to the target data set, without splitting empty statement lines or other statement lines with interference information into statement blocks. This improves the accuracy of determining statement blocks, and helps to improve the accuracy of subsequent identification of table names.

[0052] Optionally, the target data set also includes statement line numbers corresponding to each line of statements; based on the statement start identification information and statement end identification information corresponding to each line of statements, multiple lines of statements in the target data set are divided into blocks to obtain at least one statement block, including: traversing the statement start identification information and statement end identification information corresponding to each line of statements in ascending order of the statement line numbers of each line of statements; determining the statement line with the statement start identification information as the start identifier as the start statement line; determining the end statement line that is located after the start statement line and has the statement end identification information as the end identifier, and treating the statements between the start statement line and the end statement line as a statement block, wherein the end statement line is the statement line with the smallest line number interval between it and the start statement line.

[0053] Specifically, the target data set also includes statement line numbers corresponding to each line of statements. The statements in the target data set are first arranged in ascending order according to the statement line numbers of each line of statements, and the statement start identification information and statement end identification information corresponding to each line of statements are traversed in sequence according to the ascending arrangement numbers. When the statement start identification information is identified as the start identifier, the identified statement line is determined as the start statement line. Exemplarily, the start identifier can be represented by the number 1, and then the statement line after the start statement line is identified, and it is determined whether the statement end identification information is the end identifier. The statement with the statement end identification information identified after the start statement line as the end identifier is determined as the end statement line. Exemplarily, the end identifier can be represented by the number 1. If the statement line number of the end statement line is the smallest interval with the statement line number of the previously determined start statement line, the statements between the end statement line and the previous start statement line are determined as a statement block. It should be noted that each statement block is independent of each other and there are no crossover statements.

[0054] S140: For each statement block, extract at least one table name in the statement block, and build a table association relationship based on the at least one table name in the statement block.

[0055] Specifically, for each statement block, a pre-built table name query operation statement is executed to obtain at least one table name in each statement block. It should be noted that if only one table name is identified, the user can be reminded to make further confirmation. If there is no problem with the confirmation, it is determined that there is no table with a query relationship for a table name in the statement block. However, the statement block and the corresponding table name can also be stored, so that the table association relationship in each statement block in the target data set can be fully understood; if two or more table names are identified, it is determined that there is a query association relationship between the tables corresponding to the two or more table names, and then the query association relationship between the two or two table names is established. In the case of obtaining the table association relationship corresponding to each statement block, each statement block and the corresponding at least one table name and the corresponding table association relationship can be stored to provide a basis for the subsequent determination of the table sharding strategy selection. It should be noted that, for each statement block, a corresponding regular expression is set according to the positional characteristics of the table name appearing in the query operation statement, and a corresponding table name query operation statement is constructed according to the regular expression to identify the table name in each statement block. It can be understood that the table name usually appears near keywords such as FROM...WHERE, JOIN and UPDATE. Therefore, according to this idea, the content before FROM and after WHERE can be removed first, and then the text content after FROM, JOIN and UPDATE can be identified to extract the corresponding table name. The above-mentioned removal information and extraction of the table name are both realized by the table name query operation statement constructed by the regular expression set according to the query conditions.

[0056] The technical solution of this embodiment is to obtain the data to be processed, construct a target data set based on the data to be processed, wherein the target data set includes multiple lines of statements; determine the statement identification information corresponding to each line of statement in the target data set, wherein the statement identification information includes statement start identification information and statement end identification information; based on the statement start identification information and statement end identification information corresponding to each line of statement, the multiple lines of statements in the target data set are processed in blocks to obtain at least one statement block, wherein the statement block includes at least one line of statement; for each statement block, extract at least one table name in the statement block, and construct a table association relationship based on at least one table name in the statement block. This solution processes the data to be processed to obtain the corresponding target data set, identifies each statement in the target data set, determines the statement start identification information and statement end of each statement, and then processes the statements in blocks according to the statement start identification information and statement end of each statement, and then identifies the table name in the statement block to accurately obtain the table with an association relationship, thereby solving the problem of being unable to accurately determine the data table with an association relationship in the data to be processed, and improving the accuracy of determining the table name in the data to be processed and the accuracy of determining the table association relationship.

[0057] Embodiment 2

[0058] Figure 2 This is a flow chart of a data processing method provided by the second embodiment of the present invention. This embodiment is a further optimization of the method of the above embodiment. Optionally, a preset keyword set is obtained, and sentence start identification information is set for each line of sentence in the target data set based on the preset keyword set; a preset character set is obtained, and sentence end identification information is set for each line of sentence in the target data set based on the preset character set. Figure 2 As shown, the method includes:

[0059] S210: Obtain data to be processed, and construct a target data set based on the data to be processed, wherein the target data set includes multiple lines of statements.

[0060] S220: Obtain a preset keyword set, and set statement start identification information for each line of statement in the target data set based on the preset keyword set.

[0061] In this embodiment, the preset keyword set specifically represents a set formed by keywords located at the beginning of a statement block, which is used to determine the start statement of the statement block. The preset keyword set includes but is not limited to SELECT, UPDATE, DELETE, CURSOR...FOR and FOR...IN.

[0062] Specifically, a preset keyword set is read from a preset storage space, each line of statements is matched according to one or more items in the preset keyword set, and corresponding start identification information is set according to the matching result. Optionally, the statement start identification information is set for each line of statements in the target data set based on the preset keyword set, including: constructing at least one regular expression based on one or more items in the preset keyword set, constructing a statement start position mark operation statement based on at least one regular expression corresponding to the preset keyword set; executing the statement start position mark operation statement, and determining the statement start identification information of each line of statements in the target data set. According to one or more keywords in the preset keyword set, a regular expression for identifying the start of a statement is set, and then one or more regular expressions are used to construct a statement start position mark operation statement according to the query statement construction rules, and the statement start position mark operation statement is executed. According to the execution result, the statement start identification information of each line of the statement is set. If it is recognized that it belongs to the beginning of the statement, the corresponding statement start identification information is set as the start identifier. For example, the start identifier can be set to the number 1, otherwise, a non-start identifier or empty is set. For example, the non-start identifier can be set to the number 0. After traversing each line of the statement in the target data set, the statement start identification information of each line of the statement in the target data set is obtained. It should be noted that if new characters that can identify the start line of a statement are obtained, they can be added to the preset keyword set, and the corresponding regular expression can be constructed. When needed, it can be directly matched and called according to the corresponding keyword.

[0063] S230: Obtain a preset character set, and set statement end identification information for each line of statement in the target data set based on the preset character set.

[0064] The preset character set can be specifically understood as a set used to determine the end of a statement block, including but not limited to the ";" character, LOOP, UNION, SELECT, INSERT, UPDATE, DELETE, MERGE, FROM, WHERE, GROUP BY, ORDER BY, HAVING, and AND. A regular expression corresponding to each character in the preset character combination is set to construct a query statement for identifying the end line of the statement; the statement end identification information corresponding to each line of the statement in the target data set is determined by executing the query statement.

[0065] Specifically, a preset character set is read from a preset storage space, each line of statements is matched according to one or more items in a preset keyword set, and corresponding start identification information is set according to the matching result. Optionally, statement end identification information is set for each line of statements in a target data set based on the preset character set, including: constructing at least one regular expression based on one or more items in the preset character set, constructing a statement end position mark operation statement based on at least one regular expression corresponding to the preset character set, wherein the preset character set includes an end keyword and an end symbol; executing the statement end position mark operation statement, and determining the statement end identification information of each line of statements in the target data set. Specifically, at least one regular expression is constructed according to one or more items in the preset character set, and the constructed regular expression is used to construct a statement end position mark operation statement according to the query statement construction rule, and the statement end position mark operation statement is executed. The statement end identification information of each line of the statement is set according to the execution result. If it is identified as a statement end sentence, the corresponding statement end identification information is set as the end identifier. For example, the end identifier can be set to the number 1, otherwise, a non-end identifier is set or empty. For example, the non-end identifier can be set to the number 0. After traversing each line of the statement in the target data set, the statement end identification information of each line of the statement in the target data set is obtained. It should be noted that if new characters that can identify the end line of the statement are obtained, they can be added to the preset character set, and the corresponding regular expression can be constructed. When needed, the corresponding regular expression can be directly matched according to the corresponding character and the regular expression that matches successfully can be called.

[0066] S240. Based on the statement start identification information and the statement end identification information corresponding to each line of statement, multiple lines of statements in the target data set are processed into blocks to obtain at least one statement block, wherein the statement block includes at least one line of statement.

[0067] S250: For each statement block, extract at least one table name in the statement block, and build a table association relationship based on the at least one table name in the statement block.

[0068] The technical solution of this embodiment is to obtain data to be processed, build a target data set based on the data to be processed, wherein the target data set includes multiple lines of statements; obtain a preset keyword set, and set statement start identification information for each line of statements in the target data set based on the preset keyword set; obtain a preset character set, and set statement end identification information for each line of statements in the target data set based on the preset character set; perform block processing on the multiple lines of statements in the target data set based on the statement start identification information and statement end identification information corresponding to each line of statements, to obtain at least one statement block, wherein the statement block includes at least one line of statements; for each statement block, extract at least one table name in the statement block, and build a table association relationship based on the at least one table name in the statement block. This solution processes the data to be processed to obtain the corresponding target data set, identifies each statement in the target data set, determines the statement start identification information and statement end of each statement, and then performs statement block processing according to the statement start identification information and statement end of each statement, and then identifies the table name in the statement block to accurately obtain the table with an associated relationship, thereby solving the problem of being unable to accurately determine the data table with an associated relationship in the data to be processed, and improving the accuracy of determining the table name in the data to be processed and the accuracy of determining the table association relationship.

[0069] Embodiment 3

[0070] Figure 3 Schematic diagram of the structure of a data processing device provided by Embodiment 3 of the present invention. Figure 3 As shown, the device comprises:

[0071] The target data set determination module 310 is used to obtain the data to be processed and construct a target data set based on the data to be processed, wherein the target data set includes multiple lines of statements;

[0072] The statement identification information determination module 320 is used to determine the statement identification information corresponding to each line of the statement in the target data set, wherein the statement identification information includes statement start identification information and statement end identification information;

[0073] A statement block determination module 330 is used to perform block processing on multiple lines of statements in the target data set based on statement start identification information and statement end identification information corresponding to each line of statement, to obtain at least one statement block, wherein a statement block includes at least one line of statement;

[0074] The table association relationship building module 340 is used to extract at least one table name in each statement block, and build a table association relationship based on the at least one table name in the statement block.

[0075] The technical solution of this embodiment is to obtain the data to be processed through the target data set determination module, and construct the target data set based on the data to be processed, wherein the target data set includes multiple lines of statements; the statement identification information determination module determines the statement identification information corresponding to each line of statement in the target data set, wherein the statement identification information includes statement start identification information and statement end identification information; the statement block determination module performs block processing on the multiple lines of statements in the target data set based on the statement start identification information and statement end identification information corresponding to each line of statement, and obtains at least one statement block, wherein the statement block includes at least one line of statement; the table association relationship construction module extracts at least one table name in the statement block for each statement block, and constructs a table association relationship based on at least one table name in the statement block. This solution processes the data to be processed to obtain the corresponding target data set, identifies each statement in the target data set, determines the statement start identification information and statement end of each statement, and then performs statement block processing according to the statement start identification information and statement end of each statement, and then identifies the table name in the statement block to accurately obtain the table with an associated relationship, thereby solving the problem of being unable to accurately determine the data table with an associated relationship in the data to be processed, and improving the accuracy of determining the table name in the data to be processed and the accuracy of determining the table association relationship.

[0076] On the basis of the above embodiment, optionally, the target data set determination module 310 is specifically used to obtain preset filtering conditions, filter the data to be processed based on the preset filtering conditions to obtain filtered data, and construct an initial data set based on the filtered data, wherein the preset filtering conditions include a regular expression for keyword matching in at least one function parameter and a regular expression for preset character matching; remove the set type statements in the initial data set to obtain the target data set, wherein the set type statements include set characters.

[0077] The target data set determination module 310 is further specifically used to construct a corresponding data filtering operation statement based on one or more regular expressions in the preset filtering conditions, and process the data to be processed based on the data filtering operation statement to obtain filtered data.

[0078] Optionally, the statement identification information determination module 320 is specifically used to obtain a preset keyword set, and set statement start identification information for each line of statements in the target data set based on the preset keyword set; obtain a preset character set, and set statement end identification information for each line of statements in the target data set based on the preset character set.

[0079] Optionally, the statement identification information determination module 320 is also specifically used to construct at least one regular expression based on one or more items in a preset keyword set, and to construct a statement start position mark operation statement based on at least one regular expression corresponding to the preset keyword set; execute the statement start position mark operation statement, and determine the statement start identification information of each line of statements in the target data set.

[0080] Optionally, the statement identification information determination module 320 is further specifically used to construct at least one regular expression based on one or more items in a preset character set, and to construct a statement end position mark operation statement based on at least one regular expression corresponding to the preset character set, wherein the preset character set includes an end keyword and an end symbol; execute the statement end position mark operation statement to determine the statement end identification information of each line of statements in the target data set.

[0081] Optionally, the target data set also includes statement line numbers corresponding to each line of statements; a statement block determination module 330 is specifically used to traverse the statement start identification information and statement end identification information corresponding to each line of statements in ascending order according to the statement line numbers of each line of statements; determine the statement line with the statement start identification information as the start identifier as the start statement line; determine the end statement line that is located after the start statement line and has the statement end identification information as the end identifier, and regard the statements between the start statement line and the end statement line as a statement block, wherein the end statement line is the statement line with the smallest line number interval between it and the start statement line.

[0082] The data processing device provided in the embodiment of the present invention can execute the data processing method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0083] Embodiment 4

[0084] Figure 4 1 is a schematic diagram of the structure of an electronic device provided in Embodiment 4 of the present invention. The electronic device 10 is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0085] like Figure 4As shown, the electronic device 10 includes at least one processor 11, and a memory connected to the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 to the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0086] A number of components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0087] The processor 11 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 performs the various methods and processes described above, such as a data processing method.

[0088] In some embodiments, the data processing method may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the data processing method described above may be performed. Alternatively, in other embodiments, the processor 11 may be configured to perform the data processing method in any other appropriate manner (e.g., by means of firmware).

[0089] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0090] The computer program for implementing the data processing method of the present invention can be written in any combination of one or more programming languages. These computer programs can be provided to a processor of a general-purpose computer, a special-purpose computer or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program can be executed entirely on the machine, partially on the machine, partially on the machine as a stand-alone software package and partially on a remote machine, or entirely on a remote machine or server.

[0091] Embodiment 5

[0092] Embodiment 5 of the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to cause a processor to execute a data processing method, the method comprising:

[0093] Acquire the data to be processed, and construct a target data set based on the data to be processed, wherein the target data set includes multiple lines of statements;

[0094] Determine the statement identification information corresponding to each line of the statement in the target data set, wherein the statement identification information includes statement start identification information and statement end identification information;

[0095] Based on the statement start identification information and the statement end identification information respectively corresponding to each line of the statement, multiple lines of statements in the target data set are processed into blocks to obtain at least one statement block, wherein the statement block includes at least one line of statement;

[0096] For each statement block, at least one table name in the statement block is extracted, and a table association relationship is established based on the at least one table name in the statement block.

[0097] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in combination with an instruction execution system, device or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0098] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0099] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0100] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0101] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0102] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A data processing method, characterized in that: include: Acquire the data to be processed, and construct a target data set based on the data to be processed, wherein the target data set includes multiple lines of statements; Determine the statement identification information corresponding to each line of the statement in the target data set, wherein the statement identification information includes statement start identification information and statement end identification information; Based on the statement start identification information and the statement end identification information respectively corresponding to the statement lines, the multiple statement lines in the target data set are processed into blocks to obtain at least one statement block, wherein the statement block includes at least one statement line; For each of the statement blocks, at least one table name in the statement block is extracted, and a table association relationship is established based on the at least one table name in the statement block.

2. The method according to claim 1, characterized in that The step of constructing a target data set based on the data to be processed includes: Obtaining a preset filtering condition, filtering the data to be processed based on the preset filtering condition to obtain filtered data, and constructing an initial data set based on the filtered data, wherein the preset filtering condition includes a regular expression for keyword matching in at least one function parameter and a regular expression for preset character matching; The target data set is obtained by removing the setting type statement in the initial data set, wherein the setting type statement includes setting characters.

3. The method according to claim 2, characterized in that The filtering the data to be processed based on the preset filtering condition to obtain filtered data includes: A corresponding data filtering operation statement is constructed based on one or more regular expressions in the preset filtering conditions, and the data to be processed is processed based on the data filtering operation statement to obtain the filtered data.

4. The method according to claim 1, characterized in that The determining of statement identification information corresponding to each line of statement in the target data set includes: Acquire a preset keyword set, and set statement start identification information for each line of statement in the target data set based on the preset keyword set; A preset character set is obtained, and statement end identification information is set for each line of statement in the target data set based on the preset character set.

5. The method according to claim 4, characterized in that The step of setting statement start identification information for each line of statement in the target data set based on the preset keyword set includes: Constructing at least one regular expression based on one or more items in the preset keyword set, and constructing a statement start position marking operation statement based on at least one regular expression corresponding to the preset keyword set; The statement start position marking operation statement is executed to determine statement start identification information of each line of statement in the target data set.

6. The method according to claim 4, characterized in that The step of setting statement end identification information for each line of statement in the target data set based on the preset character set includes: Constructing at least one regular expression based on one or more items in the preset character set, and constructing a statement end position mark operation statement based on at least one regular expression corresponding to the preset character set, wherein the preset character set includes an end keyword and an end symbol; The statement end position marking operation statement is executed to determine statement end identification information of each line of statement in the target data set.

7. The method according to claim 1, characterized in that The target data set also includes statement line numbers corresponding to each line of statements; The block processing of the multiple lines of statements in the target data set based on the statement start identification information and the statement end identification information respectively corresponding to the lines of statements to obtain at least one statement block includes: Traversing the statement start identification information and the statement end identification information respectively corresponding to each line of statements in ascending order of the statement line numbers of each line of statements; Determine the statement line whose statement start identification information is the start identifier as the start statement line; Determine an ending statement line that is located after the start statement line and whose statement end identification information is an end identifier, and treat the statements between the start statement line and the ending statement line as a statement block, wherein the ending statement line is the statement line with the smallest line number interval between it and the start statement line.

8. A data processing device, characterized in that: include: A target data set determination module, used to obtain the data to be processed and construct a target data set based on the data to be processed, wherein the target data set includes multiple lines of statements; A statement identification information determination module, used to determine the statement identification information corresponding to each line of statement in the target data set, wherein the statement identification information includes statement start identification information and statement end identification information; A statement block determination module, configured to perform block processing on multiple lines of statements in the target data set based on the statement start identification information and the statement end identification information respectively corresponding to each line of statement, to obtain at least one statement block, wherein the statement block includes at least one line of statement; The table association relationship building module is used to extract at least one table name in each statement block, and build a table association relationship based on the at least one table name in the statement block.

9. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can perform the data processing method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the data processing method according to any one of claims 1 to 7 when executed.

Images