Data processing method and device, equipment, storage medium and program product

Abstract

The invention provides a data processing method and device, equipment, a storage medium and a program product, relates to the technical field of data processing, can automatically generate a corresponding design document according to a program code, and improves the working efficiency while ensuring the consistency of the program code and the design document. According to the specific technical scheme, a program code file is obtained; identifying and extracting at least one data processing program code from the program code file; analyzing each data processing program code to obtain a code segment list containing one or more code segments; sQL statement code segments in the code segment list are preprocessed, and formatted SQL statements are obtained; analyzing the formatted SQL statement according to target table information, target table field information, source table information, source table field information, filtering conditions, information of grouping clauses and information of sorting clauses; and filling the information according to a predefined data mapping document template format to generate a data mapping design document.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a data processing method, apparatus, device, storage medium, and program product. Background Technology

[0002] In software design and development, the usual process is for designers and developers to receive business requirements, first conduct requirements analysis, then write design documents, and then carry out coding development, testing, and production deployment.

[0003] However, in actual development work, developers don't always write design documents before starting to develop program code. They may only have a simple design before moving on to coding, resulting in the lack of design documents. Or, design documents may have been written in the early stages of development, but the program code was adjusted later as requirements changed, and the design documents were not updated accordingly, leading to a certain degree of discrepancy between the design documents and the program code.

[0004] Currently, the design documents are manually supplemented and improved one by one. This not only requires a lot of manpower and time, but is also inefficient. In addition, manual operation is prone to introducing new errors, which can lead to inconsistencies between the design documents and the program code. Summary of the Invention

[0005] This application provides a data processing method, apparatus, device, storage medium, and program product that can automatically generate corresponding design documents based on program code, ensuring consistency between program code and design documents while improving work efficiency.

[0006] To achieve the above objectives, the embodiments of this application adopt the following technical solutions: Firstly, a data processing method is provided, comprising: acquiring a program code file and storing the program code file as a string; identifying and extracting at least one data processing program code from the string of the program code file; parsing each data processing program code to obtain a code segment list containing one or more code segments, wherein the code segments in the code segment list are classified into code segments that do not require processing, logical code segments, and structured query language (SQL) statement code segments; preprocessing the classified SQL statement code segments to obtain formatted SQL statements; parsing the formatted SQL statements to obtain data processing information, which includes: target table information, target table field information, source table information, source table field information, filtering conditions, grouping clauses, and sorting clauses; and filling the formatted SQL statements, target table information, target table field information, source table field information, source table information, filtering conditions, grouping clauses, and sorting clauses according to a predefined data mapping document template format to generate a data mapping design document.

[0007] This application automatically identifies program code files, analyzes the SQL statements in the code, and fills in the formatted SQL statements, target table information, target table field information, source table field information, source table information, filtering conditions, grouping clauses, and sorting clauses according to a predefined data mapping document template to generate a data mapping design document. This automated process avoids omissions and misinterpretations that may occur during manual interpretation, and ensures a high degree of consistency between the generated design document and the program code. In addition, this automated process can shorten working time, thereby improving work efficiency.

[0008] In addition, this method of filling the parsed information according to a predefined data mapping document template format makes the design document output have a standardized format, which is easy for different developers to understand and also provides a foundation for subsequent development using this unified design document format.

[0009] In one possible implementation of the first aspect, parsing the formatted SQL statement to obtain data processing information includes: splitting the formatted SQL statement into a list of elements and determining the processing type of the formatted SQL statement based on the list of elements; parsing the target table schema name, target table name, target table name, and target table alias based on the processing type of the list of elements; and organizing the processing type, target table schema name, target table name, target table name, and target table alias into target table information.

[0010] Because SQL statements can be written in various ways, this application first breaks down the SQL statement into a list of elements and determines its processing type (such as INSERT, UPDATE, MERGE, etc.), and then parses the target table information based on the syntax rules corresponding to different types. This structured parsing method can adapt to SQL statements with different writing styles and complexities, accurately identifying each component such as schema name, table name, and aliases, thereby avoiding parsing failures caused by changes in SQL writing style, which improves the reliability of the solution. In addition, converting the SQL statement into target table information with a clear structure and well-defined fields (including schema name, table name, aliases, etc.) provides a data foundation for the subsequent steps of filling the design document according to the template.

[0011] In another possible implementation of the first aspect, parsing the formatted SQL statement to obtain data processing information further includes: parsing a target table field list and a source table field list of the formatted SQL statement based on the processing type and the element list. The target table field list includes one or more target table fields, and the source table field list includes one or more source table fields. The target table fields and the source table fields are used to determine the mapping relationship between the fields. Based on the metadata information in the database, each target table field in the target table field list is queried to obtain the field type, field name, and primary key information of the target table field. Based on the target table fields, the field type, field name, and primary key information of the target table fields, the target table field information is compiled.

[0012] After parsing the basic target table field list and source table field list (the target table fields in the target table field list and the source table fields in the source table field list have a mapping relationship), this application automatically queries the database's metadata information repository to complete the remaining information for each field (the field type, field name, and primary key information of the target table field). This method ensures that the data in the final generated target table field information not only includes the mapping relationship from which source table it comes, but also contains some key information, which provides a data foundation for the subsequent step of filling in the design document according to the template.

[0013] In another possible implementation of the first aspect, the method further includes: parsing source table field information, a list of field aliases, and a list of source table information for a formatted SQL statement based on the processing type, the element list, and the source table field list, wherein the source table information list includes one or more code segments; parsing source table information, filtering conditions, grouping clauses, or sorting clauses based on the source table information list, wherein the source table information includes the source table schema name, source table name, source table name, source table alias, inter-table join method, and inter-table association conditions; if the target table field information obtained based on the target table fields, the field types of the target table fields, the field names, and the primary key information is empty, then reconstructing the target table field information based on the list of field aliases.

[0014] This application obtains information such as source table information, filtering conditions, grouping clauses, sorting clauses, inter-table join methods, and inter-table association conditions through further in-depth analysis of SQL statements. This enables business personnel to quickly understand the relevant logic of the data without having to delve into the details of the code, thereby improving the readability of the design document.

[0015] In addition, this application sets a key condition judgment: if the target table field information cannot be obtained by querying the database metadata, it will automatically fall back to the list of field aliases parsed from the SQL statement itself to reconstruct the target table field information. This mechanism can ensure the smooth progress of the whole process.

[0016] In another possible implementation of the first aspect, the above-mentioned splitting of the formatted SQL statement into an element list and determining the processing type of the formatted SQL statement based on the element list specifically includes: splitting the formatted SQL statement into an element list by spaces and newlines, and determining the processing type of the formatted SQL statement based on the first element in the element list; the above-mentioned parsing to obtain the target table schema name, target table name, target table name, and target table alias based on the processing type and the element list specifically includes: if the processing type is the first type, then the third element in the corresponding element list is recorded as the first identifier, and the fourth element in the element list is recorded as the second identifier; if the processing type is the second type, then the second element in the corresponding element list is recorded as the first identifier, and the third element in the element list is recorded as the second identifier; based on the first identifier, the target table schema name, target table name, and target table name are determined; based on the second identifier, the target table alias is determined.

[0017] This application utilizes the standard syntax structure of the SQL language. First, it determines the processing type of the SQL based on the first element, and then assigns different parsing rules to different processing types. This parsing method based on fixed syntax rules is highly efficient, logically clear, and very accurate.

[0018] In another possible implementation of the first aspect, the above-mentioned parsing based on the processing type and element list yields a target table field list and a source table field list for a formatted SQL statement. The target table field list includes one or more target table fields, and the source table field list includes one or more source table fields. Specifically, this includes: if the processing type is type three, then marking the fourth element in the corresponding element list as a third identifier and marking the fifth element in the element list as a fourth identifier; type one includes type three; based on the third identifier, the fourth identifier, and the element list, determining the target table fields; if the processing type is type four, then based on the corresponding element list, determining the target table fields and source table fields, where the target table fields include a target table primary key field, and the source table fields include a source table primary key field. The target table primary key field is used to determine the primary key status of the target table fields, and the source table primary key field is used to determine the primary key status of the source table fields; type one includes type four; if the processing type is type two, then based on the corresponding element list, determining the target table fields and source table fields; and based on the target table fields and the source table fields, organizing them to obtain the target table field list and the source table field list.

[0019] Because different types of SQL have very different syntax structures, this application does not attempt to parse all types of SQL statements with a single set of rules. Instead, it defines independent field extraction logic for different types of statements. This avoids interference between syntax rules and ensures that the field lists of each statement can be accurately identified even when faced with statements with different syntaxes, thereby guaranteeing the stability and reliability of the entire automated document generation process.

[0020] In another possible implementation of the first aspect, the above-mentioned parsing of the source table field information, field alias list, and source table information list of the formatted SQL statement based on the processing type, element list, and source table field list specifically includes: if the processing type is the third type, then the corresponding formatted SQL statement is split into a first part and a second part, the first part being used to characterize the data structure of the formatted SQL statement, and the second part being used to characterize the source and combination method of the data in the formatted SQL statement; based on the first part, the source table field information and field alias list in the formatted SQL statement are determined; based on the second part, the source table information list in the formatted SQL statement is determined; wherein, if the first element of the second part is a first key, then the second part is a filtering condition; if the first element of the second part is a second key, then the second part is a grouping clause; if the first element of the second part is a third key, then the second part is a sorting clause.

[0021] This method first breaks down the SQL statement into a first part (such as the list of fields after SELECT) and a second part (such as the content after FROM, WHERE, and GROUP BY). Then, it precisely extracts information such as fields and aliases from the first part, and systematically parses complex logic such as table joins, filtering conditions, grouping, and sorting from the second part. This method makes the parsing process hierarchical and clear, enhancing the ability to parse SQL statements.

[0022] In another possible implementation of the first aspect, the method further includes: in response to an update operation on the data mapping design document, generating new program code conforming to the target programming language based on the updated data mapping design document and the target template of the code generation platform.

[0023] This application utilizes a method that automatically responds to updates in a structured data mapping design document, generating new, standard-compliant code using a target template from a code generation platform. This allows for easy switching between different programming languages ​​when generating new code based on the design document, enabling designers and developers to focus more time and effort on implementing business functions and improving work efficiency. Furthermore, code modifications can be made directly within the data mapping design document, which is more convenient and efficient than simply modifying the code itself.

[0024] Secondly, a data processing apparatus is provided. The method includes: a data acquisition module for acquiring a program code file and storing the program code file as a string; a data processing module for identifying and extracting at least one data processing program code from the string of the program code file; the data processing module is further used to parse each data processing program code to obtain a code segment list containing one or more code segments, wherein the code segments in the code segment list are classified into code segments that do not require processing, logical code segments, and structured query language (SQL) statement code segments; the data processing module is further used to preprocess the classified SQL statement code segments to obtain formatted SQL statements; a data parsing module for parsing the formatted SQL statements to obtain data processing information, which includes: target table information, target table field information, source table information, source table field information, filtering conditions, grouping clauses, and sorting clauses; and a design document generation module for filling the formatted SQL statements, target table information, target table field information, source table field information, source table information, filtering conditions, grouping clauses, and sorting clauses according to a predefined data mapping document template format to generate a data mapping design document.

[0025] Thirdly, an electronic device is provided, the method comprising: a memory and at least one processor. The memory is communicatively connected to the processor. The memory is used to store computer program code, the computer program code including computer instructions. When the processor executes the computer instructions, it causes the electronic device to perform the method as described in the first aspect and any possible implementation thereof.

[0026] Fourthly, embodiments of this application provide a computer-readable storage medium storing computer instructions. When executed by a processor, these computer instructions are used to implement the method described in the first aspect and any possible implementation thereof.

[0027] Fifthly, embodiments of this application provide a computer program product that, when run on a computer or executed by a computer's processor, implements the method described in the first aspect and any possible design thereof. The computer may be the data processing apparatus described in the second aspect and any possible implementation thereof.

[0028] It is understood that the beneficial effects achieved by the data processing apparatus described in the second aspect, the electronic device described in the third aspect, the computer-readable storage medium described in the fourth aspect, and the computer program product described in the fifth aspect can be referred to as the beneficial effects in the first aspect and any possible implementation thereof, which will not be repeated here. Attached Figure Description

[0029] Figure 1 A flowchart illustrating a data processing method provided in an embodiment of this application; Figure 2 This application provides a schematic diagram of an SQL statement parsing and processing process according to an embodiment of the present application. Figure 3 This is a schematic diagram of the process for obtaining target table information provided in an embodiment of this application; Figure 4 This is a schematic diagram of the process for obtaining target table field information provided in an embodiment of this application; Figure 5 This is a schematic diagram illustrating the process of obtaining source table field information provided in an embodiment of this application; Figure 6 A flowchart illustrating the parsing of the SELECT field portion provided in an embodiment of this application; Figure 7 A flowchart illustrating the parsing of the FROM field provided in an embodiment of this application; Figure 8 This is a schematic diagram of the process for obtaining source table information provided in an embodiment of this application; Figure 9 This is a schematic diagram of the structure of a data processing device provided in an embodiment of this application; Figure 10 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0030] Hereinafter, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this embodiment, unless otherwise stated, "a plurality of" means two or more.

[0031] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0032] The technical solutions provided in this application, including the collection, storage, use, processing, transmission, provision, and disclosure of financial data or user data, comply with relevant laws and regulations and do not violate public order and good morals.

[0033] It should be noted that in the embodiments of this application, certain software, components, models and other existing solutions in the industry may be mentioned. These should be regarded as exemplary and are only intended to illustrate the feasibility of implementing the technical solution of this application. However, it does not mean that the applicant has used or necessarily used the solution.

[0034] In software design and development, the usual process is for designers and developers to receive business requirements, first conduct requirements analysis, then write design documents, and then carry out coding development, testing, and production deployment.

[0035] However, in actual development work, developers do not necessarily write data mapping design documents before developing program code. They may only do a simple design and then start coding, resulting in the lack of design documents. Or, design documents may have been written in the early stages of development, but the program code was adjusted as requirements changed, and the design documents were not updated accordingly, resulting in a certain degree of discrepancy between the design documents and the program code.

[0036] Currently, the design documents are manually supplemented and improved one by one. This not only requires a lot of manpower and time, but is also inefficient. In addition, manual operation is prone to introducing new errors, which can lead to inconsistencies between the design documents and the program code.

[0037] Therefore, this application provides a data processing method, which includes: obtaining a program code file and storing the program code file as a string; identifying and extracting at least one data processing program code from the string of the program code file; parsing each data processing program code to obtain a code segment list containing one or more code segments, wherein the code segments in the code segment list are classified into code segments that do not require processing, logical code segments, and Structured Query Language (SQL) statement code segments; preprocessing the classified SQL statement code segments to obtain formatted SQL statements; parsing the formatted SQL statements to obtain data processing information, which includes: target table information, target table field information, source table information, source table field information, filtering conditions, grouping clauses, and sorting clauses; and filling the formatted SQL statements, target table information, target table field information, source table field information, source table information, filtering conditions, grouping clauses, and sorting clauses according to a predefined data mapping document template format to generate a data mapping design document.

[0038] This application automatically identifies program code files, analyzes the SQL statements in the code, and fills in the formatted SQL statements, target table information, target table field information, source table field information, source table information, filtering conditions, grouping clauses, and sorting clauses according to a predefined data mapping document template to generate a data mapping design document. This automated process avoids omissions and misinterpretations that may occur during manual interpretation, and ensures a high degree of consistency between the generated design document and the program code. In addition, this automated process can shorten working time, thereby improving work efficiency.

[0039] In addition, this method of filling the parsed information according to a predefined data mapping document template format makes the design document output have a standardized format, which is easy for different developers to understand and also provides a foundation for subsequent development using this unified design document format.

[0040] The following describes a data processing method provided by an embodiment of this application in detail with reference to the accompanying drawings.

[0041] Figure 1 This is a flowchart illustrating a data processing method provided in an embodiment of this application. The data processing method provided in this application can be applied to a computing device. Specifically, the computing device can be a single server or a server cluster composed of multiple servers, a computer, or a processor or processing chip within a server or computer, etc. This application does not limit the specific form of the computing device.

[0042] S101. Obtain the program code file and store it as a string.

[0043] First, store the entire program code file as a string. This program code file can be a source code file, configuration file, script file, or data file, etc. Furthermore, storing the program file as a string facilitates subsequent parsing and processing, such as identifying and extracting at least one data processing program code or formatting SQL statements from the string.

[0044] S102. Identify and extract at least one data processing program code from the strings in the program code file.

[0045] Next, the program code is read line by line, the program code type is determined, and the program code is stored as a dictionary, with key-value pairs mapping the program code name and program code content. If a program code file contains multiple data processing programs, it is stored as multiple key-value pairs.

[0046] In some examples, each data processing program is given an identifier as a key, and then all the code content corresponding to that data processing program is used as the value to form a dictionary containing multiple key-value pairs, such as: {"Program 1":"Code content 1...","Program 2":"Code content 2..."}.

[0047] S103. Parse and process each data processing program code to obtain a code segment list containing one or more code segments. The code segments in the code segment list are classified into code segments that do not require processing, logical code segments, and structured query language (SQL) statement code segments.

[0048] Next, the dictionary key-value pairs of program code obtained in the previous step are processed in a loop. The values ​​of the key-value pairs—the program code content—are parsed, and lexical analysis, preprocessing, and syntax analysis are performed on the program code. For stored procedures and Perl program code, the code is divided into multiple code segments according to semicolons, and for Python programs, it is divided according to indentation format, and stored in the processing list PL.

[0049] In some examples, each data processing program code is broken down into a "process list (PL)". For instance, an SQL program might be broken down into: 'Step 1: CREATE TABLE...', 'Step 2: INSERT INTO...', 'Step 3: UPDATE...'. This breaks down complex programs into atomic-level operation steps, facilitating the step-by-step analysis of the data processing program.

[0050] For SQL, stored procedures, and PERL: Since semicolons typically represent the end of a statement in these languages, code is broken down into blocks using semicolons. For Python: It doesn't rely on semicolons but instead looks at indentation, because Python uses indentation to indicate code blocks (such as the inner code of loops and conditional statements).

[0051] S104. Preprocess the classified SQL statement code segments to obtain formatted SQL statements.

[0052] Next, the PL (Program File System) is processed in a loop: it automatically skips code snippets that do not need to be processed, and for code snippets that need to be processed, it parses the code content: if it is a logical judgment, loop definition, variable definition, or data cleanup code, it is directly saved as a program code snippet; if the code snippet is a data processing SQL statement, it is formatted, comments are removed, keywords are converted to uppercase, and the SQL statement is beautified by adding newlines and spaces.

[0053] In some examples, each "step" in the resulting process list PL is examined one by one, and the function of each code segment is determined. Code segments that do not require processing are skipped, such as comments or other useless fragments. If it's logic code, such as if statements, for loops, variable definitions, or data cleanup (e.g., calls to specific functions), mark it as a "code snippet" and set it aside temporarily. If it's SQL statements, such as statements starting with SELECT, INSERT, UPDATE, etc., format them, remove comments, convert keywords to uppercase (e.g., unify select, from, where to SELECT, FROM, WHERE; this is for standardization and easier subsequent identification), and make the SQL statement structure clear by adding line breaks and spaces.

[0054] S105. Parse the formatted SQL statement to obtain data processing information, which includes: target table information, target table field information, source table information, source table field information, filtering conditions, grouping clauses, and sorting clauses.

[0055] In some embodiments, parsing the formatted SQL statement to obtain data processing information includes: splitting the formatted SQL statement into an element list and determining the processing type of the formatted SQL statement based on the element list; parsing the element list based on the processing type to obtain the target table schema name, target table name, target table name, and target table alias; and organizing the processing type, target table schema name, target table name, target table name, and target table alias into target table information.

[0056] Because SQL statements can be written in various ways, this application first breaks down the SQL statement into a list of elements and determines its processing type (such as INSERT, UPDATE, MERGE, etc.), and then parses the target table information based on the syntax rules corresponding to different types. This structured parsing method can adapt to SQL statements with different writing styles and complexities, accurately identifying each component such as schema name, table name, and aliases, thereby avoiding parsing failures caused by changes in SQL writing style, which improves the reliability of the solution. In addition, converting the SQL statement into target table information with a clear structure and well-defined fields (including schema name, table name, aliases, etc.) provides a data foundation for the subsequent steps of filling the design document according to the template.

[0057] In some embodiments, parsing the formatted SQL statement to obtain data processing information further includes: parsing a target table field list and a source table field list of the formatted SQL statement based on the processing type and the element list, wherein the target table field list includes one or more target table fields, the source table field list includes one or more source table fields, and the target table fields and the source table fields are used to determine the mapping relationship between the fields; querying each target table field in the target table field list based on metadata information in the database to obtain the field type, field name, and primary key information of the target table field; and organizing the target table field information based on the target table fields, the field type, field name, and primary key information of the target table fields.

[0058] After parsing the basic target table field list and source table field list (the target table fields in the target table field list and the source table fields in the source table field list have a mapping relationship), this application automatically queries the database's metadata information repository to complete the remaining information for each field (the field type, field name, and primary key information of the target table field). This method ensures that the data in the final generated target table field information not only includes the mapping relationship from which source table it comes, but also contains some key information, which provides a data foundation for the subsequent step of filling in the design document according to the template.

[0059] In some embodiments, the method further includes: parsing source table field information, field alias list, and source table information list of a formatted SQL statement based on the processing type, element list, and source table field list, wherein the source table information list includes one or more code segments; parsing source table information, filtering conditions, grouping clauses, or sorting clauses based on the source table information list, wherein the source table information includes source table schema name, source table name, source table name, source table alias, inter-table join method, and inter-table association conditions; if the target table field information obtained based on the target table fields, target table field field types, field names, and primary key information is empty, then reconstructing the target table field information based on the field alias list.

[0060] If the target table field information is empty, the obtained field alias list is used to poll the field alias list to obtain the field name. Based on the field name, the field type and Chinese name of the field are obtained from the metadata and incremented. The target table field information is then constructed in key-value pair format.

[0061] This application obtains information such as source table information, filtering conditions, grouping clauses, sorting clauses, inter-table join methods, and inter-table association conditions through further in-depth analysis of SQL statements. This enables business personnel to quickly understand the relevant logic of the data without having to delve into the details of the code, thereby improving the readability of the design document.

[0062] In addition, this application sets a key condition judgment: if the target table field information cannot be obtained by querying the database metadata, it will automatically fall back to the list of field aliases parsed from the SQL statement itself to reconstruct the target table field information. This mechanism can ensure the smooth progress of the whole process.

[0063] In some embodiments, the above-mentioned splitting of the formatted SQL statement into an element list and determining the processing type of the formatted SQL statement based on the element list specifically includes: splitting the formatted SQL statement into an element list by spaces and newlines, and determining the processing type of the formatted SQL statement based on the first element in the element list; the above-mentioned parsing to obtain the target table schema name, target table name, target table name, and target table alias based on the processing type and the element list specifically includes: if the processing type is the first type, then the third element in the corresponding element list is recorded as the first identifier, and the fourth element in the element list is recorded as the second identifier; if the processing type is the second type, then the second element in the corresponding element list is recorded as the first identifier, and the third element in the element list is recorded as the second identifier; based on the first identifier, the target table schema name, target table name, and target table name are determined; based on the second identifier, the target table alias is determined.

[0064] This application utilizes the standard syntax structure of the SQL language. First, it determines the processing type of the SQL based on the first element, and then assigns different parsing rules to different processing types. This parsing method based on fixed syntax rules is highly efficient, logically clear, and very accurate.

[0065] In some embodiments, the above-mentioned parsing of the target table field list and source table field list of the formatted SQL statement based on the processing type and the element list, wherein the target table field list includes one or more target table fields and the source table field list includes one or more source table fields, specifically including: if the processing type is the third type, then the fourth element in the corresponding element list is marked as the third identifier, and the fifth element in the element list is marked as the fourth identifier, the first type includes the third type; based on the third identifier, the fourth identifier, and the element list, the target table fields are determined; if the processing type is the fourth type, then based on the corresponding element list, the target table fields and source table fields are determined, wherein the target table fields include the target table primary key field and the source table fields include the source table primary key field, the target table primary key field is used to determine the primary key status of the target table fields, and the source table primary key field is used to determine the primary key status of the source table fields, the first type includes the fourth type; if the processing type is the second type, then based on the corresponding element list, the target table fields and source table fields are determined; based on the target table fields and the source table fields, the target table field list and the source table field list are compiled.

[0066] Because different types of SQL have very different syntax structures, this application does not attempt to parse all types of SQL statements with a single set of rules. Instead, it defines independent field extraction logic for different types of statements. This avoids interference between syntax rules and ensures that the field lists of each statement can be accurately identified even when faced with statements with different syntaxes, thereby guaranteeing the stability and reliability of the entire automated document generation process.

[0067] In some embodiments, the above-mentioned parsing of the source table field information, field alias list, and source table information list of the formatted SQL statement based on the processing type, element list, and source table field list specifically includes: if the processing type is the third type, then splitting the corresponding formatted SQL statement into a first part and a second part, the first part being used to characterize the data structure of the formatted SQL statement, and the second part being used to characterize the source and combination method of the data in the formatted SQL statement; based on the first part, determining the source table field information and field alias list in the formatted SQL statement; based on the second part, determining the source table information list in the formatted SQL statement; wherein, if the first element of the second part is a first key, then the second part is a filtering condition; if the first element of the second part is a second key, then the second part is a grouping clause; if the first element of the second part is a third key, then the second part is a sorting clause.

[0068] This method first breaks down the SQL statement into a first part (such as the list of fields after SELECT) and a second part (such as the content after FROM, WHERE, and GROUP BY). Then, it precisely extracts information such as fields and aliases from the first part, and systematically parses complex logic such as table joins, filtering conditions, grouping, and sorting from the second part. This method makes the parsing process hierarchical and clear, enhancing the ability to parse SQL statements.

[0069] The following example illustrates how step S105 parses the formatted SQL statement to obtain data processing information. Example: Figure 2 This is a schematic diagram illustrating a SQL statement parsing and processing procedure provided in an embodiment of this application. Figure 2 Steps S201 to S207 are shown below: S201. Split the formatted SQL statement into a list of elements by spaces or newlines.

[0070] First, split the formatted SQL statement into an element list L by spaces and newlines.

[0071] S202. Obtain the processing type based on the first element in the element list.

[0072] Next, based on the first element of the element list L, determine which of the following four data processing types is being processed: INSERT (add data), UPDATE (update data), MERGE (merge data, update if existing, insert if not existing), or REPLACE (replace data).

[0073] S203, Obtain target table information TTL.

[0074] The TTL process for obtaining the target table information can be found in [link / reference]. Figure 3 , Figure 3 This is a schematic diagram illustrating the process of obtaining target table information provided in an embodiment of this application. For example... Figure 3 As shown: Based on the processing type obtained in step S202, if it is an INSERT, MERGE, or REPLAC case, the third element is retrieved and denoted as W1, and the fourth element is denoted as W2. If it is an UPDATE case, the second element is retrieved and denoted as W1, and the third element is denoted as W2. Here, INSERT, MERGE, and REPLAC cases can be referred to as the first type, UPDATE cases as the second type, W1 as the first identifier, and W2 as the second identifier.

[0075] Here, W1 is the target table name. It's checked whether it contains a period (.). If it does, it means the table name contains a schema name, so it's split into the target table schema name and the target table name. Otherwise, the target table schema name is empty. Then, based on metadata information, the Chinese name information of the target table is obtained. Next, W2 is analyzed. If it's an SQL keyword or contains a left parenthesis, it means no target table alias is set, so the target table alias is set to empty. Otherwise, W2 is the target table alias.

[0076] Based on the above information, the target table information TTL is obtained, including: processing type, target table schema name, target table name, target table Chinese name (also known as target table name), and target table alias.

[0077] S204. Obtain target table field information (TCD).

[0078] The process of obtaining target table field information via TCD can be found in [reference needed]. Figure 4 , Figure 4 This is a schematic diagram illustrating the process of obtaining target table field information provided in an embodiment of this application. For example... Figure 4 As shown: Based on the processing type determination obtained in step S202, for INSERT and REPLACE types (which can also be called the third type), the 4th element in the element list L is denoted as W4, and the 5th element is denoted as W5. It is determined whether W4 and W5 contain a left parenthesis. If either one contains a left comma, it means the SQL statement contains target table field information. Starting from the left parenthesis, elements are concatenated with spaces to form the string S1, continuing until a right parenthesis is encountered. String S1 is split by English commas and stored in the target table field list TCL. Here, W4 can also be called the third identifier, and W5 can also be called the fourth identifier.

[0079] For the MERGE type (which can also be called the fourth type), there are three cases: Scenario 1: First, retrieve the primary key field. Poll the element list L, retrieve elements with zero parentheses and values ​​between 'ON' or 'ON()' and WHEN, and concatenate them into the string PKS. For example, poll the element list L, calculate the number of parentheses, increment the count by 1 if an element contains one English left parenthesis, and decrement the count by 1 if an element contains one English right parenthesis. If the count of parentheses is zero and the element value is 'ON' or 'ON()', start concatenating the elements with spaces to form the string PKS, continuing until keywords such as WHEN MATCHED are encountered. If PKS exists, extract the PKS equals sign expression. Fields containing "target table alias." are the primary key fields of the target table and stored in the target table primary key field list PK_TCL. Fields on the other side of the equals sign are the source table fields and stored in the source table primary key field list PK_SCL. If the fields do not contain a period, the left side of the equals sign is assumed to be the target table field and stored in the target table primary key field list PK_TCL, and the right side is assumed to be the source table field and stored in the source table primary key field list PK_SCL.

[0080] Scenario 2: Retrieving Updated Fields. Poll the element list L, taking each element after the UPDATE SET keyword and concatenating them with spaces to form the string UDS. Continue until a WHERE or WHEN NOT keyword is encountered, and the number of parentheses is zero. If a UDS exists, extract the equals sign expression from it. The field on the left side of the equals sign is the target table field and is stored in the target table's update field list UD_TCL. The field on the right side of the equals sign is the source table field and is stored in the source table's update field list UD_SCL.

[0081] Scenario 3: Retrieving the Insert Field. Poll the element list L, concatenating the elements within parentheses after the INSERT keyword into the string TIS, until the VALUS keyword ends. Concatenate the elements within parentheses after VALUES into the string SIS. Poll the TIS string character by character. If the number of parentheses is 0 and it is an English comma, it indicates a field; concatenate these characters and store them in the target table's insert field list IN_TCL. Process SIS in the same way and store it in the source table's insert field list IN_SCL. For example, poll the TIS string character by character, calculate the number of parentheses; if the character is an English left parenthesis, increment the count by 1; if the character is an English right parenthesis, decrement the count by 1. If the number of parentheses is 0 and the character is an English comma, it indicates the end of a field; concatenate these characters into a string and store it as a field in the target table's insert field list IN_TCL.

[0082] Next, the primary key, update, and insert field lists of the source and target tables are collected together, deduplicated, sorted, and stored in the target table field list TCL and the source table field list SCL, respectively.

[0083] For UPDATE types, the elements after the SET keyword and before the WHERE keyword are concatenated into a string UPS. The UPS string is iterated character by character, and the number of parentheses is calculated. If the character is an English left parenthesis, the count is incremented by 1; if the character is an English right parenthesis, the count is decremented by 1. If the count is 0 and the character is an English comma, it indicates the end of a field. These characters are then concatenated into a string, which is then split by an equals sign. The left side of the equals sign represents the target table fields, stored in the target table field list TCL; the right side of the equals sign represents the source table fields, stored in the source table field list SCL.

[0084] Next, poll TCL to obtain information such as field type and Chinese name of the field based on the metadata.

[0085] Finally, the target table field information TCD (stored in key-value pairs) is compiled as follows: Number: Field Name, Field Type, Chinese Field Name (or Field Name), Primary Key Information.

[0086] S205. Obtain source table field information SCD and field alias list SAL.

[0087] S206. Obtain source table information (STD), filtering conditions, grouping clauses, and sorting clauses.

[0088] The process of obtaining source table field information (SCD) can be found in [reference needed]. Figure 5 , Figure 5 This is a schematic diagram illustrating the process of obtaining source table field information provided in an embodiment of this application. For example... Figure 5 As shown: Based on the processing type determination obtained in step S202, for MERGE and UPDATE types, the source table field information SCD is constructed by incrementing the number of the source table field list SCL. For INSERT and REPLACE types, the SELECT clause is split into two parts: the SELECT field and the FROM field. The SELECT field part is parsed, and the number is incremented to construct the source table field information SCD and the field alias list SAL. The SELECT field part can also be referred to as the first part, and the FROM field part can also be referred to as the second part.

[0089] The process of parsing the SELECT field part can be found in [reference needed]. Figure 6 , Figure 6 This is a flowchart illustrating the parsing of the SELECT field portion as provided in an embodiment of this application. Figure 6 As shown: Parse the SELECT field part -SS string to determine if it is SELECT*.

[0090] If it's a SELECT * statement, the source table field information is equivalent to the target table field information; otherwise, it iterates through the SS string. If the number of parentheses is 0 and it's an English comma, it indicates a single field. These characters are concatenated into the string CS, and then split into the source table field list SCL by spaces or newlines. For example, the SS string is parsed character by character, and the number of parentheses is calculated. If the character is an English left parenthesis, the number of parentheses is incremented by 1; if the character is an English right parenthesis, the number of parentheses is decremented by 1. If the number of parentheses is 0 and the character is an English comma, it indicates the end of a field. These characters are concatenated into the string CS as a single field, and then split into the field list SCL by spaces and newlines.

[0091] Next, poll SCL to determine if the list element string contains "AS" or ")AS". If so, split the string CS according to the regular expression "AS" or ")AS". One part is the source table field information, numbered incrementally, and stored as key-value pairs (SCD). The other part is the field alias, which is stored in the field alias list (SAL).

[0092] If the list element string does not contain "AS" or ")AS", then check if the number of elements in the SCL list is equal to 1. If it is, it indicates a single field. Increment the number and store the source table field information SCD in key-value pair format. The field alias is the same as the field name, and the field name is stored in the field alias list SAL.

[0093] If the SCL list has more than one element, then check if the last element is a corresponding target table field. If it is, remove the last element, increment the number, and store the source table field information (SCD) in key-value pair format. The last element is the field alias, which is stored in the field alias list (SAL).

[0094] If the last element is not a corresponding target table field, then check if the last field contains parentheses or an evaluated expression. If not, remove the last element, increment the number, and store the source table field information (SCD) in key-value pair format. If the last element is a field alias, store it in the field alias list (SAL). If parentheses or an evaluated expression exist, increment the number, store the source table field information (SCD) in key-value pair format, and set the source table field alias to null before storing it in SAL.

[0095] Ultimately, the source table field information SCD is obtained, and its stored key-value pair format can be: number: source table field.

[0096] The process of parsing the FROM field can be found in [reference needed]. Figure 7 , Figure 7 This is a flowchart illustrating the parsing of the FROM field portion provided in an embodiment of this application. Figure 7As shown: The FROM section is parsed, and the code in the FROM field is split into lists by spaces and newlines. Each element in the list is iterated over, and the number of parentheses is calculated. If the number of parentheses is 0, and the list element contains an English comma, it is considered a source table. For example, if a list element contains one English left parenthesis, the parenthesis count is incremented by 1; if a list element contains one English right parenthesis, the parenthesis count is decremented by 1. If the number of parentheses is 0, and the list element contains an English comma, then these elements together constitute a source table.

[0097] Next, determine the position of the English comma. If the list element has only one English comma, or the English comma is the last character, then add spaces between the list elements to concatenate them into a string, remove the last English comma, store it in the FROM information list FL, and prepend the words INNER JOIN before the next element.

[0098] If the first character of the English comma element is followed by other characters, it indicates that this list element is information from a new source table. In this case, add spaces between the list elements before the current element to form a string, store it in the FROM information list FL, and replace the English comma in the current element with INNER JOIN.

[0099] If the number of parentheses is 0, there exists a previous list element, and the list element is one of the keywords ("INNER", "JOIN", "LEFT", "RIGHT", "FULL", "WHERE", "GROUP", "ORDER"), it means that the list element before this element is a source table information. Concatenate the list elements before the current element with spaces to form a string, store it in the FROM information list FL, and set the previous element to null. If the list element is JOIN, replace it with INNER JOIN.

[0100] Next, the generated FROM information list (FL) is processed and split into eight parts: table schema name, table name, table alias, table join method, table join conditions, filter conditions, grouping clause, and sorting clause. The processing procedure can be found in [link to relevant documentation]. Figure 8 , Figure 8 This is a schematic diagram illustrating the process of obtaining source table information provided in an embodiment of this application. For example... Figure 8 As shown: Poll the list FL, splitting each element into a new list NL by spaces or newlines. Check the value of the first element FC in NL. If the value of FC is one of the keywords (FROM, INNER, LEFT, RIGHT, FULL), for example, if it's FROM, remove FROM from NL and set the join type to empty. If the first word is "JOIN", it indicates an inner join, and the join type is assigned INNER JOIN. Otherwise, concatenate the first two elements of NL with spaces to form a string and assign it to the join type. In these JOIN cases, remove the JOIN information from the NL elements. Finally, concatenate the elements of NL with spaces to form the string FLS.

[0101] The string FLS is split into lists using the "ON" keyword. If a list contains two elements, one part represents the source table information, and the other part represents the inter-table join conditions. Otherwise, the entire FLS string represents the source table information, and the inter-table join conditions are set to empty.

[0102] Next, process the source table information. First, determine if there is a subquery in the source table. If there is, set the source table schema name to null, the subquery to the source table name, and the Chinese name of the source table to null. If there is another element after the subquery, it is the source table alias; otherwise, set the source table alias to null.

[0103] If no subquery exists, the first word of the source table information is the source table name. We then check if it contains a period (.). If so, we split it by the period; the first part is the source table schema name, and the second part is the source table name. Otherwise, the source table schema name is left empty. If the first part of the source table information contains a second word, it is the source table alias; otherwise, the alias is empty. Finally, we retrieve the Chinese annotation of the table name from the metadata information and store it as the Chinese name of the source table.

[0104] Next, increment the number and use the number as the key. The corresponding value is a list consisting of six parts: source table schema name, source table name, source table alias, source table Chinese name, inter-table join method, and inter-table association conditions. That is, source table information STD: number: source table schema name, source table name, source table Chinese name, source table alias, inter-table join method, inter-table association conditions.

[0105] If the value of the first element FC in a NL is the keyword "WHERE", then the corresponding FL element in this NL is the filter condition. The "WHERE" keyword can also be called the first keyword; the "GROUP" keyword can also be called the second keyword; and the "ORDER" keyword can also be called the third keyword.

[0106] If the value of the first element FC in the NL is the keyword "GROUP", then the corresponding FL element in this NL is a grouping clause.

[0107] If the value of the first element FC in the NL is the keyword "ORDER", then the corresponding FL element in this NL is the sorting clause.

[0108] Finally, the SQL statement is parsed to obtain the target table information (TTL), target table field information (TCD), source table field information (SCD), source table information (STD), filter conditions, grouping clauses, and sorting clauses.

[0109] S207. Poll the obtained information and write it into the cells of the data mapping document according to the template format.

[0110] Finally, the obtained program code snippets, target table information TTL, target table field information TCD, source table field information SCD, source table information STD, filter conditions, grouping clauses, and sorting clauses are processed step by step in a loop, and cells of the corresponding format are generated according to the mapped Excel template format.

[0111] S106. Fill in the formatted SQL statements, target table information, target table field information, source table field information, source table information, filter conditions, grouping clauses, and sorting clauses according to the predefined data mapping document template format to generate a data mapping design document.

[0112] Finally, the acquired program code snippets, target table information TTL, target table field information TCD, source table field information SCD, source table information STD, filter conditions, grouping clauses, and sorting clauses are processed step by step in a loop. Based on the mapped Excel template format, corresponding formatted cells are generated, and data is filled into the cells. This completes the reverse engineering of the program code into a mapped design document, achieving design document backfilling and ensuring consistency between the program code and the design document. Subsequent use only requires maintaining the design document; the program code can then be automatically generated through the automated development platform, realizing the design-as-code design philosophy and improving productivity.

[0113] In some embodiments, the method further includes: in response to an update operation on the data mapping design document, generating new program code conforming to the target programming language based on the updated data mapping design document and the target template of the code generation platform.

[0114] This application utilizes a method that automatically responds to updates in a structured data mapping design document, generating new, standard-compliant code using a target template from a code generation platform. This allows for easy switching between different programming languages ​​when generating new code based on the design document, enabling designers and developers to focus more time and effort on implementing business functions and improving work efficiency. Furthermore, code modifications can be made directly within the data mapping design document, which is more convenient and efficient than simply modifying the code itself.

[0115] The data processing program code of this application enables convenient, accurate, and efficient reverse engineering of data mapping design documents, avoiding the reliance on time-consuming and laborious manual revisions of design documents. It also ensures consistency between the design documents and the program code, improving work efficiency and productivity. In some embodiments, data processing may also be referred to as data manipulation, etc., and this application does not limit it to that term.

[0116] This application parses SQL statements and has a complete parsing and judgment processing branch for obtaining target table, source table, and mapping relationship information. It fully considers the SQL writing styles of various data processing scenarios, as well as the different SQL writing styles of different developers and different implementation scenarios, to achieve complete parsing and acquisition of target table, source table, and mapping relationship information, ensuring that the program code information is accurate and not lost.

[0117] Using the processing flow and parsing rules of this application, we can accurately identify flexible and diverse data processing scenarios and various SQL writing styles, thereby improving the accuracy and completeness of identification and judgment.

[0118] Figure 9 This is a schematic diagram of the structure of a data processing device provided in an embodiment of this application. Figure 9 As shown, the data processing device 10 includes: a data acquisition module 11, a data processing module 12, a data parsing module 13, and a design document generation module 14.

[0119] The data acquisition module 11 is used to acquire the program code file and store the program code file as a string.

[0120] The data processing module 12 is used to identify and extract at least one data processing program code from the string in the program code file.

[0121] The data processing module 12 is also used to parse and process each data processing program code to obtain a code segment list containing one or more code segments. The code segments in the code segment list are classified into code segments that do not need to be processed, logical code segments, and structured query language SQL statement code segments.

[0122] The data processing module 12 is also used to preprocess the classified SQL statement code segments to obtain formatted SQL statements.

[0123] The data parsing module 13 is used to parse the formatted SQL statement to obtain data processing information, which includes: target table information, target table field information, source table information, source table field information, filtering conditions, grouping clauses, and sorting clauses.

[0124] The design document generation module 14 is used to fill in the formatted SQL statements, target table information, target table field information, source table field information, source table information, filter conditions, grouping clauses, and sorting clauses according to the predefined data mapping document template format to generate a data mapping design document.

[0125] In other embodiments, parsing the formatted SQL statement to obtain data processing information includes: splitting the formatted SQL statement into a list of elements and determining the processing type of the formatted SQL statement based on the list of elements; parsing the target table schema name, target table name, target table name, and target table alias based on the processing type of the list of elements; and organizing the processing type, target table schema name, target table name, target table name, and target table alias into target table information.

[0126] In other embodiments, parsing the formatted SQL statement to obtain data processing information further includes: parsing a target table field list and a source table field list of the formatted SQL statement based on the processing type and the element list. The target table field list includes one or more target table fields, and the source table field list includes one or more source table fields. The target table fields and the source table fields are used to determine the mapping relationship between the fields. Based on the metadata information in the database, each target table field in the target table field list is queried to obtain the field type, field name, and primary key information of the target table field. Based on the target table fields, the field type, field name, and primary key information of the target table fields, the target table field information is compiled.

[0127] In other embodiments, the data parsing module 13 is further configured to: parse source table field information, field alias list, and source table information list of a formatted SQL statement based on the processing type, element list, and source table field list, wherein the source table information list includes one or more code segments; parse source table information, filtering conditions, grouping clauses, or sorting clauses based on the source table information list, wherein the source table information includes source table schema name, source table name, source table name, source table alias, inter-table join method, and inter-table association conditions; if the target table field information obtained based on the target table fields, target table field types, field names, and primary key information is empty, then reconstruct the target table field information based on the field alias list.

[0128] In other embodiments, the above-mentioned splitting of the formatted SQL statement into an element list and determining the processing type of the formatted SQL statement based on the element list specifically includes: splitting the formatted SQL statement into an element list by spaces and newlines, and determining the processing type of the formatted SQL statement based on the first element in the element list; the above-mentioned parsing to obtain the target table schema name, target table name, target table name, and target table alias based on the processing type and the element list specifically includes: if the processing type is the first type, then the third element in the corresponding element list is recorded as the first identifier, and the fourth element in the element list is recorded as the second identifier; if the processing type is the second type, then the second element in the corresponding element list is recorded as the first identifier, and the third element in the element list is recorded as the second identifier; based on the first identifier, the target table schema name, target table name, and target table name are determined; based on the second identifier, the target table alias is determined.

[0129] In other embodiments, the above-described parsing of the target table field list and source table field list of the formatted SQL statement based on the processing type and element list includes: if the processing type is the third type, then the fourth element in the corresponding element list is marked as the third identifier, and the fifth element in the element list is marked as the fourth identifier, and the first type includes the third type; based on the third identifier, the fourth identifier, and the element list, the target table fields are determined; if the processing type is the fourth type, then based on the corresponding element list, the target table fields and source table fields are determined, the target table fields include the target table primary key field, and the source table fields include the source table primary key field. The target table primary key field is used to determine the primary key status of the target table fields, and the source table primary key field is used to determine the primary key status of the source table fields, and the first type includes the fourth type; if the processing type is the second type, then based on the corresponding element list, the target table fields and source table fields are determined; based on the target table fields and the source table fields, the target table field list and source table field list are compiled.

[0130] In other embodiments, the above-described method of parsing the source table field information, field alias list, and source table information list of a formatted SQL statement based on the processing type, element list, and source table field list specifically includes: if the processing type is the third type, then splitting the corresponding formatted SQL statement into a first part and a second part, wherein the first part is used to characterize the data structure of the formatted SQL statement, and the second part is used to characterize the source and combination method of the data in the formatted SQL statement; based on the first part, determining the source table field information and field alias list in the formatted SQL statement; based on the second part, determining the source table information list in the formatted SQL statement; wherein, if the first element of the second part is a first key, then the second part is a filtering condition; if the first element of the second part is a second key, then the second part is a grouping clause; if the first element of the second part is a third key, then the second part is a sorting clause.

[0131] The data processing device 10 further includes a code generation module 15. This code generation module 15 is used to generate new program code conforming to the target programming language based on the updated data mapping design document and the target template of the code generation platform, in response to an update operation of the data mapping design document.

[0132] The data processing device 10 provided in this application embodiment can execute the method shown in the above method embodiment. Its implementation principle and beneficial effects can be referred to the relevant description in the method embodiment, and will not be repeated here.

[0133] Figure 10 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Figure 10 As shown, the electronic device 20 includes a memory 21 and at least one processor 22.

[0134] The memory 21 is used to store computer program code, which includes computer instructions. These computer instructions run in the electronic device 20 to implement the method shown in the above-described method embodiments. For example, the memory may include high-speed random access memory (RAM), and may also include non-volatile memory (NVM), such as at least one disk storage device, or a USB flash drive, portable hard drive, read-only memory, magnetic disk, or optical disk, etc.

[0135] Processor 22 can be a general-purpose processor, including a Central Processing Unit (CPU), a network processor (NP), etc.; it can also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. Processor 22 can also be other general-purpose processors. The general-purpose processor can be a microprocessor or any conventional processor.

[0136] The memory 21 and processor 22 are communicatively connected. For example, the memory 21 can be connected to the processor 22 via a system bus to complete communication between them. The system bus can be a peripheral component interconnect (PCI) bus, an extended industry standard architecture (EISA) bus, an industry standard architecture (ISA) bus, etc. The system bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used to represent it in the figure, but this does not mean that there is only one bus or one type of bus.

[0137] Optionally, the memory 21 can be either standalone or integrated with the processor 22. When the memory 21 is set up independently, it is connected to the processor 22 via the system bus.

[0138] The electronic device 20 provided in this application embodiment is used to execute the technical solution of the data processing method in the above embodiment.

[0139] This application also provides a chip for executing instructions, which is used to execute the data processing method described in the above embodiments.

[0140] This application also provides a computer-readable storage medium storing computer instructions. When these computer instructions are executed by a processor, they are used to implement the technical solution of the data processing method described in the above embodiments. Specifically, when the computer instructions are executed by a processor, the electronic device 20 can perform the technical solution of the data processing method described in the above embodiments.

[0141] This application also provides a computer program product, which includes a computer program stored in a computer-readable storage medium. At least one processor can read the computer program from the computer-readable storage medium, and when the at least one processor executes the computer program, it can implement the technical solution of the data processing method in the above embodiments.

[0142] The aforementioned computer-readable storage media can be implemented from any type of volatile or non-volatile storage device or a combination thereof, such as Static Random-Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read-Only Memory (EPROM), Programmable Read-Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The computer-readable storage media can be any available medium accessible to a general-purpose or special-purpose computer.

[0143] An exemplary computer-readable storage medium is coupled to a processor, enabling the processor to read information from and write information to the storage medium. Of course, the computer-readable storage medium can also be a component of the processor. The processor and the computer-readable storage medium can reside in an application-specific integrated circuit (ASIC). Alternatively, the processor and the computer-readable storage medium can exist as discrete components in an electronic control unit or main control device; this application does not limit this.

[0144] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices, or modules, and may be electrical, mechanical, or other forms.

[0145] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to implement the solution of this embodiment according to actual needs.

[0146] Furthermore, the functional modules in the various embodiments of this application can be integrated into one processing unit, or each module can exist physically separately, or two or more modules can be integrated into one unit. The unit composed of the above modules can be implemented in hardware or in the form of hardware plus software functional units.

[0147] The integrated modules described above, implemented as software functional modules, can be stored in a computer-readable storage medium. These software functional modules, stored in a storage medium, include several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute some steps of the methods of the various embodiments of this application.

[0148] It should be understood that the steps of the method disclosed in the embodiments of this application can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules in the processor.

[0149] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0150] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

Claims

1. A data processing method, characterized in that, include: Obtain the program code file and store it as a string. Identify and extract at least one data processing program code from the strings in the program code file; Each of the data processing program codes is parsed to obtain a code segment list containing one or more code segments. The code segments in the code segment list are classified into code segments that do not require processing, logic code segments, and structured query language (SQL) statement code segments. The categorized SQL statement code segments are preprocessed to obtain formatted SQL statements; The formatted SQL statement is parsed to obtain data processing information, which includes: target table information, target table field information, source table information, source table field information, filtering conditions, grouping clauses, and sorting clauses. The formatted SQL statement, the target table information, the target table field information, the source table field information, the source table information, the filter conditions, the grouping clause, and the sorting clause are filled in according to the predefined data mapping document template format to generate a data mapping design document.

2. The method according to claim 1, characterized in that, The process of parsing the formatted SQL statement to obtain data processing information includes: The formatted SQL statement is split into a list of elements, and the processing type of the formatted SQL statement is determined based on the list of elements. Based on the processing type and the element list, the target table schema name, target table name, target table name, and target table alias are parsed. The processing type, the target table schema name, the target table name, the target table name, and the target table alias are organized into the target table information.

3. The method according to claim 2, characterized in that, The process of parsing the formatted SQL statement to obtain data processing information also includes: Based on the processing type and the element list, the target table field list and the source table field list of the formatted SQL statement are parsed to obtain the target table field list and the source table field list. The target table field list includes one or more target table fields, and the source table field list includes one or more source table fields. The target table fields and the source table fields are used to determine the mapping relationship between the fields. Based on the metadata information in the database, each target table field in the target table field list is queried to obtain the field type, field name, and primary key information of the target table field; Based on the target table fields, their field types, field names, and primary key information, the target table field information is compiled.

4. The method according to claim 3, characterized in that, The method further includes: Based on the processing type, the element list, and the source table field list, the source table field information, field alias list, and source table information list of the formatted SQL statement are parsed, and the source table information list includes one or more code segments; Based on the source table information list, the source table information, the filtering conditions, the grouping clause or the sorting clause are parsed to obtain the source table information, which includes the source table schema name, source table name, source table name, source table alias, inter-table join method and inter-table association conditions. If the target table field information is found to be empty based on the target table fields, field types, field names, and primary key information, then the target table field information is reconstructed based on the field alias list.

5. The method according to claim 4, characterized in that, The step of splitting the formatted SQL statement into a list of elements and determining the processing type of the formatted SQL statement based on the list of elements specifically includes: The formatted SQL statement is split into an element list by spaces and newlines, and the processing type of the formatted SQL statement is determined based on the first element in the element list. The process of parsing the target table schema name, target table name, target table name, and target table alias based on the processing type and the element list specifically includes: If the processing type is the first type, then the third element in the element list is recorded as the first identifier, and the fourth element in the element list is recorded as the second identifier; If the processing type is the second type, then the second element in the element list is recorded as the first identifier, and the third element in the element list is recorded as the second identifier; Based on the first identifier, the target table schema name, the target table name, and the target table name are determined. Based on the second identifier, the alias of the target table is determined.

6. The method according to claim 5, characterized in that, Based on the processing type and the element list, the target table field list and source table field list of the formatted SQL statement are parsed. The target table field list includes one or more target table fields, and the source table field list includes one or more source table fields, specifically including: If the processing type is the third type, then the fourth element in the corresponding element list is marked as the third identifier, and the fifth element in the element list is marked as the fourth identifier, and the first type includes the third type; The target table fields are determined based on the third identifier, the fourth identifier, and the element list; If the processing type is the fourth type, then based on the corresponding element list, the target table field and the source table field are determined. The target table field includes a target table primary key field, and the source table field includes a source table primary key field. The target table primary key field is used to determine the primary key status of the target table field, and the source table primary key field is used to determine the primary key status of the source table field. The first type includes the fourth type. If the processing type is the second type, then the target table fields and the source table fields are determined based on the corresponding element list; Based on the target table fields and the source table fields, the target table field list and the source table field list are obtained.

7. The method according to claim 4, characterized in that, The process of parsing the formatted SQL statement based on the processing type, the element list, and the source table field list to obtain the source table field information, field alias list, and source table information list specifically includes: If the processing type is the third type, the corresponding formatted SQL statement is split into a first part and a second part. The first part is used to characterize the data structure of the formatted SQL statement, and the second part is used to characterize the source and combination method of the data in the formatted SQL statement. Based on the first part, determine the source table field information and field alias list in the formatted SQL statement; Based on the second part, a list of source table information in the formatted SQL statement is determined; Wherein, if the first element of the second part is the first keyword, then the second part is the filtering condition; if the first element of the second part is the second keyword, then the second part is the grouping clause; if the first element of the second part is the third keyword, then the second part is the sorting clause.

8. The method according to claim 1, characterized in that, The method further includes: In response to the update operation of the data mapping design document, new program code conforming to the target programming language is generated based on the updated data mapping design document and the target template of the code generation platform.

9. A data processing apparatus, characterized in that, include: The data acquisition module is used to acquire program code files and store the program code files as strings; The data processing module is used to identify and extract at least one data processing program code from the strings in the program code file; The data processing module is also used to parse and process each of the data processing program codes to obtain a code segment list containing one or more code segments. The code segments in the code segment list are classified into code segments that do not require processing, logic code segments, and structured query language SQL statement code segments. The data processing module is also used to preprocess the classified SQL statement code segments to obtain formatted SQL statements; The data parsing module is used to parse the formatted SQL statement to obtain data processing information, which includes: target table information, target table field information, source table information, source table field information, filtering conditions, grouping clauses, and sorting clauses. The design document generation module is used to fill in the formatted SQL statement, the target table information, the target table field information, the source table field information, the source table information, the filter conditions, the grouping clause, and the sorting clause according to a predefined data mapping document template format to generate a data mapping design document.

10. An electronic device, characterized in that, include: A memory and at least one processor; the memory is communicatively connected to the processor; the memory is used to store computer program code, the computer program code including computer instructions; when the processor executes the computer instructions, the electronic device performs the method as described in any one of claims 1-8.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that, when executed by a processor, are used to implement the method as described in any one of claims 1-8.

12. A computer program product, characterized in that, When the computer program product is run on a computer / executed by the computer's processor, it implements the method as described in any one of claims 1-8.

Images