Data processing method and apparatus

Abstract

The application provides a data processing method and device, wherein the data processing method comprises: receiving service demand information of a target service, generating a database processing file based on the service demand information, wherein the database processing file contains service parameters and a database processing flow; loading the database processing file in a flow engine according to a service request of the target service; determining a target process statement according to target service data in the service request, the service parameters and the database processing flow in the case that the database processing file contains a process statement; obtaining a target database execution statement in the target process statement, and executing the target database execution statement in a calculation engine. The flow engine is used to realize the analysis of the process statement, thereby enriching the analysis mode of the process statement.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to data processing methods. This application also relates to a data processing apparatus, a computing device, and a computer-readable storage medium. Background Technology

[0002] With the continuous development of computer technology, users have increasingly higher needs for databases.

[0003] Currently, computing engines are typically used to perform calculations on database processing tasks. Furthermore, processing procedure statements can be generated based on the processing statements in the database processing tasks, thereby improving the efficiency of database processing.

[0004] However, when processing data in a database based on procedural statements, it is usually necessary to generate stored procedures and register them with the database to parse the procedural statements. If user permissions to create stored procedures on the database are restricted, procedural statements cannot be used.

[0005] Therefore, efficiently and accurately executing procedural statements has become a technical problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0006] In view of this, embodiments of this application provide a data processing method. This application also relates to a data processing apparatus, a computing device, and a computer-readable storage medium to solve the aforementioned problems existing in the prior art.

[0007] According to a first aspect of the embodiments of this application, a data processing method is provided, including:

[0008] Receive business requirement information for the target business, and generate a database processing file based on the business requirement information, wherein the database processing file contains business parameters and database processing flow;

[0009] Based on the business request for the target business, the database processing file is loaded in the process engine;

[0010] When the database processing file includes procedural statements, the target procedural statement is determined based on the target business data in the business request, the business parameters, and the database processing flow.

[0011] Obtain the target database execution statement from the target process statement, and execute the target database execution statement in the computing engine.

[0012] According to a second aspect of the embodiments of this application, a data processing apparatus is provided, comprising:

[0013] The generation module is configured to receive business requirement information for a target business and generate a database processing file based on the business requirement information, wherein the database processing file contains business parameters and database processing flow.

[0014] The loading module is configured to load the database processing file in the process engine based on the business request for the target business.

[0015] The determination module is configured to determine the target process statement based on the target business data in the business request, the business parameters, and the database processing flow, when the database processing file includes a process statement.

[0016] The execution module is configured to retrieve the target database execution statement from the target process statement and execute the target database execution statement in the computing engine.

[0017] According to a third aspect of the present application, a computing device is provided, including a memory, a processor, and computer instructions stored in the memory and executable on the processor, wherein the processor executes the computer instructions to implement the steps of the data processing method.

[0018] According to a fourth aspect of the embodiments of this application, a computer-readable storage medium is provided that stores computer instructions, which, when executed by a processor, implement the steps of the data processing method.

[0019] The data processing method provided in this application receives business requirement information for a target business, generates a database processing file based on the business requirement information, wherein the database processing file includes business parameters and a database processing flow; loads the database processing file in a process engine according to the business request for the target business; if the database processing file includes process statements, determines a target process statement based on the target business data in the business request, the business parameters, and the database processing flow; obtains a target database execution statement from the target process statement, and executes the target database execution statement in a computing engine.

[0020] One embodiment of this application implements the following: a database processing file is generated based on business requirement information, so as to process the database corresponding to the target business based on the database processing file; the target process statement is determined by the process engine, and then the target database execution statement is obtained from the target process statement, thereby realizing the parsing of the process statement by the process engine, enriching the parsing method of the process statement, and improving the reusability of the process statement; the target database execution statement obtained by the process engine is executed by the calculation engine, thereby realizing the processing of data in the database and improving the data processing efficiency. Attached Figure Description

[0021] Figure 1 This is a schematic diagram of a data processing method provided in an embodiment of this application;

[0022] Figure 2 This is a flowchart of a data processing method provided in an embodiment of this application;

[0023] Figure 3 This is a flowchart illustrating a data processing method applied to fund business, provided in an embodiment of this application.

[0024] Figure 4 This is a schematic diagram of the structure of a data processing device provided in an embodiment of this application;

[0025] Figure 5 This is a structural block diagram of a computing device provided in one embodiment of this application. Detailed Implementation

[0026] Many specific details are set forth in the following description to provide a full understanding of this application. However, this application can be implemented in many other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of this application; therefore, this application is not limited to the specific embodiments disclosed below.

[0027] The terminology used in one or more embodiments of this application is for the purpose of describing particular embodiments only and is not intended to limit the scope of one or more embodiments of this application. The singular forms “a,” “the,” and “the” used in one or more embodiments of this application and in the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” used in one or more embodiments of this application refers to and includes any or all possible combinations of one or more associated listed items.

[0028] It should be understood that although the terms first, second, etc., may be used to describe various information in one or more embodiments of this application, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, first may also be referred to as second without departing from the scope of one or more embodiments of this application, and similarly, second may also be referred to as first. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to a determination."

[0029] First, the terms and concepts involved in one or more embodiments of this application will be explained.

[0030] Database: A database is a "repository for organizing, storing, and managing data according to a data structure." It is a large collection of organized, shareable, and uniformly managed data that is stored permanently in a computer.

[0031] Computation engine: A program specifically designed for data computation.

[0032] Process engine: A program used to drive business processes to flow according to a set fixed process.

[0033] Procedure statements: Statements that contain multiple SQL statements.

[0034] Process coding component: A component used for editing processes.

[0035] SQL statements: SQL stands for "Structured Query Language", a language for manipulating databases, including creating databases, deleting databases, querying records, modifying records, and adding fields.

[0036] MySQL: MySQL is a relational database management system.

[0037] PostgreSQL: PostgreSQL is a feature-rich free software object-relational database management system.

[0038] DDL statements: DDL (Data Definition Language) statements are data definition statements.

[0039] Spark: Spark is a general-purpose big data computing framework, a computing model based on RDD (Resilient Distributed Dataset).

[0040] Information technology application innovation database: that is, database for innovative applications of information technology.

[0041] JDBC (Java Database Connectivity) is a Java API for executing SQL statements. It provides unified access to various relational databases and consists of a set of classes and interfaces written in Java.

[0042] Antlr4: Antlr4 is a parser generation tool that can be used to read, process, execute, and translate structured text or binary files.

[0043] LiteFlow: LiteFlow is a very powerful and modern rules engine framework that combines orchestration features with all the features of a rules engine.

[0044] ETL stands for Extract-Transform-Load, which describes the process of extracting, transforming, and loading data from a source to a destination. While ETL is commonly used in data warehousing, its application is not limited to data warehousing.

[0045] Currently, data computing engines support operations on various relational databases, such as reading and writing data from MySQL, Oracle, and PostgreSQL. However, native computing engines (such as Spark) only support common DDL statements like INSERT and SELECT when writing data to databases, and cannot handle procedural statements.

[0046] The common method for handling procedural statements is to generate a stored procedure from the logic, register it with the database, and then call it using JDBC's native `Call` method. This requires granting users permission to create stored procedures on the database. However, with securities firms, funds, and banks switching to domestically developed databases for data security reasons, they no longer grant permission to register stored procedures, rendering many SQL functions based on procedural statements unusable. Currently, the common approach is to use Antlr4 for SQL parsing to implement procedural statements or to implement this logic through hard coding.

[0047] While Antlr4 can implement procedural SQL parsing logic, it suffers from the problem of needing to adapt to different database dialects, making syntax expansion extremely difficult. Hard-coding, on the other hand, results in low data processing efficiency, lacks distributed processing capabilities, and presents single points of failure and project maintenance difficulties.

[0048] This application addresses the issue of SQL queries implemented using stored procedures becoming unusable in scenarios where clients in the financial sector, such as securities firms and funds, no longer grant access to stored procedures. The solution is based on secondary development of components from a workflow engine. It uses the workflow engine's ifelse, loop, and while components to express ifelse, loop, and while statements within the procedure statements, as well as for loop, while, and switch components. Other DDL statements are executed using a computation engine. Based on a computation engine (such as Spark) and a workflow engine (such as LiteFlow), this solution enables procedures to be executed outside the database, and the data is then written back to the database via the computation engine. This solves the problems of incomplete SQL parsing, difficulty in expansion, low data processing efficiency, and single points of failure associated with using Antlr4. Specifically, it involves component development based on a workflow engine (such as LiteFlow), using custom components for task orchestration, resolving the issue of if, loop, and cursor statements in procedure statements being unexpressable using a computation engine (such as Spark), and then using the computation engine to execute DDL statements. This solves the problem of executing procedure statements outside the database using a computation engine when users do not have access to stored procedures, and then writing the data back to the database. Furthermore, a drag-and-drop interface was used to orchestrate tasks, improving the user experience. Then, a computing engine was used to perform ETL processing on data from various databases, resolving the single point of failure issue in the application. This improved the reusability of business SQL and the scalability of the project.

[0049] This application provides a data processing method, and also relates to a data processing apparatus, a computing device, and a computer-readable storage medium, which will be described in detail in the following embodiments.

[0050] Figure 1 The illustration shows a scenario diagram of a data processing method according to an embodiment of this application, specifically including:

[0051] Users generate business requirements for their target business by dragging and dropping components in the web UI. The UI allows for the import and export of workflow tasks, enabling workflow task editing and job import / export. Users can also edit workflow tasks by dragging and dropping components. Furthermore, users can fill in SQL statements, along with if-else, for, while, and switch statements for conditional statements, loops, and loop exit conditions, generating a flowchart in the UI. An SQL statement parser analyzes the SQL statements in the flowchart, converting them into corresponding variable modules, input modules, process statement recognition and conversion modules, and writing modules. This generates an executable XML file that the workflow engine can run, which can then be exported via the UI. In short, the SQL statement parser can parse the SQL statements in the UI into corresponding variable modules, input modules, process statement recognition and conversion modules, and writing modules.

[0052] Furthermore, based on the business request for the target business, the executable file of the process engine is passed to the process engine, which is the flow module. The process engine parses the file and executes it to start the job process. Each module in the process engine parses the SQL into SQL statements that the computing engine can execute and submits them to the computing engine for execution. The computing engine reads and writes data to the data source through connectors. The connectors include database connectors and file connectors, which support reading and writing database and file data sources. The data sources include files and database 1, database 2, database 3, etc.

[0053] Finally, the computing engine is responsible for executing SQL statements, pulling data from the data source into memory for ETL processing through the connector. After the data processing is complete, when the process engine reaches the write module, it writes the final processed data back to the data source, i.e., the database or file, through the computing engine.

[0054] The data processing method of this application generates a database processing file based on business requirement information, so as to process the database corresponding to the target business based on the database processing file; the process engine determines the target process statement, and then obtains the target database execution statement from the target process statement, thereby realizing the parsing of the process statement by the process engine, enriching the parsing method of the process statement and improving the reusability of the process statement; the calculation engine executes the target database execution statement obtained by the process engine, thereby realizing the processing of data in the database and improving the data processing efficiency.

[0055] Figure 2 A flowchart of a data processing method according to an embodiment of this application is shown, which specifically includes the following steps:

[0056] Step 202: Receive business requirement information for a target business, and generate a database processing file based on the business requirement information. The database processing file includes business parameters and a database processing flow.

[0057] The target business refers to a business that stores database processing requirements, such as a purchase business, a query business, a login business, etc. The present application does not make specific limitations. Business requirement information refers to requirement information for generating a database processing file corresponding to the target business. A database processing file refers to a file that can be executed by a process engine. The database processing file includes business parameters and a database processing flow. Business parameters refer to parameters used to determine whether business data can execute corresponding statements. The database processing flow refers to a business database processing flow generated based on the business requirement information.

[0058] Specifically, the server receives business requirement information for a target business, and generates a database processing file including business parameters and a database processing flow according to the business requirement information.

[0059] In a specific embodiment of the present application, the server receives fund business requirement information for a fund business, and generates a database processing file corresponding to the fund business according to the fund business requirement information. The database processing file includes fund business parameters and a fund business database processing flow.

[0060] By receiving business requirement information, and then generating a database processing file based on the business requirement information, so as to implement the processing of data in the database based on the database processing file subsequently.

[0061] In practical applications, the method for generating a database processing file based on the business requirement information may include:

[0062] Obtain a process orchestration component according to the business requirement information;

[0063] Add at least one business parameter and database execution statements corresponding to each business parameter in the process orchestration component to construct a database processing flow;

[0064] Generate a database processing file based on the database processing flow.

[0065] The process orchestration component refers to a component that has a predefined execution process. For example, the process encoding component can be an if-else statement framework, a for-whlie statement framework, a switch-case statement framework, etc. Database execution statements refer to database processing statements that can be executed under the preset conditions corresponding to business parameters. For example, when business data 5 meets the preset condition 1 < i < 6 corresponding to business parameter i, the SQL statement corresponding to the preset condition 1 < i < 6 can be executed.

[0066] Specifically, based on business requirements information, the process orchestration component corresponding to the business requirements information is obtained from the preset component library; business parameters, preset conditions corresponding to the business parameters, and database execution statements that need to be executed under the preset conditions are added to the process orchestration component, and the process orchestration component with added business parameters, preset conditions, and database processing flow is used as the database processing flow; and a database processing file is generated according to the database processing flow.

[0067] In one specific embodiment of this application, based on fund business requirement information, a process orchestration component corresponding to the business requirement information is obtained from a preset component library, including an if-else statement framework. Specifically, the user can drag and drop the process orchestration component corresponding to the fund business requirement information from the component bar of the process orchestration interface to the current orchestration interface. Further, a fund business parameter i, a preset condition ">0" corresponding to the business parameter i, and a database execution statement SQL1 executable under the preset condition are added to the process orchestration component to obtain the constructed database processing flow. A database processing file corresponding to the fund business is generated according to the database processing flow.

[0068] By acquiring process orchestration components based on business requirements, custom task orchestration can be achieved, solving the problem that process statements cannot be expressed using the computing engine; database processing files are generated based on the constructed database processing flow, so that subsequent processing of data in the database can be achieved through the database processing files.

[0069] In practical applications, methods for generating database processing files based on the aforementioned database processing flow may include:

[0070] Parse the database processing flow to generate at least one business data processing module corresponding to the database processing flow;

[0071] Generate database processing files based on each business data processing module.

[0072] The business data processing module refers to the module used to process a portion of the business logic in the database processing flow.

[0073] Specifically, after obtaining the database processing flow, it is impossible to determine when to invoke the processing steps within the database processing flow. Therefore, the database processing flow can be further analyzed to determine the processing functions corresponding to each part of the database processing flow. Based on the database processing flow corresponding to each processing function, corresponding business data processing modules are generated. Then, the business data processing modules are arranged according to a preset order. For example, the preset conditions corresponding to the business parameters in the database processing flow are used to determine whether the business data meets the execution requirements. An execution judgment module can be generated based on the preset conditions. Similarly, one or more business processing modules are generated based on the database processing flow. Then, based on the business processing modules and the preset processing order corresponding to the business processing modules, a database processing file with an execution order is generated.

[0074] In one specific embodiment of this application, the database processing flow is parsed to generate business data processing modules corresponding to the database processing flow, including a variable module, an input module, a process statement recognition module, and a writing module; the business data processing modules are sorted according to a preset order, and the sorted business data processing modules are converted to obtain an XML format database processing file.

[0075] By parsing the database processing flow, the business data processing module is obtained; then, based on the business data processing module, a database processing file is generated, which facilitates the subsequent execution of the database processing file according to the execution order corresponding to the business data processing module.

[0076] Step 204: Load the database processing file in the process engine according to the business request for the target business.

[0077] Here, a business request refers to a request to process database data based on the business data of the target business; a process engine refers to an engine that drives the business to flow according to a set fixed process. In this embodiment, the process engine can drive the flow of the process based on database processing files.

[0078] Specifically, upon receiving a business request for a target business, the business request is forwarded to the process engine; the process engine loads the database processing file based on the business identifier of the target business contained in the business request.

[0079] In one specific embodiment of this application, a business request for fund business is received; the business request is forwarded to the process engine, so that the process engine loads the database processing file corresponding to the fund business.

[0080] Based on business requests, the process engine loads database processing files so that subsequent database execution statements can be executed based on the database data files.

[0081] In practical applications, the method for loading the database processing file in the process engine based on the business request for the target business may include:

[0082] Parse the business request to obtain the target business data corresponding to the target business;

[0083] Based on the target business data, the database processing file is loaded into the process engine.

[0084] Among them, target business data refers to business data in the target business; specifically, the business request is parsed to obtain the target business data contained in the business request, or the business identifier contained in the business request is determined and the target business data is obtained based on the business identifier; according to the target business data, the process engine loads the database processing file corresponding to the target business data.

[0085] In one specific embodiment of this application, the fund business request is parsed to obtain the fund business data carried by the business request; the process engine loads the corresponding database processing file based on the fund business data, wherein the database processing file contains business parameters that can be used to judge the fund business data.

[0086] By parsing the business request, the target business data is obtained, and then a database processing file is loaded based on the target business data so that subsequent data processing in the database can be performed based on the loaded database file.

[0087] Step 206: If the database processing file includes a procedure statement, determine the target procedure statement based on the target business data in the business request, the business parameters, and the database processing flow.

[0088] Among them, a procedure statement refers to a statement consisting of two or more database execution statements; a target procedure statement refers to any procedure statement selected from the database processing file; and a database execution statement refers to a statement that processes data in the database, such as an SQL statement.

[0089] Specifically, the database processing file contains procedural statements and database execution statements. After determining the procedural statements in the database processing file, the target procedural statements that can be executed by the target business data are determined based on the target business data, business parameters, and database processing flow in the business request. That is, the business parameters contained in the target procedural statements can be used to determine the target business data.

[0090] In a specific embodiment of this application, it is determined that the database processing file corresponding to the fund business contains procedural statements; the fund business request is parsed to obtain fund business data; based on the fund business data, the business parameters corresponding to the fund business data, and the database processing flow, the target procedural statement is determined in the procedural statements; specifically, the business parameter i corresponding to the fund business data is determined, and then the target procedural statement is determined as {if(i>1) execute SQL1, else execute SQL2} based on the fund business data and the business parameter i.

[0091] By including procedural statements in the database processing file, the target procedural statement is determined based on the target business data, business parameters, and database processing flow, so that subsequent adjustments to the database data can be made based on the target procedural statement.

[0092] Furthermore, if the database processing file includes procedural statements, before determining the target procedural statement based on the target business data in the business request, the business parameters, and the database processing flow, the following is also included:

[0093] The process engine determines the statement type of the processing statement in the database processing file, wherein the statement type includes procedural statements and database execution statements;

[0094] Accordingly, when the database processing file includes procedural statements, the target procedural statement is determined based on the target business data in the business request, the business parameters, and the database processing flow, including:

[0095] The target business parameters of the target business data are determined in the database processing flow.

[0096] Based on the target business data and the target business parameters, determine the target process statement.

[0097] Among them, the target business parameters refer to the business parameters in the database processing flow that correspond to the target business data.

[0098] Specifically, the process engine loads the database processing file and determines the statement type of the processing statements contained in the database processing file to determine whether the database processing file contains procedural statements. If the database processing flow contains procedural statements, the target business parameters are determined in the database processing flow based on the target business data. The target procedural statement is determined based on the target business data and the target business parameters. That is, the target procedural statement is determined by the correspondence between the target business data and the preset conditions corresponding to the target business parameters. For example, if the target business parameter corresponding to the target business data is i, and the preset condition corresponding to i is determined to be ifi>0 to execute SQL1 statement, then this procedural statement is taken as the target procedural statement.

[0099] In a specific embodiment of this application, the target business parameters corresponding to the fund business data are obtained in the database processing flow of the fund business; the correspondence between the fund business data and the preset conditions of the target business parameters is determined, and the target process statement is determined according to the correspondence. That is, if the fund business data is determined to be i=6, and the preset conditions corresponding to the target business parameters are i>1 and i<1, then the process statement containing i>1 is taken as the target process statement.

[0100] By determining the target business parameters of the target business data in the database processing flow, and then determining the target procedure statement in the database processing flow based on the target business parameters and the target business data, the efficiency of filtering the target procedure statement is improved.

[0101] Step 208: Obtain the target database execution statement in the target process statement, and execute the target database execution statement in the computing engine.

[0102] Among them, the target database execution statement refers to the database execution statement parsed in the target procedure statement.

[0103] Specifically, after the target procedure statement is determined, it is parsed to obtain the target database execution statement contained within it; the calculation engine then executes the target database execution statement to process the data in the database.

[0104] In one specific embodiment of this application, the target process statement corresponding to the fund business is parsed; the target process statement is parsed to obtain the target database execution statement contained in the target process statement, specifically, the target database execution statement {deletefromstudentwhereid=10} is parsed from the target process statement {if(i>1)deletefromstudentwhereid=10}, and the target database execution statement is executed by the computing engine.

[0105] By obtaining the target database execution statement from the target process statement, the process engine can parse the process statement, avoiding the need for the calculation engine to parse the process statement, thereby improving the processing efficiency of the calculation engine.

[0106] In practical applications, after obtaining the target database execution statement in the target procedure statement, the following is also included:

[0107] The execution statement for the target database is sent to the computing engine.

[0108] Specifically, after the process engine determines the target process statement based on the target business data, business parameters, and data processing flow, it sends the target database execution statement in the target process statement to the computing engine, so that the computing engine can execute the target database execution statement.

[0109] The target database execution statement obtained by the process engine is sent to the calculation engine so that the calculation engine can execute the target database execution statement to process the data in the database.

[0110] Furthermore, the method for executing the target database execution statement in the computing engine may include:

[0111] The target data is obtained by executing statements based on the target database in the computing engine.

[0112] Add the target data to the target database.

[0113] Here, target data refers to the data determined by the execution statement based on the target database; target database refers to the database where the data adjusted by the execution statement of the target database is located.

[0114] Specifically, the process involves obtaining the data to be adjusted based on the execution statement in the target database; processing the data to be adjusted according to the execution statement in the target database to obtain the target data; and adding the target data to the target database corresponding to the execution statement in the target database.

[0115] Furthermore, the method for obtaining target data by executing statements based on the target database in the computing engine may include:

[0116] The data source is determined based on the aforementioned database processing flow;

[0117] The computing engine retrieves data from the data source.

[0118] The target database is used to execute statements to update the data source data, thereby obtaining the target data.

[0119] Here, "data source" refers to the source of data used to adjust data in the database; "data source data" refers to data obtained from the data source.

[0120] Specifically, the data source is parsed within the database processing flow; the computing engine retrieves the data source data from the corresponding data source, such as data from tables, databases, or documents; and the data source data is updated according to the execution statements of the target database to obtain the target data.

[0121] In one specific embodiment of this application, the database processing flow is parsed to obtain the data source as database A, which is the storage location of the data to be processed; the calculation engine obtains the data source data "name: q" from the data source, which is the data to be adjusted; based on the data correction information in the execution statement of the target database, the data source data is processed to adjust the data source data "name: q" to "name: s", and "name: s" is used as the target data.

[0122] By acquiring data from the data source through the computing engine, the data source data can be updated by executing statements based on the target database to obtain the target data. This enables the computing engine to extract and process the data, so that the target database can be adjusted based on the target data in the future.

[0123] The above describes the process of processing procedural statements in a database processing file to obtain the target database execution statement. However, in practical applications, a database processing file can contain both procedural statements and database execution statements. Therefore, the execution of the database execution statement can be achieved based on the following steps:

[0124] If the processing statement in the data processing file is a database execution statement, the database execution statement is sent to the computing engine;

[0125] The database execution statement is executed in the computing engine.

[0126] Specifically, if the processing statement to be executed is determined to be a database execution statement, the database execution statement is sent to the computing engine, which then executes the database execution statement without requiring further processing by the process engine.

[0127] In one specific embodiment of this application, if the processing statement in the data processing file is determined to be a database execution statement, then the process engine sends the database execution statement to the calculation engine; and the calculation engine executes the database execution statement.

[0128] By determining the statement type of the processing statements in the data processing file, if the statement type is a database execution statement, then the process engine does not need to further parse the processing statements, thereby improving the processing efficiency of the processing statements.

[0129] The data processing method of this application receives business requirement information for a target business, generates a database processing file based on the business requirement information, wherein the database processing file includes business parameters and a database processing flow; loads the database processing file in a process engine according to the business request for the target business; if the database processing file includes process statements, determines a target process statement based on the target business data in the business request, the business parameters, and the database processing flow; obtains a target database execution statement from the target process statement, and executes the target database execution statement in a computing engine.

[0130] One embodiment of this application implements the following: a database processing file is generated based on business requirement information, so as to process the database corresponding to the target business based on the database processing file; the target process statement is determined by the process engine, and then the target database execution statement is obtained from the target process statement, thereby realizing the parsing of the process statement; the target database execution statement obtained by the process engine is executed by the calculation engine, thereby realizing the processing of data in the database.

[0131] The following is in conjunction with the appendix Figure 3 Taking the application of the data processing method provided in this application in fund business as an example, the data processing method will be further explained. Figure 3 This application provides a flowchart illustrating a data processing method for fund operations according to an embodiment of the present application, which specifically includes the following steps:

[0132] Step 302: Arrange the fund business process in the user interface and generate the fund database processing file.

[0133] Specifically, the process statements are broken down, and the ifelse, loop, while, and switch operations in the process statements are represented using custom components. The statements that the calculation engine can execute, as well as the loop parameters and conditions, are filled into the corresponding components. The user interface is used to edit and generate an XML file that the process engine can execute, which is the fund database processing file.

[0134] Step 304: Load the fund database processing file into the process engine based on the business request for fund business.

[0135] Step 306: The process engine parses the fund business data in the business request.

[0136] Specifically, the process engine starts executing the job based on the file processing of the fund database, generates a shared computing engine session object, and puts the object into the slot of the process engine, so that the entire process object can share a single computing engine session. When the process engine executes the variable module, it puts the variable into the slot of the process engine in the form of key-value pairs for use by subsequent modules.

[0137] Step 308: The calculation engine obtains data from the data source based on the database processing flow in the fund database processing file.

[0138] Specifically, the process engine executes the input module of the database processing flow. The calculation engine pulls data from different data sources to the Driver node of the calculation engine based on the data source information and query SQL in the input module, and generates an in-memory table. If the table fields have indexes, they can be read in a distributed manner. Based on the configured JDBC connection information, the calculation engine reads table data from different data sources to generate an in-memory table.

[0139] Step 310: The process engine determines and parses the process statements in the fund database processing file to obtain the SQL statements.

[0140] Specifically, the process engine continues execution. If it executes a procedural statement, it uses ifelse, for, while, and switch statements from the custom components to express it. Then, it processes the SQL statement using statements that the calculation engine can recognize, through the SQL statements supported by the calculation engine. If it is not a procedural statement, it uses the general Transform module to parse it and submits the SQL statement to the calculation engine for execution to generate an in-memory table for transformation.

[0141] Step 312: The SQL statement is executed by the computing engine.

[0142] Specifically, the process engine executes the write module and submits the SQL statements in the write module to the calculation engine for execution. The calculation engine then writes the transformed data back to the data source, completing the data writing operation to the database.

[0143] The data processing method provided in this application receives business requirement information for a target business, generates a database processing file based on the business requirement information, wherein the database processing file includes business parameters and a database processing flow; loads the database processing file in a process engine according to the business request for the target business; if the database processing file includes process statements, determines a target process statement based on the target business data in the business request, the business parameters, and the database processing flow; obtains a target database execution statement from the target process statement, and executes the target database execution statement in a computing engine.

[0144] In this embodiment, the data processing method is handled by a workflow engine that manages the workflow flow. The core operations that the computing engine needs to perform include distributed data reading, creating temporary tables, performing transformation operations on table data, and combining the workflow with custom components of the workflow engine to orchestrate various database stored procedure logic replacement operations.

[0145] Corresponding to the above method embodiments, this application also provides data processing apparatus embodiments. Figure 4 A schematic diagram of the structure of a data processing apparatus according to an embodiment of this application is shown. Figure 4 As shown, the device includes:

[0146] The generation module 402 is configured to receive business requirement information for a target business and generate a database processing file based on the business requirement information, wherein the database processing file contains business parameters and database processing flow.

[0147] Loading module 404 is configured to load the database processing file in the process engine based on a business request for the target business.

[0148] The determination module 406 is configured to determine the target process statement based on the target business data in the business request, the business parameters, and the database processing flow when the database processing file includes a process statement.

[0149] The execution module 408 is configured to obtain the target database execution statement in the target process statement and execute the target database execution statement in the computing engine.

[0150] Optionally, the generation module 402 is further configured to:

[0151] Based on the aforementioned business requirements information, process orchestration components are obtained;

[0152] Add at least one business parameter and the corresponding database execution statement to the process orchestration component to construct the database processing flow.

[0153] Generate a database processing file based on the aforementioned database processing flow.

[0154] Optionally, the generation module 402 is further configured to:

[0155] Parse the database processing flow to generate at least one business data processing module corresponding to the database processing flow;

[0156] Generate database processing files based on each business data processing module.

[0157] Optionally, the loading module 404 is further configured to:

[0158] Parse the business request to obtain the target business data corresponding to the target business;

[0159] Based on the target business data, the database processing file is loaded into the process engine.

[0160] Optionally, the determining module 406 is further configured to:

[0161] The process engine determines the statement type of the processing statement in the database processing file, wherein the statement type includes procedural statements and database execution statements;

[0162] Accordingly, when the database processing file includes procedural statements, the target procedural statement is determined based on the target business data in the business request, the business parameters, and the database processing flow, including:

[0163] The target business parameters of the target business data are determined in the database processing flow.

[0164] Based on the target business data and the target business parameters, determine the target process statement.

[0165] Optionally, the device further includes a switching module configured to:

[0166] The execution statement for the target database is sent to the computing engine.

[0167] Optionally, the execution module 408 is further configured to:

[0168] The target data is obtained by executing statements based on the target database in the computing engine.

[0169] Add the target data to the target database.

[0170] Optionally, the execution module 408 is further configured to:

[0171] The data source is determined based on the aforementioned database processing flow;

[0172] The computing engine retrieves data from the data source.

[0173] The target database is used to execute statements to update the data source data, thereby obtaining the target data.

[0174] Optionally, the device further includes an execution submodule configured to:

[0175] If the processing statement in the data processing file is a database execution statement, the database execution statement is sent to the computing engine;

[0176] The database execution statement is executed in the computing engine.

[0177] The data processing apparatus provided in this application receives business requirement information for a target business, generates a database processing file based on the business requirement information, wherein the database processing file includes business parameters and a database processing flow; loads the database processing file in a process engine according to the business request for the target business; if the database processing file includes process statements, determines a target process statement based on the target business data in the business request, the business parameters, and the database processing flow; obtains a target database execution statement from the target process statement, and executes the target database execution statement in a computing engine.

[0178] One embodiment of this application implements the following: a database processing file is generated based on business requirement information, so as to process the database corresponding to the target business based on the database processing file; the target process statement is determined by the process engine, and then the target database execution statement is obtained from the target process statement, thereby realizing the parsing of the process statement by the process engine, enriching the parsing method of the process statement, and improving the reusability of the process statement; the target database execution statement obtained by the process engine is executed by the calculation engine, thereby realizing the processing of data in the database and improving the data processing efficiency.

[0179] The above is an illustrative scheme of a data processing apparatus according to this embodiment. It should be noted that the technical solution of this data processing apparatus and the technical solution of the data processing method described above belong to the same concept. For details not described in detail in the technical solution of the data processing apparatus, please refer to the description of the technical solution of the data processing method described above.

[0180] Figure 5A structural block diagram of a computing device 500 according to an embodiment of this application is shown. The components of the computing device 500 include, but are not limited to, a memory 510 and a processor 520. The processor 520 is connected to the memory 510 via a bus 530, and a database 550 is used to store data.

[0181] The computing device 500 also includes an access device 540, which enables the computing device 500 to communicate via one or more networks 560. Examples of these networks include a Public Switched Telephone Network (PSTN), a Local Area Network (LAN), a Wide Area Network (WAN), a Personal Area Network (PAN), or a combination of communication networks such as the Internet. The access device 540 may include one or more of any type of wired or wireless network interface (e.g., a Network Interface Card (NIC)), such as an IEEE 802.11 Wireless Local Area Network (WLAN) interface, a Wi-MAX interface, an Ethernet interface, a Universal Serial Bus (USB) interface, a cellular network interface, a Bluetooth interface, a Near Field Communication (NFC) interface, and so on.

[0182] In one embodiment of this application, the aforementioned components of the computing device 500 and Figure 5 Other components, not shown, can also be connected to each other, for example, via a bus. It should be understood that... Figure 5 The block diagram of the computing device shown is for illustrative purposes only and is not intended to limit the scope of this application. Those skilled in the art can add or replace other components as needed.

[0183] The computing device 500 can be any type of stationary or mobile computing device, including mobile computers or mobile computing devices (e.g., tablet computers, personal digital assistants, laptop computers, notebook computers, netbooks, etc.), mobile phones (e.g., smartphones), wearable computing devices (e.g., smartwatches, smart glasses, etc.) or other types of mobile devices, or stationary computing devices such as desktop computers or PCs. The computing device 500 can also be a mobile or stationary server.

[0184] The processor 520 executes the computer instructions to implement the steps of the data processing method.

[0185] The above is an illustrative scheme of a computing device according to this embodiment. It should be noted that the technical solution of this computing device and the technical solution of the data processing method described above belong to the same concept. For details not described in detail in the technical solution of the computing device, please refer to the description of the technical solution of the data processing method described above.

[0186] An embodiment of this application also provides a computer-readable storage medium storing computer instructions that, when executed by a processor, implement the steps of the data processing method described above.

[0187] The above is an illustrative scheme of a computer-readable storage medium according to this embodiment. It should be noted that the technical solution of this storage medium and the technical solution of the data processing method described above belong to the same concept. For details not described in detail in the technical solution of the storage medium, please refer to the description of the technical solution of the data processing method described above.

[0188] The foregoing has described specific embodiments of this application. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired results. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0189] The computer instructions include computer program code, which may be in the form of source code, object code, executable file, or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording media, USB flash drive, portable hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium may be appropriately added to or subtracted according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media may not include electrical carrier signals and telecommunication signals.

[0190] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0191] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0192] The preferred embodiments disclosed above are merely illustrative of this application. The optional embodiments do not exhaustively describe all details, nor do they limit the invention to the specific implementations described. Clearly, many modifications and variations can be made based on the content of this application. These embodiments are selected and specifically described in this application to better explain the principles and practical applications of this application, thereby enabling those skilled in the art to better understand and utilize this application. This application is limited only by the claims and their full scope and equivalents.

Claims

1. A data processing method, characterized by, The method comprises the following steps: receiving service requirement information for a target service, and obtaining a flow arrangement component according to the service requirement information; adding at least one service parameter and a database execution statement corresponding to each service parameter in the flow arrangement component, and constructing a database processing flow; parsing the database processing flow to generate at least one service data processing module corresponding to the database processing flow; and generating a database processing file according to each service data processing module, wherein the service parameter refers to a parameter used to determine whether service data can execute a corresponding statement; loading the database processing file in a flow engine according to a service request for the target service; in the case where the database processing file includes a process statement, determining a target process statement according to target service data in the service request, the service parameter and the database processing flow; obtaining a target database execution statement in the target process statement, and executing the target database execution statement in a computing engine.

2. The method of claim 1, wherein, The method comprises the following steps: parsing the service request to obtain target service data corresponding to the target service; loading the database processing file in the flow engine based on the target service data.

3. The method of claim 1, wherein, in the case where the database processing file includes a process statement, determining a target process statement according to target service data in the service request, the service parameter and the database processing flow, before the method further comprises the following steps: determining the statement type of the processing statement in the database processing file in the flow engine, wherein the statement type includes a process statement and a database execution statement; correspondingly, in the case where the database processing file includes a process statement, determining a target process statement according to target service data in the service request, the service parameter and the database processing flow, comprises the following steps: determining a target service parameter of the target service data in the database processing flow; determining a target process statement according to the target service data and the target service parameter.

4. The method of claim 1, wherein, after obtaining a target database execution statement in the target process statement, the method further comprises the following steps: sending the target database execution statement to the computing engine.

5. The method of claim 4, wherein, executing the target database execution statement in the computing engine, comprising the following steps: obtaining target data based on the target database execution statement in the computing engine; adding the target data to a target database.

6. The method of claim 5, wherein, obtaining target data based on the target database execution statement in the computing engine, comprising the following steps: determining a data source based on the database processing flow; obtaining data source data in the data source through the computing engine; updating the data source data according to the target database execution statement to obtain target data.

7. The method of claim 1, wherein, the method further comprises the following steps: in the case where the processing statement in the data processing file is a database execution statement, sending the database execution statement to the computing engine; executing the database execution statement in the computing engine.

8. A data processing apparatus, characterized by, The method comprises the following steps: generating a module configured to receive service requirement information for a target service, and obtaining a flow arrangement component according to the service requirement information; adding at least one business parameter and a database execution statement corresponding to each business parameter in the flow orchestration component to construct a database processing flow; parsing the database processing flow to generate at least one business data processing module corresponding to the database processing flow; and generating a database processing file according to each business data processing module, wherein the business parameter refers to a parameter used to determine whether business data can execute a corresponding statement; a loading module configured to load the database processing file in a flow engine according to a business request for the target business; a determining module configured to determine a target process statement according to target business data in the business request, the business parameter, and the database processing flow in the case that the database processing file includes a process statement; an executing module configured to obtain a target database execution statement in the target process statement and execute the target database execution statement in a computing engine.

9. A computing device comprising a memory, a processor, and computer instructions stored on the memory and executable on the processor, wherein, The processor executes the computer instructions to implement the steps of the method of any one of claims 1-7.

10. A computer-readable storage medium storing computer instructions, wherein, The computer instructions are executed by the processor to implement the steps of the method of any one of claims 1-7.

Images