Data processing method and data processing system

Abstract

The present disclosure provides a data processing method and a data processing system. The method includes: receiving a first data source identifier and a structured query statement, the structured query statement including at least one query field, a data screening condition, and at least one inter-table association relationship, the inter-table association relationship being used to indicate the association between data tables of a source data source; performing a predetermined parsing operation on the structured query statement to obtain a parsing result, the parsing result including the table name of the data table where the target data corresponding to each query field is located, and the data type of the target data corresponding to each query field; based on the parsing result, the data screening condition, and the inter-table association relationship, collecting target data from at least one data table of the source data source, and writing the target data to the destination data source. The present disclosure can realize the collection of data from multiple data tables of a source data source at one time, effectively improving data acquisition efficiency.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to a data processing method, a data processing system, an electronic device, and a computer-readable medium. Background Art

[0002] A smart city is an advanced form of urban informatization that utilizes various information technologies to connect and integrate urban systems and services, thereby improving resource utilization efficiency and optimizing urban management and services. In the smart city industry, customized management systems can hinder information collection and sharing, making it difficult for management systems to interact with each other. Currently, to facilitate data exchange between management systems, the Exchangis platform (a data exchange platform) can be used to connect management systems and enable data transmission between systems. The Exchangis platform is a lightweight, highly scalable data exchange platform that supports data sharing services between heterogeneous structured and unstructured data sources.

[0003] However, in the current process of providing data sharing services for various data sources, when collecting data from the source data source through the Exchange platform, only one data table can be selected from the source data source to collect data at a time, resulting in low data collection efficiency, which in turn seriously affects the efficiency of data sharing between data sources. Summary of the Invention

[0004] The present disclosure provides a data processing method, a data processing system, an electronic device, and a computer-readable medium.

[0005] A first aspect of the present disclosure provides a data processing method, the data processing method comprising:

[0006] Receive a first data source identifier and a structured query statement, wherein the structured query statement includes at least one query field, a data screening condition, and at least one inter-table association relationship, wherein the inter-table association relationship is used to indicate an association between data tables of a source data source;

[0007] Performing a predetermined parsing operation on the structured query statement to obtain a parsing result; the parsing result includes the table name of the data table where the target data corresponding to each query field is located, and the data type of the target data corresponding to each query field;

[0008] Based on the analysis results, the data screening conditions and the relationship between the tables, the target data is collected from at least one data table of a source data source, and the target data is written to a destination data source; the source data source is the data source corresponding to the first data source identifier.

[0009] In some embodiments, the predetermined parsing operation includes a first parsing operation and a second parsing operation; the first parsing operation is used to obtain the field name of each query field in the structured query statement; the second parsing operation is used to obtain the table name of the data table where the target data corresponding to each query field in the structured query statement is located, and the data type of the target data corresponding to each query field, wherein the table name includes the alias of the data table.

[0010] In some embodiments, before receiving the first data source identifier and structured query statement input by the user, the method further includes:

[0011] Establishing communication connections with multiple data sources, wherein the multiple data sources include at least the source data source and the destination data source;

[0012] Obtaining a mapping relationship between the query field and a target field, where the target field is a field contained in a data table belonging to the target data source;

[0013] Wherein, writing the target data into the destination data source includes:

[0014] Based on the mapping relationship between the query field and the destination field, the target data corresponding to the query field is written into the data table corresponding to the destination field.

[0015] In some embodiments, after performing a predetermined parsing operation on the structured query statement and obtaining a parsing result, the method further includes:

[0016] Converting the structured query statement, the parsing result, and the mapping relationship between the query field and the target field into task configuration data of the target task, and storing the task configuration data in a local database;

[0017] The step of collecting the target data from at least one data table of the source data source based on the analysis result, the data screening condition, and the relationship between the tables, and writing the target data to the destination data source includes:

[0018] In response to the task execution instruction, the target task is executed to collect the target data from at least one data table of the source data source based on the analysis result, the data screening condition and the association relationship between the tables, and based on the mapping relationship between the query field and the destination field, write the target data corresponding to the query field into the data table corresponding to the destination data source.

[0019] In some embodiments, before collecting the target data from at least one data table of the source data source based on the parsing result, the data screening condition, and the inter-table association relationship, the method further includes:

[0020] In response to the mapping modification instruction, the destination field of the target query field mapping is modified to a new destination field, wherein the target query field is the query field pointed to by the modification instruction.

[0021] In some embodiments, the step of performing a predetermined parsing operation on the structured query statement to obtain a parsing result further includes:

[0022] Performing a verification operation on the structured query statement to determine whether the structured query statement meets preset requirements;

[0023] When the structured query statement meets the preset requirements, a predetermined parsing operation is performed on the structured query statement to obtain a parsing result.

[0024] In some embodiments, performing a verification operation on the structured query statement to determine whether the structured query statement meets preset requirements includes:

[0025] Verifying whether the structured query statement is a preset statement and whether the structured query statement complies with preset grammatical rules;

[0026] In the case that the structured query statement is a preset statement and the structured query statement complies with preset grammatical rules, the structured query statement complies with preset requirements.

[0027] In some embodiments, the target data includes at least one of government affairs data of an administrative area, management data of a place, traffic data of a road, meteorological management data, and medical service data.

[0028] In some embodiments, the data processing method further includes:

[0029] Configure a batch deletion script for the preset business data table; the batch deletion script includes the number of latest data entries to be retained, the business table name, and the amount of data to be deleted in each batch;

[0030] The batch deletion script is activated to obtain a set of business data identifiers to be deleted and to delete business data based on the set of business data identifiers to be deleted.

[0031] A second aspect of the present disclosure provides a data processing system, the data processing system comprising:

[0032] A receiving module, configured to receive a first data source identifier and a structured query statement, wherein the structured query statement includes at least one query field, a data screening condition, and at least one inter-table association relationship, wherein the inter-table association relationship is used to indicate an association between data tables of a source data source;

[0033] A parsing module, configured to perform a predetermined parsing operation on the structured query statement to obtain a parsing result; the parsing result includes the table name of the data table where the target data corresponding to each query field is located, and the data type of the target data corresponding to each query field;

[0034] a collection module, configured to collect the target data from at least one data table of a source data source based on the parsing result and the association relationship between the tables, where the source data source is a data source corresponding to the first data source identifier;

[0035] The writing module is used to write the target data into the destination data source.

[0036] A third aspect of the present disclosure provides an electronic device, including:

[0037] one or more processors;

[0038] a memory having one or more programs stored thereon, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement any one of the above-mentioned data processing methods;

[0039] One or more I / O interfaces are connected between the processor and the memory and configured to implement information interaction between the processor and the memory.

[0040] A fourth aspect of the present disclosure provides a computer-readable medium having a computer program stored thereon, which implements any of the above-mentioned data processing methods when the program is executed by a processor. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 A flowchart of a data processing method provided in an embodiment of the present disclosure;

[0042] Figure 2 A flowchart of a data processing method provided in an embodiment of the present disclosure;

[0043] Figure 3 A flowchart of a data processing method provided in an embodiment of the present disclosure;

[0044] Figure 4 A schematic diagram of a target task creation page provided in an embodiment of the present disclosure;

[0045] Figure 5 A schematic diagram of a page showing the parsing result and the mapping relationship between the query field and the target field provided by an embodiment of the present disclosure;

[0046] Figure 6 A schematic diagram of a page displaying target tasks provided by an embodiment of the present disclosure;

[0047] Figure 7 A schematic diagram of the structure of a data processing system provided in an embodiment of the present disclosure;

[0048] Figure 8 A structural diagram of an electronic device is provided in accordance with an embodiment of the present disclosure. DETAILED DESCRIPTION

[0049] In order to enable those skilled in the art to better understand the technical solutions of the embodiments of the present disclosure, the technical solutions of the data processing method and data processing system provided by the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings of the embodiments of the present disclosure.

[0050] Example embodiments will be described more fully hereinafter with reference to the accompanying drawings, but the example embodiments may be embodied in different forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete and will fully convey the scope of this disclosure to those skilled in the art.

[0051] The terms used herein are used only to describe specific embodiments and are not intended to limit the present disclosure. As used herein, the singular forms "a," "an," and "the" are intended to include the plural forms, unless the context clearly indicates otherwise. It will also be understood that when the terms "comprise" and / or "include" are used in this specification, they specify the presence of the features, wholes, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, wholes, steps, operations, elements, components, and / or groups thereof.

[0052] It will be understood that although the terms first, second, etc. may be used herein to describe various elements / structures, these elements / structures should not be limited by these terms. These terms are only used to distinguish one element / structure from another element / structure.

[0053] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art. It will also be understood that terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant art and the present disclosure, and will not be interpreted as having an idealized or overly formal meaning unless expressly defined as such herein.

[0054] Figure 1 This is a flow chart of a data processing method provided by an embodiment of the present disclosure. The data processing method is applied to a data processing system. Figure 1 As shown, the data processing method includes the following steps S101 to S104.

[0055] Step S101: Receive a first data source identifier and a structured query statement.

[0056] The first data source identifier is used to uniquely identify a source data source, which refers to a data source database.

[0057] In this embodiment, the source data source can be a relational database management system (RDBMS). Data is stored in the data source in the form of a data table. Each row of the data table is called a tuple, or a record. Each column of the data table is called a field. Each field describes the meaning of the data corresponding to the field. For example, if the field is math score, the data corresponding to the field is the math score value. The primary key is the data used to uniquely identify a tuple. The primary key can be one or more fields in the data table. For example, in a data table that records personal information, the primary key can be the ID number field.

[0058] Structured Query Language (SQL) is a statement used to manage databases. The structured query statement includes at least one query field, a data filtering condition, and at least one inter-table relationship. The query field is used to identify the fields that need to be collected, and the data filtering condition is used to indicate the collection range when collecting data, such as the time range; the inter-table relationship is used to indicate the relationship between the data tables of the source data source. For example, Table 1 is a student score table, and Table 1 includes the fields of student ID, student name, student Chinese score, and student math score; Table 2 is a student information table, and Table 2 includes the fields of student ID, student age, and student class. Then the inter-table relationship between Table 1 and Table 2 is the student ID.

[0059] In one embodiment, the step of receiving the first data source identifier and the structured query statement includes: receiving the first data source identifier and the structured query statement input by a user. The first data source identifier and the structured query statement may be information input by the user from a client page of the data processing system.

[0060] Step S102: performing a predetermined parsing operation on the structured query statement to obtain a parsing result.

[0061] A data processing request is a user-triggered request to process a structured query statement. The parsing results include the table name of the target data table for each query field, as well as the data type of the target data for each query field. The target data for each query field refers to the data that meets the data filtering criteria within the entire data table corresponding to that query field.

[0062] In one embodiment, in order to reduce the possibility of data acquisition failure, the step of performing a predetermined parsing operation on the structured query statement includes: performing a verification operation on the structured query statement to determine whether the structured query statement meets preset requirements.

[0063] The verification operation is used to verify whether the structured query statement meets preset requirements, and the preset requirements can be set according to actual application scenarios.

[0064] In some embodiments, the data processing system performs a verification operation on the structured query statement, including: verifying whether the structured query statement is a preset statement, and verifying whether the structured query statement complies with preset grammatical rules.

[0065] The preset statement is a query (SELECT) statement, and the preset grammar rules are grammar rules of the SELECT statement.

[0066] In some embodiments, when the structured query statement is not a preset statement, or the structured query statement does not comply with preset grammatical rules, the structured query statement does not comply with preset requirements.

[0067] It should be noted that if the structured query statement does not meet the preset requirements, subsequent operations based on the structured query statement will not obtain the expected results. If the structured query statement does not meet the preset requirements, the structured query statement can be ignored, or a reminder message indicating that the SQL statement verification failed can be generated and displayed, so that the SQL statement can be retrieved again, reducing the possibility of data retrieval failures in subsequent processes.

[0068] In other embodiments, when the structured query statement is a preset statement and the structured query statement complies with preset grammatical rules, the structured query statement complies with preset requirements.

[0069] In one embodiment, the predetermined parsing operation includes a first parsing operation and a second parsing operation.

[0070] The first parsing operation can be based on the Druid SQL Parser, which is used to obtain the field name of each query field in the structured query statement. The second parsing operation is a custom parsing operation, which is used to obtain the table name of the data table where the target data corresponding to each query field in the structured query statement is located, as well as the data type of the target data corresponding to each query field, wherein the table name includes the actual table name and / or alias of the data table.

[0071] In one embodiment, the predetermined parsing operation is as follows:

[0072] Step A: Use the open-source Druid SQL parser to obtain the field name set of the query field, but without the table name, data type, and whether it is a primary key.

[0073] Step B: Customize SQLSelectASTVisitor to inherit SQLASTVisitorAdapter and rewrite the visit(SQLSelectItem x) method: loop through the obtained query fields, call the visit(SQLSelectItem x) of the custom Visitor for each query field, obtain the SQLExpr in the SQLSelectItem, and call the following step C.

[0074] Step C: The method visit(SQLExpr x) is responsible for parsing the query field alias, column name, table alias, table name, and whether it is a primary key:

[0075] Case 1: If the type of x is SQLAggregateExpr, obtain the first parameter of the SQLAggregateExpr object and then recursively call step C.

[0076] Case 2: If the type of x is SQLIdentifierExpr, obtain the field name and field alias, and call step D below.

[0077] Step D: Customize the method getSelectColumn(SQLExpr expr, Column column) to resolve the field table alias, table name, and whether it is a primary key:

[0078] Step D1, obtain the parent object SQLObject of expr.

[0079] In step D2, if the SQLObject belongs to the SQLSelectItem, the parent object of the SQLSelectItem is obtained.

[0080] In step D3, if the SQLObject belongs to SQLAggregateExpr, the parent object of the parent object of SQLAggregateExpr is obtained.

[0081] Step D4: If the SQLObject belongs to SQLSelectQueryBlock, obtain the SQLTableSource.

[0082] Step D5, calls step E below to return the table alias and table name of the field.

[0083] Step D6, loop the following step E, obtain the table alias and table name set, if the table to which the current query field belongs is not empty, and the table name of the belonging table is the same as the current table name or alias, then it is considered to be the table name or table alias of the current field.

[0084] In step D7, the data type of the current field and whether it is a primary key are obtained from the database based on the column name and the table name obtained in step D6.

[0085] Step E: Customize SQLTableNameVisitor to inherit SQLASTVisitorAdapter, override the visit(SQLExprTableSource x) method, and return the table alias and table name collection.

[0086] It should be noted that step A is the first parsing operation, and steps BE are the second parsing operation. SQLSelectASTVisitor, SQLASTVisitorAdapter, visit(SQLSelectItemx), SQLTableNameVisitor, visit(SQLExprTableSource x), etc. involved in the second parsing operation are all classes or methods defined in Java.

[0087] Step S103: based on the analysis results, data screening conditions and the relationship between tables, target data is collected from at least one data table of the source data source.

[0088] The source data source is a data source corresponding to the first data source identifier. The at least one data table includes a data table where target data corresponding to each query field in the parsing result is located.

[0089] Step S104: Write the target data into the destination data source.

[0090] The destination data source refers to the database where the data is to be stored, and is also the preset data source for data interaction with the source data source.

[0091] In some embodiments, the destination data source may be a relational database.

[0092] An embodiment of the present disclosure provides a data processing method. First, a first data source identifier and a structured query statement are received. The structured query statement includes at least one query field, a data screening condition, and at least one inter-table association relationship. The inter-table association relationship is used to indicate the association between data tables of a source data source. Then, a predetermined parsing operation is performed on the structured query statement to obtain a parsing result. The parsing result includes the table name of the data table where the target data corresponding to each query field is located, and the data type of the target data corresponding to each query field. Finally, based on the parsing result, the data screening condition, and the inter-table association relationship, the target data is collected from at least one data table of the source data source, and the target data is written to the destination data source. This method can realize the one-time collection of data from multiple data tables of the source data source, effectively improve the data acquisition efficiency, and thereby improve the efficiency of data sharing between data sources.

[0093] Figure 2 A flowchart of a data processing method provided by an embodiment of the present disclosure. In one embodiment, before receiving the first data source identifier and structured query statement input by the user, as shown in FIG. Figure 2 As shown, the data processing method further includes: step S201-step S202.

[0094] Step S201: Establish communication connections with multiple data sources.

[0095] The multiple data sources include at least a source data source and a destination data source. The source data source refers to a database where the data originates, and the destination data source refers to a database where the data is stored. The communication connection includes a wired communication connection and a wireless communication connection, such as a fifth-generation mobile communication connection.

[0096] It should be noted that, in actual application, any one of the above-mentioned multiple data sources may be used as a source data source in some cases, and may be used as a destination data source in other cases.

[0097] In one embodiment, the step of establishing communication connections with multiple data sources includes: for each data source, establishing a communication connection with the data source based on the IP address of the data source and a pre-acquired user name and password corresponding to the data source.

[0098] Step S202: Acquire the mapping relationship between the query field and the target field.

[0099] The destination fields are fields contained in the data table belonging to the destination data source.

[0100] In one embodiment, in order to facilitate the management of mapping relationships, after obtaining the mapping relationship between the query field and the target field, the data processing system also generates and stores a metadata model based on the mapping relationship between the query field and the target field, wherein a mapping relationship between the query field and the target field corresponds to a metadata model.

[0101] In this embodiment, generating and storing a metadata model for the mapping relationship between the query field and the target field can realize the reuse of the mapping relationship and improve the data processing speed.

[0102] In one embodiment, the step of writing the target data into the destination data source (step S104) includes: based on the mapping relationship between the query field and the destination field, writing the target data corresponding to the query field into the destination field in the data table corresponding to the destination data source.

[0103] In this embodiment, communication connections are established with multiple data sources in advance, and the mapping relationship between the query field and the destination field is obtained. Finally, based on the mapping relationship between the query field and the destination field, the target data corresponding to the query field is written into the destination field in the data table corresponding to the destination data source. This can achieve accurate mapping of the target data from the source data source to the destination data source, help improve data acquisition efficiency, and further improve the efficiency of data sharing between data sources.

[0104] In one embodiment, before collecting target data from at least one data table of the source data source based on the parsing results, data screening conditions and inter-table associations, the above method further includes: in response to a mapping modification instruction, modifying the destination field mapped by the target query field to a new destination field.

[0105] The target query field is the query field pointed to by the modification instruction.

[0106] In this embodiment, in order to improve the intelligence of data acquisition, the mapping relationship between the query field and the target field can be modified before collecting data, thereby improving the efficiency of data acquisition and further improving the efficiency of data sharing between data sources.

[0107] In one embodiment, after performing a predetermined parsing operation on the structured query statement and obtaining the parsing result (step S102), it also includes: converting the structured query statement, the parsing result, and the mapping relationship between the query field and the target field into task configuration data of the target task, and storing the task configuration data in a local database.

[0108] The target task is a data acquisition task. The task configuration data may be in JSON format. The local database is a database belonging to the data processing system.

[0109] In this embodiment, the steps of collecting target data from at least one data table of the source data source based on the parsing results, data screening conditions, and inter-table associations, and writing the target data to the destination data source (steps S103 and S104) include:

[0110] In response to the task execution instruction, the target task is executed to collect target data from at least one data table of the source data source based on the parsing results and the relationship between tables, and based on the mapping relationship between the query field and the destination field, the target data corresponding to the query field is written into the destination field in the data table corresponding to the destination data source.

[0111] The task execution instruction is an instruction input by a user through a client page of the data processing system for instructing the execution of a target task.

[0112] In this embodiment, after obtaining the parsing results, data acquisition does not need to be performed directly. Instead, the structured query statement, parsing results, and the mapping relationship between the query field and the target field are converted into task configuration data of the target task, and the target task is executed when data acquisition is required, thereby automatically collecting data from the source data source and writing it to the destination data source, effectively improving the data acquisition efficiency, and thereby improving the efficiency of data sharing between data sources.

[0113] In one embodiment, the present disclosure further provides a data processing method. The data processing method includes:

[0114] Step 1: Configure a batch deletion script for the preset business data table.

[0115] Among them, the preset business data table is a business data table stored in the database, such as a road traffic data table (business table of captured faces or vehicles, etc.), a medical service data table (patient's medical treatment status table, etc.) and other business data tables. The preset business data table includes data identification and data creation time; the batch deletion script includes retaining the latest number of data items, the business table name and the amount of data deleted in each batch.

[0116] In some embodiments, a batch deletion script such as delBatch(num, tableName, limitNo) is used, where delBatch() is a batch deletion script function, num is the number of latest data entries to be retained, tableName is the name of the business table, and limitNo is the amount of data to be deleted in each batch.

[0117] Step 2: Activate a batch deletion script to obtain a set of business data identifiers to be deleted and delete business data based on the set of business data identifiers to be deleted.

[0118] In some embodiments, after activating the batch deletion script, the row_number() function can be used to sort the business data in the preset business data table in descending order according to the data creation time, and obtain the row number of the returned record. Then, based on the row number being greater than num and only filtering out data with a number of data items limited by limitNo, a set of business data identifiers to be deleted is obtained, and the business data is deleted based on the set of business data identifiers to be deleted.

[0119] In some embodiments, after deleting the business data, it is determined based on the returned result whether the current operation has deleted data. If so, the above step 1 is called again after sleeping for 1 second to delete the business data to reduce the pressure on the server; if no data is deleted, the current scheduled task is terminated.

[0120] This data processing method provides a method for batch deleting large amounts of data from common business tables, reducing the repetitive development workload required to batch delete expired data from different business tables. Furthermore, a scheduled task sleeps for one second between script calls, alleviating server pressure. Furthermore, the batch deletion script uses the data identifier of the data to be deleted and then deletes it based on that identifier, increasing batch deletion speed. This allows for rapid data deletion without impacting database server performance, reducing the chance that other services will be unable to access data due to large transactions.

[0121] Figure 3 This is a flow chart of a data processing method provided by an embodiment of the present disclosure. The data processing method is applied to a data processing system, such as the Exchange platform. The data processing system includes a client and a backend, wherein the client can be displayed through a web page for users to access the client through the web page and instruct the data acquisition process through the client, and the backend is used to execute various instructions received by the client to perform corresponding operations and complete the data acquisition process. Figure 3 As shown, the data processing method includes the following steps S301-S310.

[0122] Step S301: In response to a user access request, a data acquisition control page is displayed through the client.

[0123] The user access request is a request sent by a user through a user terminal to a data processing system for requesting to obtain services provided by the data processing system.

[0124] The data acquisition control page is the main page for providing data acquisition services to users. The data acquisition control page can be set according to specific application scenarios. For example, in some implementation scenarios, the data acquisition control page can include a command trigger link for the corresponding function, such as creating a new application.

[0125] Step S302: In response to the new application instruction, create a virtual management application corresponding to the user.

[0126] The new application instruction is an instruction for creating a new application triggered by the user terminal through the data acquisition control page. The virtual management application refers to a virtual application belonging to the user for managing data, for example, a virtual application that can be used for archiving data or interacting with data.

[0127] Step S303: Receive a data source creation instruction and establish communication connections with multiple data sources.

[0128] The data source creation instruction includes the IP address of the data source, the pre-acquired user name and password corresponding to the data source. The multiple data sources include at least a source data source and a destination data source.

[0129] In some embodiments, the user can enter the IP address of the data source, the pre-acquired user name and password corresponding to the data source on the page displayed by the client through the user terminal, so that the data processing system can establish a communication connection with the corresponding data source.

[0130] In one implementation scenario, taking medical services in a smart city as an example, the source data source can be multiple hospital systems, and the destination data source can be the medical data aggregation system of the administrative region that includes the multiple hospitals.

[0131] In another implementation scenario, taking schools in a smart city as an example, the source data source can be the management systems of multiple different schools, and the destination data source can be the data aggregation system of the education management agency that manages these multiple schools.

[0132] Step S304: Acquire the mapping relationship between the query field and the target field, and create a metadata model based on the mapping relationship between the query field and the target field.

[0133] The query field identifies the fields in the source data source that need to be collected. The target field is the field contained in the data table belonging to the target data source. A mapping relationship between a query field and a target field corresponds to a metadata model.

[0134] Step S305: In response to the new task instruction, a target task creation page is displayed.

[0135] The new task instruction is an instruction from the user to create a task for managing data, such as creating a data interaction task, a data aggregation task, etc. The target task creation page is a page for obtaining user creation requirements.

[0136] Figure 4 A schematic diagram of a target task creation page provided in an embodiment of the present disclosure. Figure 4As shown, the target task creation page includes a data source identification acquisition part 401 and an SQL statement acquisition part 402.

[0137] The data source identifier acquisition part 401 and the SQL statement acquisition part 402 are used to receive a first data source identifier and a structured query statement, respectively.

[0138] In some embodiments, the target task creation page may further include one or more of a page switching control portion, a source data source type input portion, a database name input portion, a portion for selecting whether to enable SQL statements, a portion for triggering SQL statement validation, and a portion for selecting whether to enable timestamps. The page switching control portion is configured to control page switching after the user enters the first data source identifier and the structured query statement.

[0139] Step S306: In response to the SQL statement verification command, a verification operation is performed on the structured query statement to obtain a verification result.

[0140] The SQL statement verification command is triggered by the user through the target task creation page to perform verification operations on the structured query statement. It should be noted that this SQL statement can implement some complex statistics as a temporary table, and then perform summary calculations on this temporary table.

[0141] In some embodiments, the data processing system performs a verification operation on the structured query statement to obtain a verification result, including: verifying whether the structured query statement is a preset statement, and verifying whether the structured query statement complies with preset grammatical rules.

[0142] The preset statement is a query (SELECT) statement, and the preset grammar rules are grammar rules of the SELECT statement.

[0143] In some embodiments, when the structured query statement is not a preset statement, or the structured query statement does not comply with preset grammatical rules, the verification result is failure.

[0144] In other embodiments, when the structured query statement is a preset statement and the structured query statement complies with preset grammatical rules, the verification result is passed.

[0145] In some embodiments, the SQL statement verification command and the following SQL statement parsing command can be triggered through the same interface of the target task creation page.

[0146] Step S307: In response to the SQL statement parsing command, a predetermined parsing operation is performed on the structured query statement to obtain a parsing result, and the parsing result and the mapping relationship between the query field and the target field are displayed.

[0147] The SQL statement parsing command is a command triggered by the user through the target task creation page to perform a validation operation on a structured query statement. The parsing results include the table name of the target data table corresponding to each query field, as well as the data type of the target data corresponding to each query field. The target data corresponding to a query field refers to the data that meets the data screening criteria within the entire data table corresponding to that query field.

[0148] Figure 5 A schematic diagram of a page showing the parsing results and the mapping relationship between the query field and the target field provided by the embodiment of the present disclosure. Figure 5 As shown, the field number, source field, type corresponding to the source field, target field and type corresponding to the target field, and operation trigger part in this page are displayed in an associated relationship.

[0149] The field sequence number is used to identify the order in which query fields are displayed; the source field consists of the field name of the query field and the table name of the data table where the target data corresponding to the query field is located, such as t.source_type and t.dataSourceld. In addition, since the table name of a data table includes the actual table name and / or alias of the data table, in order to improve the accuracy of identifying the data table during subsequent data collection, the table name in the source field is preferably displayed using the data table alias. For example, if the source fields corresponding to a collection field are t.dataSourceld and t.COUNT(j.job_name), the preferred source field is t.COUNT(j.job_name). The type corresponding to the source field is the data type of the target data corresponding to the query field, such as BIGINT, VARCHAR, INT, TEXT, DATETIME, etc. The target field is composed of the field name of the target field and the table name of the data table where the target field is located. Since the table name of the data table includes the actual table name and / or alias of the data table, in order to improve the accuracy of subsequent data written to the data table of the target data source, the table name in the source field is preferably displayed as the alias of the data table. The types corresponding to the target field are, for example: BIGINT, VARCHAR, INT, TEXT, DATETIME, etc. It should be noted that the alias of the data table can effectively reduce the name length, making the SQL statement concise and clear when the user enters the SQL statement. Figure 5 When the page shown is displayed, it can also improve the user experience.

[0150] Step S308 : In response to the target task save instruction, the structured query statement, the parsing result, and the mapping relationship between the query field and the target field are converted into task configuration data of the target task, and the task configuration data is stored in the local database.

[0151] The target task saving instruction is an instruction for converting the structured query statement, the parsing result, and the mapping relationship between the query field and the target field into task configuration data of the target task and saving the converted data.

[0152] Step S309: In response to the task execution instruction, execute the target task to collect target data from at least one data table of the source data source based on the parsing results and the relationship between tables, and write the target data corresponding to the query field into the destination field in the data table corresponding to the destination data source based on the mapping relationship between the query field and the destination field.

[0153] The task execution instruction is an instruction input by a user through a client page of the data processing system for instructing the execution of a target task.

[0154] Figure 6 A schematic diagram of a page showing target tasks provided by an embodiment of the present disclosure. Figure 6 As shown, the page includes the target task identifier, the first data source identifier, the data source type, the task description, the task creation time, and the task operation.

[0155] The target task identifier uniquely identifies the stored target task. The first data source identifier identifies the source data source corresponding to the target task. The data source type is the type of the source data source, such as MySQL. The task description briefly describes the task content. Task operations include at least one of the following: edit, delete, execute, and display task details.

[0156] In one embodiment, in response to a task execution instruction, during the execution of a target task, a database read plug-in pre-stored in the backend collects target data from at least one data table of the source data source based on the parsing results and the relationship between tables, and sends the target data to the pipeline. A data write plug-in pre-stored in the backend writes the target data corresponding to the query field in the pipeline to the destination field in the data table corresponding to the destination data source based on the mapping relationship between the query field and the destination field. The pipeline can be a buffer in the backend where users cache data.

[0157] Step S310: Check the target data synchronized in the destination data source.

[0158] The data processing method provided by the embodiment of the present disclosure can collect data from multiple data tables of the source data source at one time, effectively improving the efficiency of data acquisition, and further improving the efficiency of data sharing between data sources.

[0159] The data processing method provided by the present disclosure can be applied to the construction scenarios of smart cities. Therefore, in different implementation scenarios, the target data may include at least one of the government data of the administrative area, the management data of the venue, the traffic data of the road, the meteorological management data, and the medical service data.

[0160] In this implementation scenario, the data processing method can support the provision of data sharing (synchronization) services between structured and unstructured heterogeneous data sources, effectively improve the efficiency of data acquisition, and thereby improve the efficiency of data sharing (synchronization) between data sources, contributing to the efficient development of the smart city industry.

[0161] The steps of the various methods above are divided only for the purpose of clear description. During implementation, they can be combined into one step or some steps can be split and decomposed into multiple steps. As long as they include the same logical relationship, they are all within the scope of protection of this patent. Adding insignificant modifications or introducing insignificant designs to the algorithm or process without changing the core design of the algorithm and process are all within the scope of protection of this patent.

[0162] Figure 7 This is a structural diagram of a data processing system provided by an embodiment of the present disclosure. Figure 7 As shown, the data processing system includes: a receiving module 71 , an analyzing module 72 , a collecting module 73 and a writing module 74 .

[0163] The receiving module 71 is configured to receive a first data source identifier and a structured query statement.

[0164] The structured query statement includes at least one query field, a data screening condition, and at least one inter-table association relationship, where the inter-table association relationship is used to indicate the association between data tables of the source data source.

[0165] The parsing module 72 is used to perform a predetermined parsing operation on the structured query statement to obtain a parsing result.

[0166] The parsing result includes the table name of the data table where the target data corresponding to each query field is located, and the data type of the target data corresponding to each query field.

[0167] The collection module 73 is configured to collect the target data from at least one data table of the source data source based on the analysis results and the association relationship between tables.

[0168] The source data source is the data source corresponding to the first data source identifier.

[0169] The writing module 74 is used to write the target data into the destination data source.

[0170] It should be understood that the present disclosure is not limited to the specific configurations and processes described in the above embodiments and illustrated in the figures. For the sake of convenience and brevity, detailed descriptions of known methods are omitted here. The specific working processes of the systems, modules, and units described above can be referred to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0171] An embodiment of the present disclosure provides a data processing system, wherein a receiving module is used to receive a first data source identifier and a structured query statement, the structured query statement including at least one query field, a data screening condition and at least one inter-table association relationship, the inter-table association relationship being used to indicate the association between data tables of a source data source; a parsing module is used to perform a predetermined parsing operation on the structured query statement to obtain a parsing result, the parsing result including the table name of the data table where the target data corresponding to each query field is located, and the data type of the target data corresponding to each query field; a collection module is used to collect target data from at least one data table of the source data source based on the parsing result, the data screening condition and the inter-table association relationship, and a writing module is used to write the target data into a destination data source, thereby enabling data to be collected from multiple data tables of the source data source at one time, effectively improving data acquisition efficiency, and thereby improving the efficiency of data sharing between data sources.

[0172] Figure 8 A structural diagram of an electronic device is provided in accordance with an embodiment of the present disclosure.

[0173] Reference Figure 8 , an electronic device provided by an embodiment of the present disclosure includes:

[0174] One or more processors 801;

[0175] a memory 802 storing one or more programs, which, when executed by one or more processors, enable the one or more processors to implement any one of the aforementioned data processing methods;

[0176] One or more I / O interfaces 803 are connected between the processor and the memory and are configured to implement information exchange between the processor and the memory.

[0177] Among them, the processor 801 is a device with data processing capabilities, including but not limited to a central processing unit (CPU); the memory 802 is a device with data storage capabilities, including but not limited to random access memory (RAM, more specifically such as Synchronous Dynamic Random-Access Memory, abbreviated as SDRAM, Double Data Rate SDRAM, abbreviated as DDR, etc.), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), and flash memory (FLASH); the I / O interface (read-write interface) 803 is connected between the processor 801 and the memory 802, and can realize information exchange between the processor 801 and the memory 802, including but not limited to a data bus (Bus), etc.

[0178] In some embodiments, the processor 801 , the memory 802 , and the I / O interface 803 are connected to each other via a bus, and further connected to other components of the computing device.

[0179] This embodiment also provides a computer-readable medium on which a computer program is stored. When the program is executed by a processor, the data processing method provided in this embodiment is implemented. To avoid repeated description, the specific steps of the data processing method are not repeated here.

[0180] It will be appreciated by those skilled in the art that all or some of the steps, systems, and functional modules / units in the methods invented above may be implemented as software, firmware, hardware, and appropriate combinations thereof. In hardware implementations, the division between the functional modules / units mentioned in the above description does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed by several physical components in cooperation. Some or all physical components may be implemented as software executed by a processor, such as a central processing unit, a digital signal processor, or a microprocessor, or implemented as hardware, or implemented as an integrated circuit, such as an application-specific integrated circuit. Such software may be distributed on a computer-readable medium, which may include a computer storage medium (or non-transitory medium) and a communication medium (or temporary medium). As is well known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable, and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, Compact Disc Read-Only Memory (CD-ROM), Digital Video Disc (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and can be accessed by a computer. In addition, it is well known to those skilled in the art that communication media generally contain computer-readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media.

[0181] It should be noted that, in this document, the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or apparatus comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or apparatus comprising the element.

[0182] Those skilled in the art will understand that although some embodiments described herein include certain features included in other embodiments but not other features, the combination of features from different embodiments is meant to be within the scope of the present embodiment and to form different embodiments.

[0183] It is understood that the above embodiments are merely exemplary embodiments for illustrating the principles of the present disclosure, and the present disclosure is not limited thereto. Those skilled in the art may make various modifications and improvements without departing from the spirit and substance of the present disclosure, and such modifications and improvements are also considered to be within the scope of protection of the present disclosure.

Claims

1. A data processing method, characterized in that: The data processing method includes: Establishing communication connections with multiple data sources, wherein the multiple data sources include at least a source data source and a destination data source; Obtaining a mapping relationship between a query field and a target field, where the target field is a field contained in a data table belonging to the target data source; Receive a first data source identifier and a structured query statement, wherein the structured query statement includes at least one query field, a data screening condition, and at least one inter-table association relationship, wherein the inter-table association relationship is used to indicate an association between data tables of a source data source; Performing a predetermined parsing operation on the structured query statement to obtain a parsing result; the parsing result includes the table name of the data table where the target data corresponding to each query field is located, and the data type of the target data corresponding to each query field; Based on the analysis result, the data screening condition, and the relationship between the tables, the target data is collected from at least one data table of a source data source, and the target data is written to a destination data source; the source data source is the data source corresponding to the first data source identifier; The step of writing the target data into the destination data source includes: based on a mapping relationship between the query field and the destination field, writing the target data corresponding to the query field into a data table corresponding to the destination field in the destination data source.

2. The method according to claim 1, characterized in that The predetermined parsing operation includes a first parsing operation and a second parsing operation; the first parsing operation is used to obtain the field name of each query field in the structured query statement; the second parsing operation is used to obtain the table name of the data table where the target data corresponding to each query field in the structured query statement is located, and the data type of the target data corresponding to each query field, wherein the table name includes the alias of the data table.

3. The method according to claim 1, characterized in that After performing a predetermined parsing operation on the structured query statement and obtaining a parsing result, the method further includes: Converting the structured query statement, the parsing result, and the mapping relationship between the query field and the target field into task configuration data of the target task, and storing the task configuration data in a local database; The step of collecting the target data from at least one data table of the source data source based on the analysis result, the data screening condition, and the relationship between the tables, and writing the target data to the destination data source includes: In response to the task execution instruction, the target task is executed to collect the target data from at least one data table of the source data source based on the analysis result, the data screening condition and the association relationship between the tables, and based on the mapping relationship between the query field and the destination field, write the target data corresponding to the query field into the data table corresponding to the destination data source.

4. The method according to claim 1 or 3, characterized in that Before collecting the target data from at least one data table of a source data source based on the analysis result, the data screening condition, and the relationship between tables, the method further includes: In response to the mapping modification instruction, the destination field of the target query field mapping is modified to a new destination field, wherein the target query field is the query field pointed to by the modification instruction.

5. The method according to claim 1, wherein The step of performing a predetermined parsing operation on the structured query statement to obtain a parsing result includes: Performing a verification operation on the structured query statement to determine whether the structured query statement meets preset requirements; When the structured query statement meets the preset requirements, a predetermined parsing operation is performed on the structured query statement to obtain a parsing result.

6. The method according to claim 5, characterized in that The step of performing a verification operation on the structured query statement to determine whether the structured query statement meets preset requirements includes: Verifying whether the structured query statement is a preset statement and whether the structured query statement complies with preset grammatical rules; In the case that the structured query statement is a preset statement and the structured query statement complies with preset grammatical rules, the structured query statement complies with preset requirements.

7. The method according to claim 1, characterized in that The target data includes at least one of government affairs data of an administrative area, management data of a place, traffic data of a road, meteorological management data, and medical service data.

8. The method according to claim 1, characterized in that The data processing method further includes: Configure a batch deletion script for the preset business data table; the batch deletion script includes the number of latest data entries to be retained, the business table name, and the amount of data to be deleted in each batch; The batch deletion script is activated to obtain a set of business data identifiers to be deleted and to delete business data based on the set of business data identifiers to be deleted.

9. A data processing system, characterized in that: The data processing system comprises: A communication module, configured to establish communication connections with multiple data sources, wherein the multiple data sources include at least a source data source and a destination data source; An acquisition module, configured to acquire a mapping relationship between a query field and a target field, wherein the target field is a field contained in a data table belonging to the target data source; A receiving module, configured to receive a first data source identifier and a structured query statement, wherein the structured query statement includes at least one query field, a data screening condition, and at least one inter-table association relationship, wherein the inter-table association relationship is used to indicate an association between data tables of a source data source; A parsing module, configured to perform a predetermined parsing operation on the structured query statement to obtain a parsing result; the parsing result includes the table name of the data table where the target data corresponding to each query field is located, and the data type of the target data corresponding to each query field; a collection module, configured to collect the target data from at least one data table of a source data source based on the parsing result and the association relationship between the tables, where the source data source is a data source corresponding to the first data source identifier; The writing module is used to write the target data into the destination data source.

10. An electronic device, characterized in that: include: one or more processors; a memory having one or more programs stored thereon, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the data processing method according to any one of claims 1 to 8; One or more I / O interfaces are connected between the processor and the memory and configured to implement information interaction between the processor and the memory.

11. A computer-readable medium having a computer program stored thereon, wherein when the program is executed by a processor, the data processing method according to any one of claims 1 to 8 is implemented.

Images