Data processing method, device, server, storage medium and product

Abstract

The embodiments of the present invention disclose a data processing method, device, server, storage medium and computer program product, which relate to the field of big data. The method includes: obtaining data field definition and configuration information of a data file to be processed; selecting a corresponding script template according to the target database type, generating a data processing script based on the script template and the data field definition, executing the data processing script to load the data records in the data file to be processed into a temporary table of the target database; synchronizing the data records in the temporary table to the target table according to the data processing requirements in the configuration information. The embodiments of the present invention implement an extensible and configurable loading method of data records by configuring script templates and data processing requirements, solve the problems of complex development work and low development efficiency caused by separate development for different processing requirements, and achieve the effect of shortening the development time.

Description

Technical Field

[0001] Embodiments of the present invention relate to the field of big data, and in particular, to a data processing method, device, server, storage medium, and computer program product. Background Art

[0002] For distributed systems, each subsystem generates data, and each subsystem may use data generated by other subsystems. For example, in the financial field, there may be a need to synchronize basic data such as institutional employees and customer information, as well as business data such as loans, payment settlements, etc.

[0003] Currently, technicians design data synchronization processes for each data synchronization requirement and develop program codes based on the data synchronization process. Since data synchronization requirements are diverse in reality, the above data synchronization implementation methods will increase the complexity of development work, reduce development efficiency, and increase development time. Summary of the invention

[0004] The embodiments of the present invention provide a data processing method, device, server, storage medium and computer program product, which can simplify the complexity of development work, improve development efficiency and shorten the time of development work.

[0005] In a first aspect, an embodiment of the present invention provides a data processing method, including:

[0006] Obtain data field definition and configuration information of the data file to be processed;

[0007] Select the corresponding script template according to the target database type, generate a data processing script based on the script template and data field definition, and execute the data processing script to load the data records in the data file to be processed into the temporary table of the target database;

[0008] According to the data processing requirements in the configuration information, synchronize the data records in the temporary table to the target table.

[0009] In a second aspect, an embodiment of the present invention further provides a data processing device, the device comprising:

[0010] An information acquisition module is used to obtain data field definitions and configuration information of the data file to be processed;

[0011] A script generation module is used to select a corresponding script template according to the target database type, generate a data processing script based on the script template and data field definition, and execute the data processing script to load the data records in the data file to be processed into a temporary table of the target database;

[0012] The data synchronization module is used to synchronize the data records in the temporary table to the target table according to the data processing requirements in the configuration information.

[0013] In a third aspect, an embodiment of the present invention further provides a server, the server comprising:

[0014] one or more processors;

[0015] a memory for storing one or more programs,

[0016] When one or more programs are executed by one or more processors, the one or more processors implement the data processing method provided by any embodiment of the present invention.

[0017] In a fourth aspect, an embodiment of the present invention provides a storage medium comprising computer executable instructions, which, when executed by a computer processor, are used to execute a data processing method as described in any embodiment of the present invention.

[0018] In a fifth aspect, an embodiment of the present invention further provides a computer program product, including a computer program, which, when executed by a processor, implements the data processing method as described in any embodiment of the present invention.

[0019] The embodiment of the present invention provides a data processing method, device, server, storage medium and computer program product. After obtaining the data field definition corresponding to the data file to be processed, a script template is selected according to the target database type, and a data processing script is generated based on the data field definition and the script template. The data records in the data file to be processed are loaded into the temporary table of the target database through the data processing script; according to the data processing requirements in the configuration information, the data records in the temporary table are synchronized to the target table. The embodiment of the present invention realizes an extensible and configurable loading method of data records by configuring the script template and the data processing requirements, solves the problems of complex development work and low development efficiency caused by separate development for different processing requirements, and achieves the effect of shortening the development time. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 A flowchart of a data processing method provided by an embodiment of the present invention;

[0021] Figure 2a A flowchart of another data processing method provided by an embodiment of the present invention;

[0022] Figure 2b It is a flow chart of a method for implementing data synchronization between a target table and a temporary table;

[0023] Figure 3 A flowchart of another data processing method provided by an embodiment of the present invention;

[0024] Figure 4 An overall flow chart of a data processing method provided by an embodiment of the present invention;

[0025] Figure 5 A structural block diagram of a data processing device provided by an embodiment of the present invention;

[0026] Figure 6 A schematic diagram of the structure of a server provided by the present invention. DETAILED DESCRIPTION

[0027] The present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the present invention, rather than to limit the present invention. It should also be noted that, for ease of description, only parts related to the present invention, rather than all structures, are shown in the accompanying drawings.

[0028] Financial systems often use a distributed system architecture. Each system generates its own data, and the data of one system may be used by other related systems. For example, the various component systems in the financial field may need to synchronize basic data such as institutional employees and customer information, as well as synchronize business data such as loans, payment settlements, etc.

[0029] There may be a need to synchronize new data between the systems under development and the systems that have been put into operation. Different data synchronization needs may have different requirements. For example, the target database for data synchronization may be Oracle or MySQL. Some data synchronization requires some pre-processing before synchronization, and some data synchronization requires some simple processing of fields. Some systems want to synchronize data in the form of replica tables (the data structure of the target table is the same or similar to that of the source system table), and some only want to synchronize the changed data of the day, without synchronizing all data.

[0030] For example, after receiving the data file sent by the source system, the developer parses the data fields and field types, and based on the analyzed data fields and data types, writes a database import program according to the database type used by the target system, and executes the database import program to load the data file into the temporary table of the target system database. Then, write a data synchronization program and execute the data synchronization program to synchronize the data file in the temporary table to the target table. In summary, since data synchronization requirements are diverse, if more development is done for each data synchronization requirement, the complexity of development work is greatly increased and development efficiency is reduced.

[0031] Figure 1This is a flow chart of a data processing method provided by an embodiment of the present invention. This embodiment is applicable to the case of synchronizing data between different subsystems of a distributed system. The method can be executed by a data processing device, which can be implemented by software and / or hardware and configured in a server. Figure 1 As shown, the method includes:

[0032] Step 110: Obtain data field definition and configuration information of the data file to be processed.

[0033] The data files to be processed are the data to be synchronized that the source system sends to the target system.

[0034] In the data synchronization scenario, the source system is the system that provides data files, and the target system is the system that requests data synchronization. The target system needs to use the data generated by the source system. For example, if system B requests data synchronization from system A, then system B is the target system and system A is the source system. If system B requests data synchronization from system A, and system C requests data synchronization from system A, then in the data synchronization operation between systems AB, system B is the target system and system A is the source system. In the data synchronization operation between systems BC, system B is the source system and system C is the target system. The data source system and the target system can belong to the same distributed system or to different distributed systems. When the target system needs to use the data generated by the source system, a data synchronization request is sent to the source system. In response to the data synchronization request of the target system, the source system sends the data files to be processed to the target system.

[0035] The data field definition is used to define the fields included in the data file to be processed. Specifically, the data field definition may include field length definition, field separation method definition, and field type definition, etc. The data file can be parsed in the manner specified in the data field definition to obtain the fields included in the file and the data type of each field, etc.

[0036] Optionally, in a scenario where data synchronization is required, the source system will send a data file to the target system, and the data file sent by the source system is the data file to be processed. In addition, the source system will also send a data field definition file to the target system. The data field definition file records the data field definition of the data file to be processed.

[0037] Optionally, the technical personnel of the source system and the target system may communicate offline about the data field definition and save the data field definition to the configuration information.

[0038] Configuration information includes data synchronization requirements for the data files to be processed. The technician can specify the data synchronization requirements by configuration. The data synchronization requirements can also be modified by changing the configuration information. Specifically, the synchronization requirements of most data files can be met by configuring several attribute information in the program code. For special processing requirements of certain fields, the special processing requirements can be met by configuring optional attributes. Among them, special processing requirements include requirements for pre-processing or post-processing of certain fields.

[0039] For systems that need to synchronize data, before data synchronization, the data sender (i.e., source system) can send the data field definition file to the data receiver (i.e., target system). Alternatively, the sender and receiver's technical staff can determine the data field definition through offline communication, and the receiver's technical staff can update the database configuration information based on the data field definition.

[0040] It should be noted that script templates and data definition file processing interfaces are configured in the target system. If there are new requirements, the new requirements can be supported through the interfaces in the implementation framework of the data processing method of the present invention without changing the overall framework logic. For example, if there is a new database file loading requirement, the new processing logic can be expanded through the reserved interface to support the file loading requirements of the new type of database. Specifically, by calling the interface to add a new script template and adding the data definition file processing interface corresponding to the new type of database, etc., there is no need to rewrite the entire set of methods, and the properties can be configured according to the requirements of the original database, so that the new database data file synchronization requirements can be met.

[0041] Exemplarily, after receiving the data file to be processed, it is determined whether there is a data definition file corresponding to the file to be processed; if so, the data definition file is parsed through the data definition file processing interface to obtain the data field definition of the data file to be processed, and the data field definition is stored in association with the file identifier of the data file to be processed. The data file processing interface includes a processing method for the data file, one type of data definition file corresponds to one processing method, and the processing methods for different types of data definition files can be expanded.

[0042] Exemplarily, after receiving the data file to be processed, the target system obtains the data field definition information of the data file and also obtains the configuration information input by the user based on the synchronization requirement of the data file.

[0043] Specifically, after receiving the data file to be processed, if it is determined that there is a data definition file corresponding to the data file to be processed, the data definition file is parsed through the data definition file processing interface to obtain the data field definition of the data file to be processed, and the data field definition is saved to the target database (the target database is the database of the target system) with the file identifier of the data file to be processed as an index. The target database is queried based on the file identifier of the data file to be processed to obtain the data field definition of the data file to be processed.

[0044] Optionally, after receiving the data file to be processed, if it is determined that there is no data definition file corresponding to the data file to be processed, the file identifier of the data file to be processed is determined, and the database configuration information (the database configuration information is the database configuration information of the target system) is queried based on the file identifier to obtain the data field definition of the data file to be processed.

[0045] Step 120: Select a corresponding script template according to the target database type, generate a data processing script based on the script template and data field definition, and execute the data processing script to load the data records in the data file to be processed into a temporary table of the target database.

[0046] It should be noted that script templates for different types of databases are pre-configured. If there is a new database requirement, the processing logic required for the database can be added through the interface provided by the technical framework of the present invention. The processing logic may include data definition files and script templates.

[0047] Since the generation logic of load scripts for different types of databases may be different, when script templates corresponding to multiple types of databases are configured, you need to select a script template based on the target database type. For example, Oracle generates a sqlldr script, and MySQL generates a load data infile script. Here you can write other database file load script generation logic to expand support for different databases.

[0048] A temporary table is a data table in a target database used to temporarily store data records of a data file to be processed.

[0049] Exemplarily, a script template corresponding to the target database type is selected from a script template library, and the target field in the data field definition is filled into the script template to generate a data processing script. The data processing script is executed to load the data records in the data file to be processed into a temporary table of the target database. Specifically, the type information of the field to be filled in the script template is determined; the data field definition is queried according to the type information to determine the target field, and the target field is filled into the script template to obtain the data processing script. The data processing script is executed to load the data file to be processed into the landing table.

[0050] Step 130: Synchronize the data records in the temporary table to the target table according to the data processing requirements in the configuration information.

[0051] The target table is a data table in the target database used to store synchronized data records. For example, the target system requests synchronized data from the source system, and the synchronized data records are saved to the target table according to different data processing requirements. It should be noted that the user configures the configuration information of the data processing requirements to the database according to the processing requirements, and the data processing requirements can be determined by querying the configuration information. Data processing requirements include full data synchronization, incremental data synchronization, or deleted data synchronization.

[0052] Exemplarily, the data processing requirements in the configuration information are obtained; and the data records in the temporary table are synchronized to the target table in the target database in a manner specified by the data processing requirements.

[0053] The technical solution described in this embodiment obtains the data field definition and configuration information of the data file to be processed; selects the corresponding script template according to the target database type, generates a data processing script based on the script template and the data field definition, and executes the data processing script to load the data records in the data file to be processed into a temporary table of the target database; synchronizes the data records in the temporary table to the target table according to the data processing requirements in the configuration information, thereby solving the problems of complex development work and low development efficiency caused by separate development for different processing requirements, and achieving the effect of shortening the development time.

[0054] Figure 2a The flowchart of another data processing method provided by the embodiment of the present invention further illustrates the implementation method of synchronizing the data records in the temporary table to the target table in the target database in accordance with the method specified by the data processing requirements based on the above technical solution. Figure 2a As shown, the method includes:

[0055] Step 210: Obtain data field definition and configuration information of the data file to be processed.

[0056] The configuration information is information that is flexibly configured by the user through a configuration interface in the implementation framework of the data processing method of the present invention.

[0057] When data processing is required, the user first configures the configuration information to the database by calling the configuration interface, so that when performing data synchronization, the user can perform corresponding operations according to the configuration information to meet the data processing requirements.

[0058] Step 220: Select a corresponding script template according to the target database type, generate a data processing script based on the script template and the data field definition, and execute the data processing script to load the data records in the data file to be processed into a temporary table of the target database.

[0059] Step 230: Obtain data processing requirements in the configuration information.

[0060] Exemplarily, the configuration information of the database is read, and the data processing requirement is obtained from the configuration information.

[0061] Step 240: When the data processing requirement is full data synchronization, clear the data records in the target table, and insert all the data records in the temporary table into the cleared target table.

[0062] Exemplarily, if the data processing requirement is full data synchronization, all existing data records in the target table are cleared. Then, all data records in the temporary table are read, and the read data records are inserted into the cleared target table. For example, if it is full data synchronization, the user configures data synchronization loading in ALL form. At this time, the target table is cleared, and then all data records in the temporary table are inserted into the target table. Among them, clearing the target table is to delete the data records in the target table as a whole, and the ID is also deleted, which is equivalent to rebuilding the table, maintaining the structure of the original data table, but all states are equivalent to a new table. Executing the clear table instruction can quickly delete all data in the data table, improving data synchronization efficiency.

[0063] Step 250: When the data processing requirement is incremental data synchronization, the primary keys of the temporary table and the target table are matched respectively, and the target table is updated based on each data record according to the matching result.

[0064] Among them, the primary key generally refers to the primary keyword, which can be one or more fields in the data table. The value of the primary key is used to uniquely identify a data record in the data table. By comparing the values ​​of the primary keys, it can be determined whether the data records corresponding to two primary keys are the same.

[0065] If the data processing requirement is incremental data synchronization, it means that only the changed data needs to be synchronized to the target table. Based on the incremental data synchronization requirement, you can first determine which data records in the temporary table have changed relative to the target table, and then update the target table based on the changed data records to achieve the incremental data synchronization requirement.

[0066] Exemplarily, when the temporary table is not an empty table, arbitrarily select a primary key in the temporary table as the current primary key, and compare the value of the current primary key with the values ​​of the primary keys of the target table; when the values ​​are equal, update the data records corresponding to the corresponding primary key values ​​in the target table based on the data records corresponding to the current primary key, and delete the data records corresponding to the current primary key from the temporary table; when the values ​​are not equal, insert the data records corresponding to the current primary key into the target table.

[0067] Specifically, when the data processing requirement is incremental data synchronization, if the user configures loading in the form of MERGE, the primary keys of the data records in the temporary table and the target table are matched. If the primary key values ​​are equal, the match is considered successful. If no primary key in the target table is matched with the value of any primary key in the temporary table, the match is considered unsuccessful. If the match is successful, the data records corresponding to the corresponding primary keys in the target table are updated based on the data records corresponding to the successfully matched primary keys in the temporary table. If the match fails, the data records corresponding to the current primary key in the temporary table are inserted into the target table.

[0068] When the data processing requirement is incremental data synchronization, if the user configures loading in the form of INSERT, the data records in the temporary table will be inserted into the target table. This method can meet the system's requirement of only processing the changed data.

[0069] Optionally, if the data file sent by the source system includes both changed data and unchanged data, it is necessary to determine the changed data by matching the primary key value and insert only the changed data records into the target table.

[0070] Step 260: When the data processing requirement is to delete data synchronization, the primary keys of the temporary table and the target table are matched respectively, and the data records corresponding to the successfully matched primary keys in the target table are deleted.

[0071] Exemplarily, if the data processing requirement is to delete data synchronization, the primary keys of the temporary table and the target table are matched, and the data corresponding to the matched primary keys in the target table are synchronously deleted. For example, if it is to delete data synchronization, the user configures data synchronization loading in the form of DEL. At this time, the primary key values ​​of the temporary table and the target table are matched, the target primary key in the target table with the same value as the primary key of the temporary table is determined, and the data records corresponding to the target primary key in the target table are deleted.

[0072] Figure 2b This is a flow chart of a method for implementing data synchronization between a target table and a temporary table. Figure 2b As shown, the method specifically includes:

[0073] Step 201: Read configuration information to obtain the specification of the file loading type in the configuration information.

[0074] Step 202: Determine whether the loading type is a full data synchronization mode, an incremental MERGE mode, an incremental INSERT mode, or a delete synchronization mode.

[0075] Step 203: If the loading type is full data synchronization, clear the target table.

[0076] Step 204: Insert the data records of the temporary table into the target table.

[0077] Specifically, the method of Insert into target table from temporary table can be adopted to insert data records of temporary table into target table.

[0078] Step 205: If the loading type is incremental MERGE, merge the temporary table into the target table.

[0079] Specifically, if the loading type is incremental MERGE, the corresponding data records in the target table are updated according to the data records corresponding to the primary keys that match in the temporary table, and the data records corresponding to the primary keys that fail to match in the temporary table are inserted into the target table.

[0080] Step 206: If the loading type is incremental INSERT, insert the data records in the temporary table into the target table.

[0081] Specifically, the Insert into target table from temporary table method is adopted to insert the data records of the temporary table into the target table.

[0082] Step 207: If the loading type is the delete data synchronization mode, the data records in the target table are deleted according to the primary key field.

[0083] The technical solution described in this embodiment synchronizes the data records in the temporary table to the target table in a manner specified by the user configuring the configuration information that meets the data processing requirements, thereby realizing a configurable data processing method to meet personalized data processing requirements. There is no need to develop a separate program for each data processing requirement, which greatly simplifies the complexity of code development and improves code development efficiency.

[0084] Figure 3 The flowchart of another data processing method provided by the embodiment of the present invention further adds an implementation method for data processing scenarios in which the user configures special processing requirements on the basis of the above technical solution. Figure 3 As shown, the method includes:

[0085] Step 301: Obtain data field definition and configuration information of a data file to be processed.

[0086] Step 302: Select a corresponding script template according to the target database type, generate a data processing script based on the script template and data field definition, and execute the data processing script to load the data records in the data file to be processed into a temporary table of the target database.

[0087] Step 303: determine whether the user has configured a preprocessing operation, if so, execute step 304, otherwise execute step 305.

[0088] Exemplarily, configuration information of a database is read, a value of an attribute representing a preprocessing operation in the configuration information is determined, and whether the user has configured the preprocessing operation is determined based on the value.

[0089] Step 304 , obtaining a preprocessing method corresponding to the preprocessing operation in the configuration information, processing the data records in the temporary table according to the preprocessing method, and then executing step 305 .

[0090] The preprocessing method may be a SQL statement, a shell script, or a Java processing class, etc. The preprocessing method may be preset by the user according to actual needs, and the present invention does not specifically limit the implementation form of the preprocessing method.

[0091] Step 305: Synchronize the data records in the temporary table to the target table according to the data processing requirements in the configuration information.

[0092] Step 306 , determine whether the user has configured post-processing operations, if so, execute step 307 , otherwise, execute step 308 .

[0093] Exemplarily, the configuration information of the database is read, the value of the attribute representing the post-processing operation in the configuration information is determined, and whether the user has configured the post-processing operation is determined based on the value.

[0094] Step 307 , obtain the post-processing method corresponding to the post-processing operation in the configuration information, process the data records in the target table according to the post-processing method, and then execute step 308 .

[0095] Among them, the post-processing method can be a SQL statement, or a shell script, or a java processing class, etc. The post-processing method can be preset by the user according to actual needs, and the present invention does not specifically limit the implementation form of the post-processing method.

[0096] Step 308: End the data processing flow.

[0097] In a specific embodiment, the overall process of the data processing method of the present invention is provided. Figure 4 An overall flow chart of a data processing method provided by an embodiment of the present invention, such as Figure 4 As shown, the method includes:

[0098] Step 401: Detect an upstream data file.

[0099] The upstream data file is the data file to be processed that is sent from the source system to the target system.

[0100] For example, the target system receives a data file sent by the source system. The fields in the data file are generally separated by certain characters or the length of each field is specified, and the type of each field is agreed upon. In short, the technical staff of the source system and the target system will agree on how to parse the data file, that is, which fields are included and what the data type of each field is. This agreement is usually defined by the source system transmitting a data definition file or communicating offline.

[0101] Step 402 , determine whether a data definition file exists, if so, execute step 403 , otherwise execute step 405 .

[0102] The data definition file includes information such as upstream data fields and type definitions, and is a file sent by the source system to the target system.

[0103] Step 403: parse the data definition file.

[0104] Step 404: associate the data definition file with the file identifier of the data file and store it in a database.

[0105] Exemplarily, it is determined whether a data definition file exists. If so, a data definition file processing interface is called to parse the definition file, and then the parsed information is saved in a database.

[0106] It should be noted that the data definition file processing interface is used to implement the parsing of different types of data definition files. One type of data definition file has a corresponding parsing implementation method, and different types of data definition file processing methods can be extended to meet the parsing requirements of new data definition files.

[0107] Step 405: Read the data field definition saved in the database.

[0108] If the data definition file does not exist, you need to configure the agreed data field definition information to the database. When you need to synchronize the data file, you can directly read the database configuration information to obtain the data field definition associated with the data file.

[0109] Step 406: Generate a file loading script according to the data field definition.

[0110] Exemplarily, according to the acquired data field definition, a data file loading script of the corresponding database is generated. For example, an oracle type database generates a sqlldr script, and a mysql type database generates a load data infile script. In addition, other database file loading script generation logic can be written to expand support for unused databases, effectively avoiding the problem of needing to rewrite the entire file loading method when encountering processing requirements of other unused databases, thereby improving code reuse rate.

[0111] Step 407: Execute the file loading script to load the upstream data file into the temporary table of the database.

[0112] Step 408 , determine whether a preprocessing operation needs to be performed, if so, execute step 409 , otherwise, execute step 410 .

[0113] Step 409: read the configuration information of the database, and execute the processing method corresponding to the preprocessing operation in the configuration information.

[0114] For some data file synchronization operations, it is necessary to perform some custom logic processing after the upstream data file is loaded into the temporary table and before it is synchronized from the temporary table to the target table to be synchronized. For example, some unnecessary data may be cleared, or some additional data processing may be performed. In the embodiment of the present invention, the custom logic processing may specifically be a SQL statement, or a shell script, or a JAVA processing class, or the like.

[0115] Step 410: Read the configuration information of the database, and obtain the provisions on the loading type of the file in the configuration information.

[0116] It should be noted that for the data files provided by the source system, different operations may be performed according to different processing requirements. For example, some require full data synchronization, some require incremental data synchronization, and some require deleting archive data (that is, deleting the data records corresponding to the data files in the target table).

[0117] Step 411, determine whether the loading type is a full data synchronization mode, an incremental MERGE mode, an incremental INSERT mode, or a delete synchronization mode.

[0118] Step 412: If the loading type is full data synchronization, clear the target table, and then execute step 413.

[0119] Step 413: Execute the Insert into target table from temporary table operation, and then execute step 417.

[0120] Step 414: If the loading type is incremental MERGE, perform the operation of merging the temporary table into the target table, and then execute step 417.

[0121] Step 415: If the loading type is incremental INSERT, perform Insert into target table from temporary table operation, and then execute step 417.

[0122] Step 416: If the loading type is the delete data synchronization form, delete the data records in the target table according to the primary key field, and then execute step 417.

[0123] It should be noted that the processing logic for different synchronization requirements is different. If it is full data synchronization, you can configure the ALL form for synchronization loading. At this time, the program will clear the target table and then insert all data records in the temporary table into the target table. If it is incremental data, you can configure the incremental MERGE form or incremental INSERT form for synchronization loading. For the incremental MERGE form, match the primary keys of the data in the target table and the temporary table. If there is a matching primary key in the target table, update the matching data records in the temporary table to the target table. If a primary key in the temporary table does not have a matching primary key in the target table, insert the data records of the corresponding primary key in the temporary table into the target table. For the incremental INSERT form, it is mainly aimed at the type of synchronization requirements that only need to process data that changes every day, and insert the newly added data into the target table for subsequent logical processing. If it is to delete file data, you can configure the DEL form for synchronization loading. The program will synchronously delete the data records corresponding to the matching primary key in the target table.

[0124] Step 417, determine whether post-processing operation needs to be performed, if so, execute step 418, otherwise, execute step 419.

[0125] For some data file synchronization operations, it is necessary to perform some custom logic processing after synchronously loading the data records from the temporary table to the target table. In the embodiment of the present invention, the custom logic processing can specifically be a SQL statement, or a shell script, or a JAVA processing class, etc.

[0126] Step 418: read the configuration information of the database, and execute the processing method corresponding to the post-processing operation in the configuration information.

[0127] Step 419, end the data processing flow.

[0128] The embodiment of the present invention provides a configurable data processing method. For different data file loading requirements, most data file loading requirements can be achieved only through configuration without additional development. If there are special requirements, they can also be supported through extension, which reduces the work of developers, reduces development time, and is conducive to improving the overall project promotion efficiency.

[0129] Figure 5 This is a structural block diagram of a data processing device provided in an embodiment of the present invention. The device can execute the data processing method of any embodiment of the present invention and realize data synchronization between subsystems of a distributed system by executing the above method. The device can be implemented by software and / or hardware and configured in a server. Figure 5 As shown, the device comprises:

[0130] The information acquisition module 510 is used to acquire data field definition and configuration information of the data file to be processed;

[0131] The script generation module 520 is used to select a corresponding script template according to the target database type, generate a data processing script based on the script template and the data field definition, and execute the data processing script to load the data records in the data file to be processed into a temporary table of the target database;

[0132] The data synchronization module 530 is used to synchronize the data records in the temporary table to the target table according to the data processing requirements in the configuration information.

[0133] An embodiment of the present invention provides a data processing device, which realizes an extensible and configurable loading method of data records by configuring script templates and data processing requirements, solves the problems of complex development work and low development efficiency caused by separate development for different processing requirements, and achieves the effect of shortening the development work time.

[0134] Optionally, the device further comprises:

[0135] A definition file determination module is used to determine whether there is a data definition file corresponding to the data file to be processed before obtaining the data field definition of the data file to be processed;

[0136] The field definition storage module is used to parse the data definition file through the data definition file processing interface in the scenario where there is a data definition file corresponding to the to-be-processed file, obtain the data field definition of the to-be-processed data file, and associate and store the data field definition with the file identifier of the to-be-processed data file.

[0137] Optionally, the information acquisition module 510 is specifically used for:

[0138] The target database is queried based on the file identifier of the data file to be processed to obtain the data field definition of the data file to be processed.

[0139] Optionally, the information acquisition module 510 is further configured to:

[0140] Determine the file identifier of the data file to be processed, query the configuration information according to the file identifier, and obtain the data field definition of the data file to be processed.

[0141] Optionally, the script generation module 520 is specifically used for:

[0142] Fill the target fields in the data field definition into the script template to generate a data processing script.

[0143] Optionally, the script generation module 520 is further used to:

[0144] Determine the type information of the fields to be filled in the script template;

[0145] The data field definition is queried according to the type information to determine the target field, and the target field is filled into the script template to obtain the data processing script.

[0146] Optionally, the data synchronization module 530 includes:

[0147] The requirement acquisition submodule is used to obtain the data processing requirements in the configuration information;

[0148] The data synchronization submodule is used to synchronize the data records in the temporary table to the target table in the target database in accordance with the method specified by the data processing requirements.

[0149] Furthermore, the data processing requirements include at least one of the following: full data synchronization, incremental data synchronization, and deleted data synchronization.

[0150] Optionally, the data synchronization submodule is specifically used for:

[0151] When the data processing requirement is full data synchronization, clear the data records in the target table and insert all data records in the temporary table into the cleared target table;

[0152] When the data processing requirement is incremental data synchronization, the primary keys of the temporary table and the target table are matched respectively, and the target table is updated based on the data records in the temporary table according to the matching results;

[0153] When the data processing requirement is to delete data synchronization, the primary keys of the temporary table and the target table are matched respectively, and the data records corresponding to the successfully matched primary keys in the target table are deleted.

[0154] Optionally, when the data processing requirement is incremental data synchronization, the data synchronization submodule is further specifically used for:

[0155] When the temporary table is not an empty table, arbitrarily select a primary key in the temporary table as the current primary key, and compare the value of the current primary key with the values ​​of each primary key of the target table;

[0156] When the values ​​are equal, the data records corresponding to the corresponding primary key values ​​in the target table are updated based on the data records corresponding to the current primary key, and the data records corresponding to the current primary key are deleted from the temporary table;

[0157] When the values ​​are not equal, the data record corresponding to the current primary key is inserted into the target table.

[0158] Optionally, the device further comprises:

[0159] A preprocessing module is used to determine whether the user has configured a preprocessing operation after executing a data processing script to load data records in a data file to be processed into a temporary table of a target database;

[0160] If so, obtain the preprocessing method corresponding to the preprocessing operation in the configuration information, and process the data records in the temporary table according to the preprocessing method;

[0161] If not, the data records in the temporary table are synchronized to the target table according to the data processing requirements in the configuration information.

[0162] Optionally, the device further comprises:

[0163] The post-processing module is used to synchronize the data records in the temporary table to the target table. When it is detected that the user has configured a post-processing operation, the post-processing module obtains the post-processing method corresponding to the post-processing operation in the configuration information, and processes the data records in the target table according to the post-processing method.

[0164] The data processing device provided in the embodiment of the present invention can execute the data processing method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0165] Figure 6 A schematic diagram of the structure of a server provided by the present invention, such as Figure 6 As shown, the server includes a processor 60 and a memory 61; the number of processors 60 in the server can be one or more. Figure 6 A processor 60 is taken as an example; the processor 60 and the memory 61 in the server can be connected by a bus or other means. Figure 6 The example of connecting through bus is taken in the following.

[0166] The memory 61 is a computer-readable storage medium that can be used to store software programs, computer executable programs and modules, such as program instructions / modules corresponding to the data processing method in the embodiment of the present invention (for example, the information acquisition module 510, the script generation module 520 and the data synchronization module 530). The processor 60 executes various functional applications and data processing of the server by running the software programs, instructions and modules stored in the memory 61, that is, implements the above-mentioned data processing method.

[0167] The memory 61 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system and at least one application required for a function; the data storage area may store data created according to the use of the terminal, etc. In addition, the memory 61 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some instances, the memory 61 may further include a memory remotely arranged relative to the processor 60, and these remote memories may be connected to the server via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0168] An embodiment of the present invention further provides a storage medium containing computer executable instructions, and when the computer executable instructions are executed by a computer processor, they are used to perform a data processing method, the method comprising:

[0169] Obtain data field definition and configuration information of the data file to be processed;

[0170] Select the corresponding script template according to the target database type, generate a data processing script based on the script template and data field definition, and execute the data processing script to load the data records in the data file to be processed into the temporary table of the target database;

[0171] According to the data processing requirements in the configuration information, synchronize the data records in the temporary table to the target table.

[0172] Of course, the computer executable instructions of a storage medium containing computer executable instructions provided by an embodiment of the present invention are not limited to the method operations described above, and can also execute related operations in the data processing method provided by any embodiment of the present invention.

[0173] An embodiment of the present invention further provides a computer program product, including a computer program, wherein when the computer program is executed by a processor, the computer program implements the data processing method as described in any embodiment of the present invention.

[0174] Through the above description of the implementation methods, the technicians in the relevant field can clearly understand that the present invention can be implemented by means of software and necessary general hardware, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as a computer floppy disk, read-only memory (ROM), random access memory (RAM), flash memory (FLASH), hard disk or optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, server, or network device, etc.) to execute the methods described in each embodiment of the present invention.

[0175] It is worth noting that in the embodiment of the above-mentioned data processing device, the various units and modules included are only divided according to functional logic, but are not limited to the above-mentioned division, as long as the corresponding functions can be achieved; in addition, the specific names of the functional units are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of the present invention.

[0176] Note that the above are only preferred embodiments of the present invention and the technical principles used. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and that various obvious changes, readjustments and substitutions can be made by those skilled in the art without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in more detail through the above embodiments, the present invention is not limited to the above embodiments, and may include more other equivalent embodiments without departing from the concept of the present invention, and the scope of the present invention is determined by the scope of the appended claims.

Claims

1. A data processing method, characterized in that: include: Obtain data field definition and configuration information of the data file to be processed; Select a corresponding script template according to the target database type, generate a data processing script based on the script template and the data field definition, and execute the data processing script to load the data records in the data file to be processed into a temporary table of the target database; According to the data processing requirements in the configuration information, synchronize the data records in the temporary table to the target table; Wherein, obtaining the data field definition of the data file to be processed includes: Determine the file identifier of the data file to be processed, query the configuration information according to the file identifier, and obtain the data field definition of the data file to be processed; Wherein, generating a data processing script based on the script template and the data field definition includes: Fill the target field in the data field definition into the script template to generate a data processing script; Wherein, filling the target field in the data field definition into the script template to generate a data processing script includes: Determine type information of the field to be filled in the script template; Query the data field definition according to the type information to determine the target field, and fill the target field into the script template to obtain a data processing script; Wherein, according to the data processing requirements in the configuration information, synchronizing the data records in the temporary table to the target table includes: Obtaining data processing requirements in the configuration information; According to the data processing requirements, the data records in the temporary table are synchronized to the target table in the target database; Wherein, according to the data processing requirement, synchronizing the data records in the temporary table to the target table in the target database includes: When the data processing requirement is full data synchronization, clear the data records in the target table, and insert all the data records in the temporary table into the cleared target table; When the data processing requirement is incremental data synchronization, matching the primary keys of the temporary table and the target table respectively, and updating the target table based on the data records of the temporary table according to the matching result; When the data processing requirement is to delete data synchronization, the primary keys of the temporary table and the target table are matched respectively, and the data records corresponding to the successfully matched primary keys in the target table are deleted.

2. The method according to claim 1, characterized in that: Before obtaining the data field definition of the data file to be processed, it also includes: Determine whether there is a data definition file corresponding to the file to be processed; If so, the data definition file is parsed through a data definition file processing interface to obtain a data field definition of the data file to be processed, and the data field definition is associated and stored with a file identifier of the data file to be processed.

3. The method according to claim 2, characterized in that Obtaining the data field definition of the data file to be processed also includes: The target database is queried based on the file identifier of the data file to be processed to obtain the data field definition of the data file to be processed.

4. The method according to claim 1, characterized in that The data processing requirement includes at least one of the following: full data synchronization, incremental data synchronization, and deleted data synchronization.

5. The method according to claim 1, characterized in that Matching the primary keys of the temporary table and the target table respectively, and updating the target table based on the data records of the temporary table according to the matching results, including: When the temporary table is not an empty table, arbitrarily select a primary key in the temporary table as the current primary key, and compare the value of the current primary key with the values ​​of each primary key of the target table; When the values ​​are equal, the data record corresponding to the corresponding primary key value in the target table is updated based on the data record corresponding to the current primary key, and the data record corresponding to the current primary key is deleted from the temporary table; When the values ​​are not equal, the data record corresponding to the current primary key is inserted into the target table.

6. The method according to claim 1, characterized in that After executing the data processing script to load the data records in the to-be-processed data file into a temporary table of the target database, the method further includes: Determine whether the user has configured preprocessing operations; If yes, then obtaining a preprocessing method corresponding to the preprocessing operation in the configuration information, and processing the data records in the temporary table according to the preprocessing method; If not, the data records in the temporary table are synchronized to the target table according to the data processing requirements in the configuration information.

7. The method according to claim 1, characterized in that After synchronizing the data records in the temporary table to the target table, the method further includes: When it is detected that the user has configured a post-processing operation, a post-processing method corresponding to the post-processing operation in the configuration information is obtained, and the data records in the target table are processed according to the post-processing method.

8. A data processing device, characterized in that: include: An information acquisition module is used to obtain data field definitions and configuration information of the data file to be processed; A script generation module, used to select a corresponding script template according to the target database type, generate a data processing script based on the script template and the data field definition, and execute the data processing script to load the data records in the data file to be processed into a temporary table of the target database; A data synchronization module, used to synchronize the data records in the temporary table to the target table according to the data processing requirements in the configuration information; Wherein, obtaining the data field definition of the data file to be processed includes: Determine the file identifier of the data file to be processed, query the configuration information according to the file identifier, and obtain the data field definition of the data file to be processed; Wherein, generating a data processing script based on the script template and the data field definition includes: Fill the target field in the data field definition into the script template to generate a data processing script; Wherein, filling the target field in the data field definition into the script template to generate a data processing script includes: Determine type information of the field to be filled in the script template; Query the data field definition according to the type information to determine the target field, and fill the target field into the script template to obtain a data processing script; Wherein, according to the data processing requirements in the configuration information, synchronizing the data records in the temporary table to the target table includes: Obtaining data processing requirements in the configuration information; According to the data processing requirements, the data records in the temporary table are synchronized to the target table in the target database; Wherein, according to the data processing requirement, synchronizing the data records in the temporary table to the target table in the target database includes: When the data processing requirement is full data synchronization, clear the data records in the target table, and insert all the data records in the temporary table into the cleared target table; When the data processing requirement is incremental data synchronization, matching the primary keys of the temporary table and the target table respectively, and updating the target table based on the data records of the temporary table according to the matching result; When the data processing requirement is to delete data synchronization, the primary keys of the temporary table and the target table are matched respectively, and the data records corresponding to the successfully matched primary keys in the target table are deleted.

9. A server, characterized in that: The server comprises: one or more processors; a memory for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the data processing method according to any one of claims 1 to 7.

10. A storage medium containing computer executable instructions, characterized in that: When the computer executable instructions are executed by a computer processor, they are used to perform the data processing method according to any one of claims 1 to 7.

11. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the computer program implements the data processing method according to any one of claims 1 to 7.

Images