Data processing method and device, electronic equipment and storage medium

Abstract

The application provides a data processing method and device, electronic equipment and storage medium. It relates to big data processing technology and can be applied to the field of financial technology or other related fields. The method is applied to a unit architecture and includes the following steps: in response to a control instruction for a database in the unit architecture, obtaining import or export configuration parameters of the database, wherein the import or export configuration parameters are used to represent configuration parameters when performing an import operation or an export operation on the database; determining an operation processing mode corresponding to the control instruction, wherein the operation processing mode includes import processing and export processing; and performing an import operation of importing a data file into a file list of the database or an export operation of exporting a data file from the file list of the database according to the import or export configuration parameters and the operation processing mode. The method can realize the import or export operation of the data file on the database and is simple to implement.

Description

Technical Field

[0001] This application relates to big data processing technology, which can be used in the field of financial technology or other related fields, and in particular to a data processing method, apparatus, electronic device and storage medium. Background Technology

[0002] Currently, Oracle and MySQL databases involve a wide variety of table types, making the configuration cumbersome and implementation complex when performing data import and export operations.

[0003] For example, in the scenario of migrating existing data from an Oracle database to a MySQL database, the various types of data tables involved and the complex configuration of import and export fields for different types of data tables make the operations of exporting data from the Oracle database and importing data into the MySQL database complicated and cannot effectively guarantee the integrity of data import and export. Summary of the Invention

[0004] This application provides a data processing method, apparatus, electronic device, and storage medium for performing data file import or export operations on a database. The implementation is simple and can ensure the integrity of data import and export.

[0005] On the one hand, this application provides a data processing method applied to a unitized architecture, the method comprising:

[0006] In response to control commands to the database in the unitized architecture, the import or export configuration parameters of the database are obtained, wherein the import or export configuration parameters are used to characterize the configuration parameters when performing import or export operations on the database.

[0007] Determine the operation processing method corresponding to the above control instructions, wherein the above operation processing method includes: import processing and export processing;

[0008] Based on the above import or export configuration parameters and the above operation processing method, perform the import operation to import the data file into the file list of the above database, or perform the export operation to export the above data file from the file list of the above database.

[0009] Furthermore, if the above operation processing method is the above import processing, the above import operation, which imports the data file into the file list of the above database according to the above import or export configuration parameters and the above operation processing method, includes:

[0010] Based on the import configuration parameters, obtain the data file to be imported into the database and the filename of the data file.

[0011] If the filename of the aforementioned data file contains a predetermined suffix and a wildcard, then determine the list of files in the aforementioned database that match the aforementioned data file;

[0012] Obtain the file structure of the data file to be imported and the list structure of the file list to create a file importer;

[0013] Use the file importer described above to import the data files into the file list described above.

[0014] Furthermore, the process of obtaining the file structure of the data file to be imported and the list structure of the file list to create a file importer includes:

[0015] Retrieve the configuration file from the import configuration parameters mentioned above;

[0016] Based on the above configuration file, the file structure of the above data file and the list structure of the above file list are read, and the above file importer is constructed based on the above file structure and the above list structure.

[0017] Furthermore, the process of importing the data file into the file list using the aforementioned file importer includes:

[0018] The above-mentioned file importer is used to split the above data file to obtain multiple split sub-files;

[0019] If it is determined that the above data files need to be split and stored in the database, then traverse multiple of the above split sub-files and import each of the above split sub-files into the above file list.

[0020] If it is determined that the above data files do not need to be split and stored in the database, then the above data files can be directly imported into the above file list.

[0021] Furthermore, the above method also includes:

[0022] If it is determined that there is an error in the import processing of the above data file or the above split sub-file, then the import flag will be updated to indicate an error, and the above data file or the above split sub-file will be moved to the import failure directory.

[0023] If the import exception post-processing flag in the above import or export configuration parameters is determined to be "continue processing", then the import process will continue until it is completed, and then the import process will end.

[0024] If the import exception handling flag in the above import or export configuration parameters is determined to be "stop processing", then all data files imported after the current file in the above file list will be moved to the import failure directory and the import process will be terminated.

[0025] If it is confirmed that the above data file or the above split sub-file has been successfully imported into the above file list, then move the above data file or the above split sub-file to the directory where the import was successful until the import process is complete.

[0026] Furthermore, if the above operation processing method is the above export processing, the above export operation, which exports the above data files from the above database file list according to the above import or export configuration parameters and the above operation processing method, includes:

[0027] Based on the export configuration parameters, obtain the list of files containing the data files of the database to be exported;

[0028] Obtain the file structure of the data file to be exported and the list structure of the file list to create a file exporter;

[0029] The file exporter described above is used to determine the export path of the data file, so as to export the data file to the database based on the export path, and create a corresponding verification file for the data file after successful export.

[0030] Furthermore, the above method also includes:

[0031] In response to the import / export configuration command for the aforementioned database, import or export configuration parameters for the aforementioned database are generated. The aforementioned import or export configuration parameters include at least: Uniform Resource Locator, username, password, configuration file for the data file, directory where the data file to be imported is located, filename of the data file to be imported, temporary directory, import failure directory, import success directory, and import exception handling flag.

[0032] On the other hand, this application provides a data processing apparatus applied to a modular architecture, the apparatus comprising:

[0033] The acquisition module is used to acquire the database import or export configuration parameters in response to control commands to the database in the unitized architecture. The aforementioned import or export configuration parameters are used to characterize the configuration parameters when performing import or export operations on the database.

[0034] The determination module is used to determine the operation processing method corresponding to the above control instructions, wherein the above operation processing method includes: import processing and export processing;

[0035] The processing module is used to perform an import operation to import the data file into the file list of the database, or to perform an export operation to export the data file from the file list of the database, based on the import or export configuration parameters and the operation processing method described above.

[0036] Furthermore, if the above operation processing method is the above import processing, the above processing module includes:

[0037] The first acquisition submodule is used to acquire the data file to be imported into the above database and the file name of the above data file according to the import configuration parameters;

[0038] The determination submodule is used to determine the list of files in the database that match the data file if the filename of the data file contains a predetermined suffix and wildcard.

[0039] The second acquisition submodule is used to acquire the file structure of the data file to be imported and the list structure of the file list to create a file importer.

[0040] The import submodule is used to import the data files into the file list using the file importer described above.

[0041] Furthermore, the second acquisition submodule includes:

[0042] The acquisition unit is used to retrieve the configuration file from the imported configuration parameters mentioned above.

[0043] The construction unit is used to read the file structure of the data file and the list structure of the file list according to the configuration file, so as to construct the file importer based on the file structure and the list structure.

[0044] Furthermore, the aforementioned import submodule includes:

[0045] The splitting unit is used to split the data file using the file importer described above, so as to obtain multiple split sub-files;

[0046] The first import unit is used to, if it is determined that the above data file needs to be split and stored in the database, traverse multiple of the above split sub-files and import each of the above split sub-files into the above file list.

[0047] The second import unit is used to directly import the data files into the file list if it is determined that the data files do not need to be split and stored in the database.

[0048] Furthermore, if the above operation processing method is the above export processing, the above processing module includes:

[0049] Based on the export configuration parameters, obtain the list of files containing the data files of the database to be exported;

[0050] Obtain the file structure of the data file to be exported and the list structure of the file list to create a file exporter;

[0051] The file exporter described above is used to determine the export path of the data file, so as to export the data file to the database based on the export path, and create a corresponding verification file for the data file after successful export.

[0052] On one hand, this application provides an electronic device, including: a processor and a memory connected to the processor; the memory stores computer execution instructions; the processor executes the computer execution instructions stored in the memory to implement any of the methods described above.

[0053] On the other hand, this application provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement any of the methods described above.

[0054] On the other hand, this application provides a computer program product, including a computer program that, when executed by a processor, implements any of the methods described above.

[0055] The data processing method provided in this application is applied to a modular architecture. In response to a control command on a database in the modular architecture, it obtains import or export configuration parameters of the database, i.e., configuration parameters used to characterize the import or export operation performed on the database; determines the operation processing method corresponding to the control command, such as an import processing method and an export processing method; and performs an import operation to import a data file into the file list of the database, or performs an export operation to export the data file from the file list of the database, based on the import or export configuration parameters and the operation processing method.

[0056] This paper provides a data processing scheme for performing data file import or export operations on a database in a unitized architecture. The scheme is simple to implement and can ensure the integrity of data import and export. Attached Figure Description

[0057] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0058] Figure 1 A flowchart illustrating a data processing method provided in an embodiment of this application;

[0059] Figure 2 A flowchart illustrating an optional data processing method provided in an embodiment of this application;

[0060] Figure 3 This is a flowchart illustrating an optional data processing method provided in an embodiment of the present invention;

[0061] Figure 4 This is a flowchart illustrating an optional data processing method provided in an embodiment of the present invention;

[0062] Figure 5 This is a flowchart illustrating an optional data processing method provided in an embodiment of the present invention;

[0063] Figure 6 A structural block diagram of a data processing apparatus provided in an embodiment of this application;

[0064] Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application.

[0065] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0066] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0067] First, let me explain the terms used in this application:

[0068] Unitization: An architectural pattern where a unit is a self-contained set capable of performing all business operations. This set contains all the services required by the business and the data allocated to that unit. Unitized architecture uses units as the basic deployment unit, deploying multiple units across all data centers nationwide. The number of units in each data center is not fixed. Each unit deploys all the applications required by the system, and the data is a portion of the full dataset divided according to a certain dimension.

[0069] Heterogeneous databases are collections of multiple related database systems that enable data sharing and transparent access. These database systems already existed before being added to the heterogeneous database system and each has its own database management system. The various components of the heterogeneous database have their own autonomy. While achieving data sharing, each database system still has its own application characteristics, integrity controls, and security controls.

[0070] During the transformation of some applications to a modular architecture, taking asynchronous databases under a modular architecture as an example, existing data is migrated from an Oracle database to a MySQL database. The hundreds of millions of data points in a single database need to be split into 128 shards. Data import and export in large database systems involves a wide variety of data table types, such as transaction information, customer agreements, and system parameters. The import and export field configurations differ for different types of data tables, making it inconvenient to perform data import and export processing operations.

[0071] This disclosure proposes a data processing scheme for performing data file import or export operations on a database, aiming to solve the above-mentioned technical problems of the prior art, so as to realize the data file import or export operations on the database. The scheme is simple to implement and can ensure the integrity of data import and export.

[0072] It should be noted that the data processing method and apparatus disclosed herein can be used in the financial field. It can also be used in any field other than finance. The application areas of the data processing method and apparatus disclosed herein are not limited.

[0073] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.

[0074] This invention provides a data processing method applied to a unitized architecture. Figure 1 This is a flowchart illustrating a data processing method provided in an embodiment of the present invention, as shown below. Figure 1 As shown, the method includes:

[0075] S101, in response to control commands to the database in the unitized architecture, obtains the database import or export configuration parameters.

[0076] S102, determine the operation processing method corresponding to the above control command.

[0077] S103, based on the above import or export configuration parameters and the above operation processing method, execute the import operation to import the data file into the file list of the above database, or execute the export operation to export the data file from the file list of the above database.

[0078] The data processing method provided in this application, in response to control instructions on a database in a unitized architecture, obtains import or export configuration parameters of the database, i.e., configuration parameters used to characterize the import or export operation performed on the database; determines the operation processing method corresponding to the control instructions, such as import processing method and export processing method; and, based on the import or export configuration parameters and the operation processing method, performs an import operation to import the data file into the file list of the database, or performs an export operation to export the data file from the file list of the database.

[0079] This application's embodiments can be applied to scenarios where applications are undergoing unitized architecture transformation. Taking an asynchronous database under a unitized architecture as an example, it involves exporting existing data from an Oracle database to a MySQL database, i.e., importing existing data into a MySQL database. By providing a data processing scheme for performing data file import or export operations on databases in a unitized architecture, this scheme achieves simple implementation and ensures the integrity of data import and export.

[0080] Optionally, in this embodiment of the disclosure, the above-mentioned import or export configuration parameters are used to characterize the configuration parameters when performing import or export operations on the database.

[0081] Optionally, in this embodiment of the disclosure, the above operation processing method includes: import processing and export processing, wherein import processing imports the data file into the file list of the above database, and export processing exports the data file from the file list of the above database.

[0082] Optionally, in this embodiment of the disclosure, a data import / export configuration template can be pre-defined on the application side, taking into account the data tables and development specifications involved in the data processing process. This allows the application side to easily and conveniently perform data file import or export operations by configuring only one import / export configuration template.

[0083] For example, before implementing the import / export operation of the embodiments of this disclosure, the method provided in the embodiments of this disclosure generates import or export configuration parameters for the database in response to the import / export configuration instruction of the database on the application side. For example, the application side may provide an import / export configuration template for the database to generate import or export configuration parameters that match the database based on the import / export configuration template.

[0084] Optionally, in this embodiment of the disclosure, the above-mentioned import or export configuration parameters include at least: Uniform Resource Locator dbUrl, username dbUsername, password dbPassword (e.g., plaintext password), configuration file ConfigXml (without XML suffix) for the data file, directory ConfigPath where the configuration file is located, directory FilePath where the data file is located, file name FileName (with suffix) for the data file, temporary directory TempPath, failure directory SuccPath, success directory FailPath, and exception post-processing flag FailContinueFlag (e.g., whether to continue processing if import fails in the previous task, where true means yes and false means no).

[0085] In this embodiment of the disclosure, to achieve the mapping between fields in the data file and fields in the database table, and to be compatible with importing fixed-length and delimited files, and compatible with Windows and Unix / Linux platforms, the system allows the user to choose whether to ignore or overwrite fields when there is a primary key conflict between fields in the data file and fields in the database table during the import process of the data file into the database; it also supports special processing of fields during the export process of the data file from the database; and it supports setting formats and assigning default values ​​to fields, and performing relevant data processing on specific fields, such as truncation and padding with zeros.

[0086] In one example, due to the large database system, there are many types of data tables involved in data import and export, such as transaction information, customer agreements, system parameters, etc. The import and export field configurations of different types of data tables are different. In this embodiment of the disclosure, considering the primary key conflict handling method, it can be compatible with different data tables and different data types of operations, and the rule format is consistent and the configuration is simple.

[0087] In another example, since inline file interaction in the database generally uses two file formats: fixed length and delimiter, this embodiment of the disclosure can be compatible with both file formats and multiple platforms, such as Windows and Unix / Linux.

[0088] In one example, due to the massive amount of data in the tables of a large database system, performance needs to be considered when importing and exporting data from different tables. This embodiment supports sharding, serial processing, and parallel processing to meet flexible calling requirements. It also supports commonly used file extensions in the industry and meets the specifications for batch file processing.

[0089] In one example, the data files being imported and exported can be made perceptible in this embodiment of the disclosure. If the import / export process is completed or an abnormality occurs, the application side can be notified in a timely manner so that the user can take timely and targeted measures to address the situation.

[0090] In one example, the exported data file can not only perform data interaction with downstream applications of the database or the data lake, but also has the function of generating a verification file (CHK file) for the exported data file, and supports multiple CHK file generation methods.

[0091] Based on the above technical solutions, this invention provides a data import and export scheme for heterogeneous database migration under a unitized architecture. Based on a MySQL database, it performs data file import and export processing, which can solve the problems related to the integrity and accuracy of heterogeneous database data during migration.

[0092] In one example, if the above operation is handled as described in the import process, such as... Figure 2 As shown, the above-described import operation, which imports data files into the file list of the database based on the above-described import or export configuration parameters and operation processing methods, includes:

[0093] S201, Based on the import configuration parameters, obtain the data file to be imported into the database and the filename of the data file.

[0094] S202, if the filename of the aforementioned data file contains a predetermined suffix and a wildcard, then determine the list of files in the aforementioned database that match the aforementioned data file.

[0095] S203, obtain the file structure of the data file to be imported and the list structure of the file list to create a file importer.

[0096] S204, Use the above-mentioned file importer to import the above-mentioned data file into the above-mentioned file list.

[0097] As an optional embodiment, if the above operation processing method is import processing, the corresponding import configuration parameters may include, but are not limited to: Uniform Resource Locator dbUrl, username dbUsername, password dbPassword (e.g., plaintext password), configuration file importConfigXml (without XML suffix), directory where the configuration file is located importConfigPath, directory where the import data file is located importFilePath, filename of the import data file importFileName (with suffix), temporary directory importTempPath, failure directory importSuccPath, success directory importFailPath, and exception post-processing flag importFailContinueFlag (e.g., whether to continue processing if import fails in the previous task, where true means yes and false means no).

[0098] In one example, it is first determined whether the number of imported configuration parameters is the pre-configured number, for example, but not limited to 11 as in this embodiment. If it is determined to be 11, then it is determined whether other configuration items besides the imported configuration parameters are successfully imported. If successful, an import configuration object is constructed based on the other configuration items to set the data source and the temporary import directory.

[0099] Next, it checks if the file extension of the data file to be imported is a predetermined extension, such as bin or txt. If it is bin or txt, it checks if the filename of the data file to be imported contains the wildcard character *. If the wildcard character * exists, it finds a list of files that match the data file to be imported; it retrieves the next record in the file list; it retrieves the import success directory and failure directory from the import configuration parameters, retrieves the current date, and creates a new directory.

[0100] In one example, still as Figure 2 As shown, the process of obtaining the file structure of the data file to be imported and the list structure of the file list to create a file importer includes:

[0101] S301, retrieve the configuration file from the above import configuration parameters;

[0102] S302, based on the above configuration file, read the file structure of the above data file and the list structure of the above file list, and construct the above file importer based on the above file structure and the above list structure.

[0103] In this embodiment of the disclosure, the XML configuration file in the import or export configuration parameters is obtained. Through the XML configuration file, the file structure of the data file to be imported and the list structure of the corresponding data list are read to create a file importer. Then, the file importer is used to import the data file into the above-mentioned file list.

[0104] In one example, such as Figure 3 As shown, the above-mentioned data file is imported into the above-mentioned file list using the above-mentioned file importer, including:

[0105] S401, the above-mentioned data file is split using the file importer to obtain multiple split sub-files.

[0106] S402, if it is determined that the above data file needs to be split and stored in the database, then traverse multiple of the above split sub-files to import each of the above split sub-files into the above file list.

[0107] S403, If it is determined that the above data files do not need to be split and stored in the database, then the above data files are directly imported into the above file list.

[0108] Next, the data file to be imported is split, for example, the data file is split into subfiles with a maximum of 80,000 rows each. If a splitting exception occurs, the error code in the output exception warning message can be 90010003, and the error description can be, but is not limited to, "failed to split the data file into subfiles".

[0109] If no splitting anomaly is found, determine whether the data file needs to be split and imported into the database. If so, iterate through multiple split sub-files to import each split sub-file into the file list. When the import is confirmed to be successful, move the split sub-files to the import success directory. Check whether the iteration of multiple split sub-files is complete. If not, return to execute the import of each split sub-file into the file list.

[0110] If it is determined that the above data files do not need to be split and added to the database, directly download the above data files to the above file list and determine whether the download was successful. If successful, move the data files and the corresponding chk checksum file (if they exist) to the import success directory. If it fails, move the data files and the corresponding chk checksum file (if they exist) to the import failure directory and output an error alarm message. The error code in the error message can be 90010010, and the corresponding error description is: file import failed.

[0111] In one example, such as Figure 4 As shown, the above method also includes:

[0112] S501, if it is determined that there is an error in the import processing of the above data file or the above split sub-file, then the import flag is updated to the processing error, and the above data file or the above split sub-file is moved to the import failure directory.

[0113] S502, if it is determined that the import exception post-processing flag in the above import configuration parameters is to continue processing, then continue to execute the import process until it is completed, and then end the above import process.

[0114] S503, if it is determined that the import exception handling flag in the above import configuration parameters is to stop processing, then all data files imported after the current file in the above file list are moved to the import failure directory and the import process is terminated.

[0115] S504, if it is determined that the above data file or the above split sub-file has been successfully imported into the above file list, then move the above data file or the above split sub-file to the directory where the import was successful until the import process is completed.

[0116] In this embodiment of the disclosure, if it is determined that there is an exception during the file processing, the import exception post-processing flag (i.e., whether to continue import failure flag) in the import configuration parameters is obtained, and it is determined whether the import exception post-processing flag is true. If the import exception post-processing flag is true, the import processing continues until it is completed, and the above import processing ends. If the import exception post-processing flag is false, all files after the current file in the file list are moved to the import failure directory, and the exception information and error code that occurred during the processing are output.

[0117] In another optional example, when it is determined that the import data file failed, the import flag is set to false (initially true), the split sub-files are moved to the import failure directory, the import exception post-processing flag in the import configuration parameters is checked, and it is determined whether the import exception post-processing flag is true. If the import exception post-processing flag is true, the above judgment on whether the sub-file list traversal is complete is performed. If the traversal is not complete, all sub-files after the current sub-file in the file list are moved to the import failure directory, the loop is broken, and the judgment on whether the import flag is true is returned.

[0118] If the import flag is true, the data file and the corresponding chk check file (if they exist) will be moved to the import success directory; if the import flag is false, the data file and the corresponding chk check file (if they exist) will be moved to the import failure directory, and an error alarm message will be output. The error code in the error message can be 90010010, and the corresponding error description is: file import failed.

[0119] As an optional embodiment, the input parameters of the file importer or file exporter include: operation type 1 or 2, where 1 is import and 2 is export; import / export tasks and the number of processing task threads. In this embodiment, at least one set of import / export tasks is configured, and multiple sets are supported. Each set is separated by square brackets []. Furthermore, the task types (import / export) of multiple sets can be different. For example, one import set can be configured with multiple parameters, and different parameters can be separated by spaces. The number of processing task threads refers to the number of concurrent processing threads configured for multiple sets of tasks; if not configured, it is single-threaded processing.

[0120] In one example, an example of calling a file importer or file exporter in this disclosure embodiment may be, but is not limited to: java -jar xxx.jar 1[][]2.

[0121] Call the JAR package via command line, uploading two to three of the above input parameters; obtain the first parameter: operation type, and determine if it is operation type 1 or operation type 2; if not, output an exception message, i.e., error code: 90010002, error description: invalid operation type. If it is, obtain the second parameter: import / export task, parse it by [], and obtain a list of multiple sets of content from the uploaded parameters (if the list is empty, the process fails and the program is interrupted).

[0122] Next, iterate through the list of contents, splitting each piece of content into an array based on spaces, and placing it into an array list. Check if the length of the array list is 1. If the length of the array list is 1, call the import / export method for a single data file, return the processing result, and end. If the length of the array list is not 1, then check if the number of parameters called in the main method is 2.

[0123] If the number is 2, then iterate through the array list, call the import / export method of a single data file for each task, and get the return result. If the return result is that all tasks were executed successfully, then return 0. If the return result is that not all tasks were executed successfully, then return the error code: 90010014, error description: serial processing of multiple tasks failed.

[0124] If the number is not 2, then read the third parameter: the number of threads for processing tasks, create a thread pool, the core threads and the maximum number of threads are the number of threads to send, but the maximum is 50, and the waiting queue is 100; traverse the array list, each sub-thread processes one record, and performs import and export of a single data file; get the result of each sub-thread's processing, if all are executed successfully, return 0, otherwise return the error code: 90010012 - Failure occurred in parallel processing of multiple tasks, end.

[0125] In one example, if the above operation is processed as described in the export process, such as... Figure 5As shown, the above-described export operation, which involves exporting the data files from the file list of the database based on the import or export configuration parameters and the operation processing method, includes:

[0126] S601, based on the export configuration parameters, obtain the list of files containing the data files of the database to be exported;

[0127] S602, Obtain the file structure of the data file to be exported and the list structure of the file list to create a file exporter;

[0128] S603, the above-mentioned file exporter is used to determine the export path of the above-mentioned data file, so as to export the above-mentioned data file to the above-mentioned database based on the above-mentioned export path, and create a corresponding verification file for the above-mentioned data file after the export is successful.

[0129] Optionally, if the above operation processing method is the above export processing, the corresponding export configuration parameters may include, but are not limited to: Uniform Resource Locator dbUrl, username dbUsername, password dbPassword (e.g., plaintext password), configuration file for the exported data file exportConfigXml (without the XML suffix), directory where the configuration file for the exported data file is located exportConfigPath, directory where the exported data file is located exportFilePath, filename of the data file exportFileName (with suffix), and the condition whereCondition in the export statement. Multiple conditions can be separated by #@#.

[0130] In one example, the system first checks if the number of exported configuration parameters is a predetermined 7 or 8. If it is not a predetermined 7 or 8, an exception message is output: Error code: 90010015, Error description: Illegal parameter exception. If it is a predetermined 7 or 8, the system initializes the configuration, reads the database configuration, and checks if the data source was created successfully. If data source creation fails, an exception message is output: Error code: 90010000, Error description: Data source initialization failed.

[0131] If the data source is successfully created, then check if other configuration items besides the database configuration were successfully read. If it fails, output the following error information: Error code: 90010005, Error description: Illegal parameter exception; if it succeeds, construct the export configuration object for the above configuration items, set the data source, and set the export directory.

[0132] Next, the export configuration file name and file directory prefix are obtained from the export configuration parameters. The configuration information is read and combined with the data list in the database to construct a file exporter. The file exporter is then used to set the export file name and export file path, export the data file, and determine whether the data file has been exported successfully.

[0133] If the data file export fails, the following error message will be output: Error code: 90010011, Error description: The process ended after file export failed. If the data file export succeeds, the corresponding CHK recovery file will be created and the process ended. If the CHK file creation fails, the following error message will be output: Error code: 90010111, Error description: CHK file creation failed.

[0134] In this embodiment of the disclosure, if it is necessary to interact with the data file with a third-party downstream or data lake in the future, a verification file for the data file needs to be generated. For example, in this embodiment of the disclosure, there are two types of verification files: a general verification file is generated for all data files in the directory; and a corresponding verification file is generated for each data file in the directory.

[0135] This disclosure provides a data import and export scheme for heterogeneous database migration under a unitized architecture. It enables the import and export of MySQL data files through simple configuration, making configuration convenient. It supports fixed-length and delimiter modes, supports multiple environments including Windows and Unix, and supports various field data types. It supports inline-wide delimiter / fixed-length BIN and TXT files. This disclosure supports concurrent import / export of multiple BIN or TXT files, and supports special field processing during the import / export process. This disclosure supports automatic splitting and CHK verification of large files, and supports application-side control over whether to interrupt or rerun file processing failures. This disclosure supports file import / export job monitoring, including job processing failure notifications and alarms, and monitoring and alarms for long job execution times.

[0136] In this embodiment of the disclosure, the exported data file can generate a CHK verification file, which can be directly used for verification and inclusion in the data lake.

[0137] According to one or more embodiments of this application, a data processing apparatus is provided. Figure 6 A structural block diagram of a data processing apparatus provided in an embodiment of this application is shown below. Figure 6 As shown, the above-mentioned device includes:

[0138] The acquisition module 600 is used to acquire the import or export configuration parameters of the database in response to the control command of the database in the unitized architecture. The import or export configuration parameters are used to characterize the configuration parameters when performing import or export operations on the database.

[0139] The determining module 601 is used to determine the operation processing method corresponding to the above control instructions, wherein the above operation processing method includes: import processing and export processing;

[0140] The processing module 602 is used to perform an import operation to import the data file into the file list of the database, or to perform an export operation to export the data file from the file list of the database, based on the import or export configuration parameters and the operation processing method described above.

[0141] Furthermore, if the above operation processing method is the above import processing, the above processing module includes:

[0142] The first acquisition submodule is used to acquire the data file to be imported into the above database and the file name of the above data file according to the import configuration parameters;

[0143] The determination submodule is used to determine the list of files in the database that match the data file if the filename of the data file contains a predetermined suffix and wildcard.

[0144] The second acquisition submodule is used to acquire the file structure of the data file to be imported and the list structure of the file list to create a file importer.

[0145] The import submodule is used to import the data files into the file list using the file importer described above.

[0146] Furthermore, the second acquisition submodule includes:

[0147] The acquisition unit is used to retrieve the configuration file from the imported configuration parameters mentioned above.

[0148] The construction unit is used to read the file structure of the data file and the list structure of the file list according to the configuration file, so as to construct the file importer based on the file structure and the list structure.

[0149] Furthermore, the aforementioned import submodule includes:

[0150] The splitting unit is used to split the data file using the file importer described above, so as to obtain multiple split sub-files;

[0151] The first import unit is used to, if it is determined that the above data file needs to be split and stored in the database, traverse multiple of the above split sub-files and import each of the above split sub-files into the above file list.

[0152] The second import unit is used to directly import the data files into the file list if it is determined that the data files do not need to be split and stored in the database.

[0153] Furthermore, if the above operation processing method is the above export processing, the above processing module includes:

[0154] Based on the export configuration parameters, obtain the list of files containing the data files of the database to be exported;

[0155] Obtain the file structure of the data file to be exported and the list structure of the file list to create a file exporter;

[0156] The file exporter described above is used to determine the export path of the data file, so as to export the data file to the database based on the export path, and create a corresponding verification file for the data file after successful export.

[0157] In an exemplary embodiment, this application also provides an electronic device, including: a processor, and a memory connected to the processor;

[0158] The aforementioned memory stores instructions executed by the computer;

[0159] The processor executes computer execution instructions stored in the memory to implement any of the methods described above.

[0160] In an exemplary embodiment, this application also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the methods described above.

[0161] In an exemplary embodiment, this application also provides a computer program product, including a computer program that, when executed by a processor, implements any of the methods described above.

[0162] To implement the above embodiments, this application also provides an electronic device.

[0163] refer to Figure 7The diagram illustrates a structural schematic of an electronic device 700 suitable for implementing embodiments of this application. The electronic device 700 can be a terminal device or a server. The terminal device can include, but is not limited to, mobile terminals such as mobile phones, laptops, digital radio receivers, personal digital assistants (PDAs), portable Android devices (PADs), portable media players (PMPs), and in-vehicle terminals (e.g., in-vehicle navigation terminals), as well as fixed terminals such as digital TVs and desktop computers. Figure 7 The electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0164] like Figure 7 As shown, the electronic device 700 may include a processing unit (e.g., a central processing unit, a graphics processing unit, etc.) 701, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 702 or a program loaded from a storage device 708 into a random access memory (RAM) 703. The RAM 703 also stores various programs and data required for the operation of the electronic device 700. The processing unit 701, ROM 702, and RAM 703 are interconnected via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0165] Typically, the following devices can be connected to I / O interface 705: input devices 706 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 707 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 708 including, for example, magnetic tapes, hard disks, etc.; and communication devices 709. Communication device 709 allows electronic device 700 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 7 An electronic device 700 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively.

[0166] Specifically, according to embodiments of this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication device 709, or installed from storage device 708, or installed from ROM 702. When the computer program is executed by processing device 701, it performs the functions defined in the methods of embodiments of this application.

[0167] It should be noted that the computer-readable medium described above in this application can be a computer-readable signal medium, a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this application, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this application, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0168] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device.

[0169] The aforementioned computer-readable medium carries one or more programs, which, when executed by the electronic device, cause the electronic device to perform the methods shown in the above embodiments.

[0170] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof. These programming languages ​​include object-oriented programming languages—such as Java, Smalltalk, and C++—and conventional procedural programming languages—such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a Local Area Network (LAN) or a Wide Area Network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0171] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0172] The units described in the embodiments of this application can be implemented in software or in hardware. The name of a unit does not necessarily limit the unit itself; for example, the first acquisition unit can also be described as "a unit that acquires at least two Internet Protocol addresses".

[0173] The functions described above in this document can be performed, at least in part, by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: Field Programmable Gate Arrays (FPGAs), Application-Specific Integrated Circuits (ASICs), Application Standard Products (ASSPs), System-on-Chip (SoCs), Complex Programmable Logic Devices (CPLDs), and so on.

[0174] In the context of this application, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. Machine-readable media can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

Claims

1. A data processing method, characterized in that, Applied to a modular architecture, the method includes: In response to a control command on a database in a unitized architecture, import or export configuration parameters of the database are obtained, wherein the import or export configuration parameters are used to characterize the configuration parameters when performing import or export operations on the database in the unitized architecture. Determine the operation processing method corresponding to the control command, wherein the operation processing method includes: import processing and export processing; If the operation processing method is the import processing, the data file to be imported into the database and the file name of the data file are obtained according to the import configuration parameters; If the filename of the data file contains a predetermined suffix and a wildcard, then a list of files in the database that match the data file is determined; Obtain the file structure of the data file to be imported and the list structure of the file list to create a file importer; The data file is imported into the file list using the file importer. If the operation processing method is the export processing, obtain the file list where the data files of the database to be exported are located according to the export configuration parameters; Obtain the file structure of the data file to be exported and the list structure of the file list to create a file exporter; The file exporter is used to determine the export path of the data file, so as to export the data file to the database based on the export path, and create a corresponding verification file for the data file after successful export.

2. The method according to claim 1, characterized in that, The step of obtaining the file structure of the data file to be imported and the list structure of the file list to create a file importer includes: Retrieve the configuration file from the imported configuration parameters; Based on the configuration file, the file structure of the data file and the list structure of the file list are read to construct the file importer.

3. The method according to claim 1, characterized in that, The step of importing the data file into the file list using the file importer includes: The data file is split using the file importer to obtain multiple split sub-files; If it is determined that the data file needs to be split and stored in the database, then multiple split sub-files are traversed to import each split sub-file into the file list; If it is determined that the data file does not need to be split and stored in the database, then the data file is directly imported into the file list.

4. The method according to claim 3, characterized in that, The method further includes: If it is determined that there is an error in the import process of the data file or the split sub-file, the import flag is updated to indicate an error, and the data file or the split sub-file is moved to the import failure directory. If the import exception post-processing flag in the import or export configuration parameters is determined to be "continue processing", then the import process continues until it is completed, and then the import process ends. If the import exception post-processing flag in the import or export configuration parameters is determined to be "stop processing", then all data files imported after the current file in the file list will be moved to the import failure directory, and the import process will be terminated. If it is determined that the data file or the split sub-file has been successfully imported into the file list, then the data file or the split sub-file is moved to the directory where the import was successful until the import process is complete.

5. The method according to any one of claims 1 to 4, characterized in that, The method further includes: In response to the import / export configuration command for the database, import or export configuration parameters for the database are generated, wherein the import or export configuration parameters include at least: Uniform Resource Locator, username, password, configuration file for the data file, directory of the data file to be imported, filename of the data file to be imported, temporary directory, import failure directory, import success directory, and import exception post-processing flag.

6. A data processing apparatus, characterized in that, Applied to a modular architecture, the device includes: The acquisition module is used to acquire the import or export configuration parameters of the database in response to the control command of the database in the unitized architecture, wherein the import or export configuration parameters are used to characterize the configuration parameters when performing import or export operations on the database. The determination module is used to determine the operation processing method corresponding to the control command, wherein the operation processing method includes: import processing and export processing; The processing module is used to perform an import operation to import the data file into the file list of the database, or to perform an export operation to export the data file from the file list of the database, based on the import or export configuration parameters and the operation processing method. If the operation processing method is the import processing, the processing module includes: The first acquisition submodule is used to acquire the data file to be imported into the database and the file name of the data file according to the import configuration parameters; The determination submodule is used to determine a list of files in the database that match the data file if the filename of the data file contains a predetermined suffix and a wildcard. The second acquisition submodule is used to acquire the file structure of the data file to be imported and the list structure of the file list in order to create a file importer; The import submodule is used to import the data file into the file list using the file importer. If the operation processing method is the export processing, the processing module includes: Based on the export configuration parameters, obtain the list of files containing the data files of the database to be exported; Obtain the file structure of the data file to be exported and the list structure of the file list to create a file exporter; The file exporter is used to determine the export path of the data file, so as to export the data file to the database based on the export path, and create a corresponding verification file for the data file after successful export.

7. The apparatus according to claim 6, characterized in that, The second acquisition submodule includes: The acquisition unit is used to acquire the configuration file in the imported configuration parameters; The construction unit is used to read the file structure of the data file and the list structure of the file list according to the configuration file, so as to construct the file importer based on the file structure and the list structure.

8. The apparatus according to claim 6, characterized in that, The import submodule includes: A splitting unit is used to split the data file using the file importer to obtain multiple split sub-files; The first import unit is used to, if it is determined that the data file needs to be split and stored in the database, traverse multiple split sub-files to import each split sub-file into the file list; The second import unit is used to directly import the data file into the file list if it is determined that the data file does not need to be split and stored in the database.

9. An electronic device, characterized in that, include: A processor, and a memory connected to the processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory to implement the method as described in any one of claims 1 to 5.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of claims 1 to 5.

11. A computer program product, characterized in that, Includes a computer program that, when executed by a processor, implements the method of any one of claims 1 to 5.

Images