Data migration method and device, electronic equipment and storage medium

Abstract

The invention discloses a data migration method and device, electronic equipment and a storage medium, and the method comprises the steps: responding to a data file of an old system, and judging a reading mode of the data file; if it is judged that the data file reading mode is a file needing conversion and verification, the data file is stored in a first mode; if it is judged that the data file reading mode is a file which does not need to be converted and verified, the data file is stored in a second mode, the first mode at least comprises file data conversion and verification, and the second mode directly stores the data file in a copying mode. According to the method and the device, data files with different databases and different business requirements can be flexibly and automatically processed. Meanwhile, aiming at the data difference between the new system and the old system, data conversion and data quality verification of the data accurate to each field are supported, and the accuracy and integrity of the data are ensured.

Description

Technical Field

[0001] The present application relates to the technical field of data migration between old and new banking systems, and in particular to a data migration method, device, electronic device, and storage medium. Background Art

[0002] During the data migration process between the old and new banking systems, the customer data text files exported from the old system need to be converted and verified according to the rules and then loaded into the intermediate table database.

[0003] Since files are exported based on tables, the business scenarios of each table are the same, resulting in huge differences in the amount of data in different files. File data from different tables also need to be entered into different databases. To ensure the accuracy and completeness of the data, the data needs to be converted and quality-verified according to business needs. The conversion and verification rules are different for different tables and fields. Data migration has extremely high timeliness requirements, and a flexible and efficient file processing method is needed.

[0004] Existing technologies cannot flexibly provide flexible file data processing and warehousing solutions for complex business scenarios. They cannot accurately convert and verify data during the file warehousing process, and the accuracy and integrity of the data cannot be guaranteed. They cannot meet the needs of banks migrating massive data from old and new systems in complex scenarios. Summary of the Invention

[0005] The embodiments of the present application provide a data migration method, device, electronic device, and storage medium to quickly convert, verify, and load files into a database based on file characteristics and data characteristics.

[0006] The embodiments of this application adopt the following technical solutions:

[0007] In a first aspect, an embodiment of the present application provides a data migration method, wherein the method includes:

[0008] In response to a data file of the old system, determining a reading method of the data file;

[0009] If it is determined that the data file reading method is a file that needs to be converted and verified, then it is stored in the first method;

[0010] If it is determined that the data file reading method is a file that does not require conversion or verification, it is stored in the second method.

[0011] The first method at least includes the conversion and verification of file data, and the second method directly stores the data in the database by copying.

[0012] In some embodiments, if it is determined that the data file reading method is a file that needs to be converted and verified, then the data file is stored in the first manner, including:

[0013] For the data file that needs to be converted, parse each row of data and query the conversion rule table preloaded into memory to check whether there are any fields that need to be converted;

[0014] If yes, then call the method corresponding to the code according to the query conversion rule code to perform data conversion and replace the original value to complete the conversion;

[0015] Determine verification rules based on analysis documents, formulate verification rule codes and pre-allocate them into the verification rule table;

[0016] For data files that require quality verification, the migration process queries the pre-loaded verification rule table in memory when parsing each row of data to check whether the current data file contains fields that require verification. The quality verification items include at least one of the following: code value range, date format, and field length.

[0017] If yes, then call the method corresponding to the code for verification according to the query verification rule code;

[0018] If the verification passes, continue parsing the next line;

[0019] If the verification fails, the row will be recorded as a verification failure record and will not be loaded any further, and the remaining rows will continue to be parsed.

[0020] In some embodiments, if it is determined that the data file reading method is a file that needs to be converted and verified, then the data file is stored in the first manner, including:

[0021] For files that need to be converted and verified in the data file reading mode, the table structure corresponding to the table that needs to be stored is queried by querying the pre-parsed table name and its table structure configuration;

[0022] Assemble the table structure and the parsed data list into an SQL structured query statement;

[0023] Then the data source is switched according to the pre-acquired data source and finally falls into the database.

[0024] In some embodiments, the method further comprises:

[0025] The files that need to be converted and verified are processed according to the configuration and finally stored in the database;

[0026] Set the field order to be the same as the data order in the file as the loading table name and table structure configuration of the loading details table;

[0027] and / or, according to the target system code, the target DUS distinguishes the data source information of the configured full meter;

[0028] And / or, based on user needs, tables that do not require conversion and verification are configured to be stored in copy mode, and the file data corresponding to the remaining tables are parsed and stored in the database using the conversion and verification method;

[0029] And / or, before processing the file, configure the conversion rule configuration table and the verification rule configuration table according to requirements.

[0030] In some embodiments, the method further comprises:

[0031] Record the abnormal data in the abnormal file by line;

[0032] Exceptions at the file level during parsing are recorded in the exception information column of the load details table. Exceptions at the data level during parsing are written into the exception file.

[0033] If an exception occurs during data entry, data rollback or full data entry will be adopted.

[0034] If you choose the data rollback method, if a row of data fails, the entire batch of data will fail;

[0035] If you select the "enter as much data as possible" method, the error row will be skipped and the remaining data will continue to be stored.

[0036] In some embodiments, the method further comprises:

[0037] Get a file name that includes at least one of the following: target system code, target DUS, D model table name;

[0038] When parsing a file, parse the file name and obtain the D model table name, target system code, and target DUS;

[0039] Get the file data source from the configuration table and record the data source so that the data source can be switched when the file is dropped from the library.

[0040] In some embodiments, the method further comprises:

[0041] After the file is saved in the database, the total number of rows in the file is verified.

[0042] Determine whether all files have been processed successfully. Based on the comparison results, update the status in the file data loading details table to success, failure, or partial success.

[0043] In a second aspect, an embodiment of the present application further provides a data migration device, wherein the device includes:

[0044] a response module, configured to respond to a data file of the old system and determine a reading method of the data file;

[0045] A first-mode warehousing module, configured to, if it is determined that the data file reading mode is a file that needs conversion and verification, store it in the warehousing according to the first mode;

[0046] The second mode storage module is used to store the data file in the second mode if it is determined that the data file reading mode does not require conversion or verification.

[0047] The first method at least includes the conversion and verification of file data, and the second method directly stores the data in the database by copying.

[0048] In a third aspect, an embodiment of the present application further provides an electronic device, comprising: a processor; and a memory arranged to store computer-executable instructions, wherein the executable instructions, when executed, enable the processor to perform the above method.

[0049] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, which stores one or more programs. When the one or more programs are executed by an electronic device including multiple application programs, the electronic device executes the above method.

[0050] At least one of the above technical solutions adopted in the embodiments of the present application can achieve the following beneficial effects: in response to the data files of the old system, the reading method of the data files is judged, and if the reading method of the data files is judged to be files that need to be converted and verified according to the reading method, the data files are stored in the first method; if the reading method of the data files is judged to be files that do not need to be converted and verified, the data files are stored in the second method. The first method at least includes the conversion and verification of the file data, and the second method directly stores the data in the copying method. Through the above method, when processing files of different databases and different business needs, it has flexibility and full automation. In view of the data differences between the old and new systems, it supports data conversion and data quality verification accurate to each field, ensuring the accuracy and integrity of the data. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0052] Figure 1 This is a schematic diagram of the principle of data file loading in the embodiment of the present application;

[0053] Figure 2 Schematic diagram of the data migration method in the embodiment of the present application;

[0054] Figure 3 This is a flow chart of data file loading in an embodiment of the present application;

[0055] Figure 4 This is a schematic diagram of the data conversion process in the embodiment of the present application;

[0056] FIG5( a ) is a schematic diagram of a sample configuration of conversion rules in an embodiment of the present application;

[0057] FIG5( b ) is a schematic diagram of a sample configuration of a code value conversion table in an embodiment of the present application;

[0058] Figure 6 This is a schematic diagram of the data verification process in the embodiment of this application;

[0059] Figure 7 This is a schematic diagram of an example of a verification rule table in an embodiment of the present application;

[0060] Figure 8 This is a structural diagram of a data migration device in an embodiment of the present application;

[0061] Figure 9 This is a schematic structural diagram of an electronic device in an embodiment of the present application. DETAILED DESCRIPTION

[0062] To make the purpose, technical solutions, and advantages of this application more clear, the technical solutions of this application will be clearly and completely described below in conjunction with the specific embodiments of this application and the corresponding drawings. Obviously, the embodiments described are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0063] The technical terms involved in the embodiments of this application are as follows:

[0064] Data migration: Smoothly migrate the business and technical information carried by the data in the original system to the target system, and ensure that the business meaning of all customer service data before and after the migration does not change.

[0065] Intermediate table: A table used to temporarily store data after exporting it from the source system during data migration. It maintains the same structure as the target database table and is used to detect data problems before importing it into the target database to avoid affecting production user data.

[0066] D model: It is the data unit objects and related entities in IT design, but based on the specific implementation platform, it adds database management system-specific specifications, such as index and partition design, and ultimately forms an application physical model at the system implementation level.

[0067] Target system code: The proprietary system code of the system to which the target database belongs during data migration.

[0068] DUS: refers to a service deployment unit, which is a self-contained collection that can complete all business operations. The collection includes all services required by all businesses and the data allocated to this unit.

[0069] Data source: A data source refers to the database or database server used by a database application. All information for establishing a database connection is stored in the data source. By providing the correct data source name, you can find the corresponding database connection.

[0070] The technical solutions provided by the embodiments of the present application are described in detail below with reference to the accompanying drawings.

[0071] like Figure 1 As shown in the figure, a file loading mechanism is provided between the legacy system data files and the target database. Specifically, it includes configuration, preprocessing, file parsing, and data import. The file parsing section also includes data conversion and verification. The data storage section includes both COPY and standard data import. The configuration section configures the table names and structures to be loaded. The preprocessing section splits files with large data volumes that require standard parsing and import. The file parsing section supports concurrent processing of multiple files. When processing multiple files simultaneously, each thread processes a single file concurrently. Exception handling is also included to handle parsing exceptions. File-level exceptions are recorded in the exception information column of the load details table, while row-level exceptions are written to the exception file. This flexible combination of loading methods significantly improves file loading performance. Whether processing large amounts of data or complex data processing scenarios, the system can respond quickly and efficiently. This not only enhances the system's practicality but also makes the migration process smoother and more efficient.

[0072] The present application embodiment provides a data migration method, such as Figure 2 As shown, a flow chart of a data migration method according to an embodiment of the present application is provided, wherein the method comprises at least the following steps S210 to S230:

[0073] Step S210 , in response to the data file of the old system, determining a reading method of the data file.

[0074] To transfer legacy system data files to the target database, each file corresponding to a table's data is exported from the legacy system. The file reading method is determined based on the characteristics of the legacy system's data files. Files that require conversion and verification are stored using the standard parsing method, while those that do not require conversion and verification are directly stored using the copy method.

[0075] Step S220: If it is determined that the data file reading method is a file that needs to be converted and verified, it is stored in the first method.

[0076] Specifically, if it is determined that the data file reading method is a file that needs to be converted and verified, the file data will be converted and verified.

[0077] Step S230: If it is determined that the data file reading method is a file that does not require conversion or verification, it is stored in the second method, wherein the first method at least includes conversion and verification of file data, and the second method is directly stored in the library by copying.

[0078] Select different file processing methods based on different needs, and perform complex conversions and data verification on the file data during the file processing process. It can also record in detail any abnormal scenarios that occur during file parsing and storage, and provide different handling options for these abnormal scenarios.

[0079] Through the above method, a method for quickly processing and loading text files is provided, which supports the rapid conversion, verification and loading of files into the warehouse according to the file characteristics and data characteristics. Specifically, for data files that do not require special processing, they are directly put into the warehouse according to the characteristics of the relevant database to ensure the accuracy and efficiency of data migration. For data files exported from the old system, each file corresponds to the data of a table. It also supports judging the file reading method based on file characteristics. For files that need to be converted and verified, ordinary parsing and warehousing are used, and those that do not need them are directly put into the warehouse through copying. Files that need to be converted and verified are processed according to the configuration and finally fall into the database.

[0080] Specifically, the following data migration between the old and new banking systems uses an Oracle database as the legacy system and an openGauss database as the new system. The openGauss database is primarily used for transactional applications, such as those with high concurrency, high data volumes, and those primarily focused on online transaction processing.

[0081] like Figure 3As shown, during the file data migration process, the customer data text files exported from the old system need to be converted and verified according to the rules and then loaded into the intermediate table database. The amount of data in different files varies greatly, and the data in different tables also need to fall into different databases. In order to ensure the accuracy and integrity of the data, the data needs to be converted and quality-verified accordingly. Since the migration is a real-time, non-stop migration based on customers, the timeliness requirements are extremely high. Therefore, the method in the embodiment of the present application is based on the business foundation of data migration in complex business scenarios, which has high requirements for data accuracy and timeliness. It has flexibility and full automation when processing files from different databases and different business needs. It can quickly and automatically locate the data source based on the file identifier and flexibly switch the loading method, so as to easily cope with the complex needs of fast processing of large files and loading files from multiple data sources.

[0082] In one embodiment of the present application, if it is determined that the data file reading method is a file that needs to be converted and verified, it is stored in the database according to the first method, including: for data files that need to be converted, parsing each row of data to query the conversion rule table preloaded into the memory to query whether there are fields that need to be converted; if so, calling the method corresponding to the code according to the query conversion rule code to perform data conversion and replace the original value to complete the conversion; determining the verification rules based on the analysis document, formulating the verification rule code and pre-allocating it into the verification rule table; for data files that need quality verification, when parsing each row of data during the migration process, querying the verification rule table preloaded into the memory to query whether there are fields that need to be verified in the current data file, and the quality verification items include at least one of the following: code value range, date format, field length; if so, calling the method corresponding to the code according to the query verification rule code to perform verification; if the verification passes, continuing to parse the next row; if the verification fails, the row is recorded in the verification failure record and no longer loaded, and the remaining rows are continued to be parsed.

[0083] Optionally, for files with large data volumes that require ordinary parsing and storage, file splitting is performed. First, the number of file entries is obtained. For files with data files larger than a set threshold, the files are split according to the set threshold. The sequence number at the end of the file is increased by 1, and the source file is automatically moved to the backup folder as a preprocessing part.

[0084] The old system database is an Oracle database, and the new system database is an openGauss database. After data is exported from the old system, due to differences in the rules between the old and new systems, the attributes of the data models differ (mainly including but not limited to data length, data type, code value, etc.). When loading files, the data must be converted according to specific rules before it can be stored in the new system. Key conversions include shard value calculation, code value conversion, and organizational level conversion. By identifying the D model to be converted and its attributes, the conversion rules are determined. These rules are pre-assigned to the conversion rule table, and the rule code and rule parameters are filled in.

[0085] like Figure 4 The following is a data conversion flow chart. Taking Table A as an example, the relevant rules of Table A are obtained in the conversion rule configuration table, the relevant data of the corresponding fields in the rules are obtained, the corresponding method of the rule code is called to complete the data conversion, and the original value is replaced with the converted data.

[0086] Figure 5(a) is a sample of the conversion rule configuration. Figure 4 As shown: When parsing each row of data, the conversion rule table preloaded into memory is queried to see if any fields need to be converted. If so, the conversion rule code is queried and the corresponding method is called to convert the data, replacing the original value to complete the conversion. For example, in the example of Figure 5(a), the code value conversion rule with the ID 1 is uniformly named codeMapping. The rule parameters are configured. When parsing the data, if the rule ID 1 is matched based on the table name and field name, the codeMapping method is called. Based on the rule parameters, the code value to be converted is queried from the code value conversion table shown in Figure 5(a). The data is converted and the original value is replaced.

[0087] Continuing with Table A as an example, to ensure data quality, data quality verification is required, including code value range, date format, field length, etc. Based on the analysis document, verification rules are determined, verification rule codes are formulated, and pre-configured into the verification rule table. Verification process is as follows Figure 6 As shown: During the migration process, when parsing each row of data, the verification rule table preloaded into the memory is queried to check whether the data has fields that need to be verified. If so, the verification rule code is queried and the method corresponding to the code is called for verification. If the verification passes, the next row is parsed. If the verification fails, the row is recorded as a verification failure record and no further loading is carried out. The remaining rows are parsed. Figure 7In the example validation rule table shown, Rule 1 specifies that the value of this field can only be the value configured in the rule parameters. During data parsing, if Rule 1 is matched based on the table name and field name, the FORMAT00001 method is called. If the value of field a in the row is 156, the validation passes; otherwise, the validation fails. As you can see, after a batch of data is parsed, the processed data is imported.

[0088] For files that need to be processed, the method in the embodiment of the present application can perform complex data conversion operations accurate to the field on each line of data in the file during the file processing process, and conduct in-depth verification of data quality and data consistency, fully ensuring the accuracy and integrity of the data. At the same time, in view of the data differences between the old and new systems, it supports data conversion and data quality verification accurate to each field, ensuring the accuracy and integrity of the data. The highly configurable and combinable nature of the file loading process gives the system extremely high flexibility and reusability, meeting the diverse needs of different business scenarios.

[0089] In one embodiment of the present application, if it is determined that the data file reading method is a file that needs to be converted and verified, it is stored in the database according to the first method, including: for the data file reading method that needs to be converted and verified, by querying the pre-parsed table name and its table structure configuration, the table structure corresponding to the table that needs to be stored is queried; the table structure and the data list that has been parsed are assembled into an SQL structured query statement; then the data source is switched according to the pre-acquired data source, and finally stored in the database.

[0090] The old system database is an Oracle database, and the new system database is an openGauss database. Figure 5(b) shows an example of a code value conversion table configuration. For the standard drop-in method: First, based on the table name parsed in the previous step, query the table name and its table structure to find the table structure corresponding to the table to be dropped. Combine the table structure with the data list parsed in the previous step to form an insert SQL statement. Then, switch the data source based on the pre-obtained data source. Finally, drop into the database.

[0091] In addition, for the Copy storage method: for files that are determined to need to be copied in the previous steps, the loading method is automatically switched, and the data source is switched according to the pre-acquired data source. The Copy command of the openGauss database is used to directly store the file data into the database without the need for memory parsing, which is very efficient.

[0092] In response to different business requirements for data files and based on different file identification features, the method in the embodiment of the present application designs different file loading processes, automatically switches the loading method, and fully utilizes the characteristics of the openGauss database for files that do not require parsing and processing, thereby improving loading performance.

[0093] In one embodiment of the present application, the method also includes: processing the files that need to be converted and verified according to the configuration, and finally putting them into the database; making the field order the same as the data order in the file, as the loading table name and table structure configuration of the loading details table; and / or, according to the target system code, the target DUS distinguishes and configures the data source information of the full table; and / or, according to user needs, configuring the tables that do not need to be converted and verified to be copied into the warehouse, and parsing the file data corresponding to the remaining tables into the warehouse according to the conversion and verification method; and / or, configuring the conversion rule configuration table and the verification rule configuration table according to needs before file processing.

[0094] The old system database is an Oracle database, and the new system database is an openGauss database. The files that need to be converted and verified are processed according to the configuration, including but not limited to:

[0095] (1) The name of the table to be loaded and its table structure configuration: that is, the order of the fields is the same as the order of the data in the file.

[0096] (2) Table and data source correspondence configuration table: Configure the data source information of the full table, distinguished by the target system code and target DUS.

[0097] (3) File processing method configuration: According to user needs, for tables that do not require conversion and verification, they are configured to be stored in the copy mode, and the files corresponding to the remaining tables are stored in the normal parsing mode.

[0098] (4) Conversion rule configuration table; Verification rule configuration table. Users can flexibly configure the relevant configuration according to their needs before processing the file.

[0099] The method in the embodiment of the present application has a configurable feature. For new tables, new data sources, new conversion and verification rules, users only need to add configurations without the need for complex custom development, thereby achieving functional reusability and reducing maintenance costs.

[0100] In one embodiment of the present application, the method further includes: recording data exceptions in an exception file by line; recording file-level exceptions during parsing in an exception information column of the loading details table; and writing exceptions to each line of data during parsing in an exception file; if an exception is encountered during data entry, adopting a data rollback or a data entry-as-needed processing method; if the data rollback method is selected, then if a row of data fails, then the entire batch of data fails; if the data entry-as-needed method is selected, then the error row is skipped during data entry and the remaining data is continued to be entered.

[0101] The old system database is an Oracle database, and the new system database is an openGauss database.

[0102] Parsing exceptions, including file-level exceptions, are recorded in the exception information column of the load details table. Exceptions down to the data row level are written to the exception file. Data rollback and full data inclusion are supported when data is dropped. This switch controls the rollback method. If a row of data fails, the entire batch of data fails. If the full data inclusion method is selected, the error row is skipped during data drop, and the remaining data is dropped, ensuring the maximum amount of data dropped. Dropped data exceptions are recorded row by row in the exception file.

[0103] The various exception handling methods in the embodiments of the present application record detailed information on exceptions during file processing and loading, making it easier for users to troubleshoot. Furthermore, users can flexibly choose to roll back or re-enter the system for exception scenarios when files are dropped from the library, thus satisfying more application scenarios. Meanwhile, the various exception handling methods record detailed information on exceptions during file processing and loading, making it easier for users to troubleshoot. Furthermore, users can flexibly choose to roll back or re-enter the system for exception scenarios when files are dropped from the library, thus satisfying more application scenarios.

[0104] In one embodiment of the present application, the method further includes: obtaining a file name that includes at least one of the following: a target system code, a target DUS, and a D model table name; parsing the file name when parsing the file, and obtaining the D model table name, the target system code, and the target DUS; obtaining the file data source from the configuration table and recording the data source to switch the data source when the file is stored in the warehouse.

[0105] The old system database is an Oracle database, and the new system database is an openGauss database. When obtaining the file data source, the method in the embodiment of the present application supports processing file names that should contain the following information: target system code, target DUS, and D model table name. When parsing a file, first parse the file name to obtain the D model table name, target system code, and DUS. Then, based on the three elements, the file data source (one or more) can be obtained from the configuration table, and the data source is recorded and switched when the data source is dropped into the library.

[0106] For scenarios where data files from multiple data sources are processed simultaneously, the method in the embodiments of the present application can intelligently identify and automatically obtain one or more target data sources based on file identifiers. When files are stored in the library, the system will automatically switch to the corresponding data source, increasing the application scenario.

[0107] In one embodiment of the present application, the method further includes: after the storage is completed, checking is performed based on the total number of lines of the file obtained; determining whether all files have been processed successfully, and updating the status in the loading details table of the file data to success, failure or partial success based on the comparison results.

[0108] The old system database is an Oracle database, and the new system database is an openGauss database. After the file is loaded into the database, the total number of rows in the file obtained from the previous process is verified to determine whether all files have been successfully processed. Based on the comparison results, the status of the file in the loading details table is updated to success, failure, or partial success.

[0109] The embodiment of the present application also provides a data migration device 800, such as Figure 8 , a schematic diagram of the structure of a data migration device in an embodiment of the present application is provided. The data migration device 800 includes at least: a response module 810, a first-mode storage module 820, and a second-mode storage module 830, wherein:

[0110] In one embodiment of the present application, the response module 810 is specifically configured to: respond to a data file of the old system and determine a reading method of the data file.

[0111] To transfer legacy system data files to the target database, each file corresponding to a table's data is exported from the legacy system. The file reading method is determined based on the characteristics of the legacy system's data files. Files that require conversion and verification are stored using the standard parsing method, while those that do not require conversion and verification are directly stored using the copy method.

[0112] In one embodiment of the present application, the first-mode warehousing module 820 is specifically configured to: if it is determined that the data file reading mode is a file that needs to be converted and verified, then warehousing is performed in the first mode.

[0113] Specifically, if it is determined that the data file reading method is a file that needs to be converted and verified, the file data will be converted and verified.

[0114] In one embodiment of the present application, the second-mode warehousing module 830 is specifically used to: if it is determined that the data file reading method is a file that does not require conversion or verification, then it is stored in the warehouse according to the second method, wherein the first method at least includes conversion and verification of file data, and the second method is directly stored in the warehouse by copying.

[0115] Select different file processing methods based on different needs, and perform complex conversions and data verification on the file data during the file processing process. It can also record in detail any abnormal scenarios that occur during file parsing and storage, and provide different handling options for these abnormal scenarios.

[0116] In one embodiment of the present application, the first-mode warehousing module 820 is further configured to:

[0117] For the data file that needs to be converted, parse each row of data and query the conversion rule table preloaded into memory to check whether there are any fields that need to be converted;

[0118] If yes, then call the method corresponding to the code according to the query conversion rule code to perform data conversion and replace the original value to complete the conversion;

[0119] Determine verification rules based on analysis documents, formulate verification rule codes and pre-allocate them into the verification rule table;

[0120] For data files that require quality verification, the migration process queries the pre-loaded verification rule table in memory when parsing each row of data to check whether the current data file contains fields that require verification. The quality verification items include at least one of the following: code value range, date format, and field length.

[0121] If yes, then call the method corresponding to the code for verification according to the query verification rule code;

[0122] If the verification passes, continue parsing the next line;

[0123] If the verification fails, the row will be recorded as a verification failure record and will not be loaded any further, and the remaining rows will continue to be parsed.

[0124] In one embodiment of the present application, the first-mode warehousing module 820 is further configured to:

[0125] For files that need to be converted and verified in the data file reading mode, the table structure corresponding to the table that needs to be stored is queried by querying the pre-parsed table name and its table structure configuration;

[0126] Assemble the table structure and the parsed data list into an SQL structured query statement;

[0127] Then the data source is switched according to the data source obtained in the previous order, and finally falls into the database.

[0128] In one embodiment of the present application, a configuration module is further included for:

[0129] The files that need to be converted and verified are processed according to the configuration and finally stored in the database;

[0130] Set the field order to be the same as the data order in the file as the loading table name and table structure configuration of the loading details table;

[0131] and / or, according to the target system code, the target DUS distinguishes the data source information of the configured full meter;

[0132] And / or, based on user needs, tables that do not require conversion and verification are configured to be stored in copy mode, and the file data corresponding to the remaining tables are parsed and stored in the database using the conversion and verification method;

[0133] And / or, before processing the file, configure the conversion rule configuration table and the verification rule configuration table according to requirements.

[0134] In one embodiment of the present application, an exception handling module is further included, which is used to:

[0135] Record the abnormal data in the abnormal file by line;

[0136] Exceptions at the file level during parsing are recorded in the exception information column of the load details table. Exceptions at the data level during parsing are written into the exception file.

[0137] If an exception occurs during data entry, data rollback or full data entry will be adopted.

[0138] If you choose the data rollback method, if a row of data fails, the entire batch of data will fail;

[0139] If you select the "enter as much data as possible" method, the error row will be skipped and the remaining data will continue to be stored.

[0140] In one embodiment of the present application, the data source acquisition module is used to:

[0141] Get a file name that includes at least one of the following: target system code, target DUS, D model table name;

[0142] When parsing a file, parse the file name and obtain the D model table name, target system code, and target DUS;

[0143] Get the file data source from the configuration table and record the data source so that the data source can be switched when the file is dropped from the library.

[0144] In one embodiment of the present application, the post-processing module:

[0145] After the file is saved in the database, the total number of rows in the file is verified.

[0146] Determine whether all files have been processed successfully. Based on the comparison results, update the status in the file data loading details table to success, failure, or partial success.

[0147] It can be understood that the above-mentioned data migration device can implement each step of the data migration method provided in the above-mentioned embodiment. The relevant explanations about the data migration method are applicable to the data migration device and will not be repeated here.

[0148] Figure 9 This is a schematic diagram of the structure of an electronic device according to an embodiment of the present application. Figure 9 At the hardware level, the electronic device includes a processor and, optionally, an internal bus, a network interface, and memory. The memory may include internal memory, such as high-speed random-access memory (RAM), or non-volatile memory, such as at least one disk drive. Of course, the electronic device may also include other hardware required for its services.

[0149] The processor, network interface, and memory can be interconnected via an internal bus, which can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus, or an EISA (Extended Industry Standard Architecture) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 9 Only one bidirectional arrow is used in the diagram, but this does not mean that there is only one bus or one type of bus.

[0150] The memory is used to store programs. Specifically, the program may include program code, which includes computer operating instructions. The memory may include internal memory and non-volatile memory, and provides instructions and data to the processor.

[0151] The processor reads the corresponding computer program from the non-volatile memory into the internal memory and then runs it, forming a data migration device at the logical level. The processor executes the program stored in the memory and is specifically used to perform the following operations:

[0152] In response to a data file of the old system, determining a reading method of the data file;

[0153] If it is determined that the data file reading method is a file that needs to be converted and verified, then it is stored in the first method;

[0154] If it is determined that the data file reading method is a file that does not require conversion or verification, it is stored in the second method.

[0155] The first method at least includes the conversion and verification of file data, and the second method directly stores the data in the database by copying.

[0156] The above application Figure 2The method performed by the data migration device disclosed in the illustrated embodiment can be applied to a processor or implemented by a processor. The processor may be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method can be completed by hardware integrated logic circuits in the processor or by software instructions. The above processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The various methods, steps, and logic block diagrams disclosed in the embodiments of this application can be implemented or executed. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in conjunction with the embodiments of this application can be directly implemented and executed by a hardware decoding processor, or by a combination of hardware and software modules in the decoding processor. The software module can be located in a storage medium well-known in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, etc. The storage medium is located in the memory, and the processor reads the information in the memory and, in conjunction with its hardware, completes the steps of the above method.

[0157] The electronic device may also perform Figure 2 The method executed by the data migration device in Figure 2 The functions of the illustrated embodiment will not be described in detail in the embodiments of the present application.

[0158] The embodiment of the present application also provides a computer-readable storage medium, which stores one or more programs, wherein the one or more programs include instructions, which, when executed by an electronic device including multiple application programs, can enable the electronic device to execute Figure 2 The method performed by the data migration device in the illustrated embodiment is specifically used to perform:

[0159] In response to a data file of the old system, determining a reading method of the data file;

[0160] If it is determined that the data file reading method is a file that needs to be converted and verified, then it is stored in the first method;

[0161] If it is determined that the data file reading method is a file that does not require conversion or verification, it is stored in the second method.

[0162] The first method at least includes the conversion and verification of file data, and the second method directly stores the data in the database by copying.

[0163] It will be understood by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Thus, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0164] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0165] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0166] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0167] In a typical configuration, a computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0168] Memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0169] Computer-readable media includes permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. The information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory computer-readable media (transitory media), such as modulated data signals and carrier waves.

[0170] It should also be noted that the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, commodity, or apparatus that includes a series of elements includes not only those elements but also other elements not explicitly listed, or includes elements inherent to such process, method, commodity, or apparatus. In the absence of further limitations, an element defined by the phrase "comprises a ..." does not exclude the presence of other identical elements in the process, method, commodity, or apparatus that includes the element.

[0171] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present application may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0172] The foregoing is merely an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application should all be included within the scope of the claims of the present application.

Claims

1. A data migration method, wherein: The method comprises: In response to a data file of the old system, determining a reading method of the data file; If it is determined that the data file reading method is a file that needs to be converted and verified, then it is stored in the first method; If it is determined that the data file reading method is a file that does not require conversion or verification, it is stored in the second method. The first method at least includes the conversion and verification of file data, and the second method directly stores the data in the database by copying.

2. The method according to claim 1, wherein: If it is determined that the data file reading method is a file that needs to be converted and verified, then the data file is stored in the first method, including: For the data file that needs to be converted, parse each row of data and query the conversion rule table preloaded into memory to check whether there are any fields that need to be converted; If yes, then call the method corresponding to the code according to the query conversion rule code to perform data conversion and replace the original value to complete the conversion; Determine verification rules based on analysis documents, formulate verification rule codes and pre-allocate them into the verification rule table; For data files that require quality verification, the migration process queries the pre-loaded verification rule table in memory when parsing each row of data to check whether the current data file contains fields that require verification. The quality verification items include at least one of the following: code value range, date format, and field length. If yes, then call the method corresponding to the code for verification according to the query verification rule code; If the verification passes, continue parsing the next line; If the verification fails, the row will be recorded as a verification failure record and will not be loaded any further, and the remaining rows will continue to be parsed.

3. The method according to claim 2, wherein: If it is determined that the data file reading method is a file that needs to be converted and verified, then the data file is stored in the first method, including: For files that need to be converted and verified in the data file reading mode, the table structure corresponding to the table that needs to be stored is queried by querying the pre-parsed table name and its table structure configuration; Assemble the table structure and the parsed data list into an SQL structured query statement; Then the data source is switched according to the pre-acquired data source and finally falls into the database.

4. The method according to claim 1, wherein: The method further comprises: The files that need to be converted and verified are processed according to the configuration and finally stored in the database; Set the field order to be the same as the data order in the file as the loading table name and table structure configuration of the loading details table; and / or, according to the target system code, the target DUS distinguishes the data source information of the configured full meter; And / or, based on user needs, tables that do not require conversion and verification are configured to be stored in copy mode, and the file data corresponding to the remaining tables are parsed and stored in the database using the conversion and verification method; And / or, before processing the file, configure the conversion rule configuration table and the verification rule configuration table according to requirements.

5. The method according to claim 1, wherein: The method further comprises: Record the abnormal data in the abnormal file by line; Exceptions at the file level during parsing are recorded in the exception information column of the load details table. Exceptions at the data level during parsing are written into the exception file. If an exception occurs during data entry, data rollback or full data entry will be adopted. If you choose the data rollback method, if a row of data fails, the entire batch of data will fail; If you select the "enter as much data as possible" method, the error rows will be skipped and the remaining data will continue to be stored.

6. The method of claim 1, wherein: The method further comprises: Get a file name that includes at least one of the following: target system code, target DUS, D model table name; When parsing a file, parse the file name and obtain the D model table name, target system code, and target DUS; Get the file data source from the configuration table and record the data source so that the data source can be switched when the file is dropped from the library.

7. The method of claim 1, wherein: The method further comprises: After the file is saved in the database, the total number of rows in the file is verified. Determine whether all files have been processed successfully. Based on the comparison results, update the status in the file data loading details table to success, failure, or partial success.

8. A data migration device, wherein: The device comprises: a response module, configured to respond to a data file of the old system and determine a reading method of the data file; A first-mode warehousing module, configured to, if it is determined that the data file reading mode is a file that needs conversion and verification, store it in the warehousing according to the first mode; The second mode storage module is used to store the data file in the second mode if it is determined that the data file reading mode does not require conversion or verification. The first method at least includes the conversion and verification of file data, and the second method directly stores the data in the database by copying.

9. An electronic device comprising: processor; as well as A memory arranged to store computer executable instructions, which when executed cause the processor to perform the method of any one of claims 1 to 7.

10. A computer-readable storage medium storing one or more programs, which, when executed by an electronic device including a plurality of application programs, causes the electronic device to execute the method according to any one of claims 1 to 7.

Images