Database migration methods, apparatus, equipment, and media based on encoding formats
By identifying and adjusting the length of character fields during the database migration process, the problem of insufficient character field capacity in cross-environment database migration was solved, improving the accuracy and efficiency of the migration.
Patent Information
- Application Number
- CN202611516610.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-09-28
- Publication Date
- 2026-11-17
AI Technical Summary
Existing technologies fail to effectively identify and adjust the length of character fields in scripts during database migration across environments. This can lead to insufficient field capacity due to differences in encoding formats, resulting in truncated character data or data writing failures.
By acquiring the script file to be migrated, the data synchronization link mapping table, and the system affiliation table, the original encoding format is identified, the storage length relationship of characters under different encoding formats is determined, the character field definition is adjusted, the target encoding script file is generated, and a link matching file is generated based on the link target table identifier and the system target table identifier.
It enables adaptation of character field lengths during database migration, reducing missing character content and script deployment anomalies, and improving the accuracy and efficiency of database migration.
Smart Images

Figure CN123115092A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of development support technology, and in particular to a database migration method, apparatus, device, and medium based on an encoding format. Background Technology
[0002] As database operating environments are continuously updated, database scripts often face encoding format differences during cross-environment migration. Different encoding formats may use different storage lengths for the same character, and existing migration methods typically only convert the script file's encoding without simultaneously handling the character field lengths within the script. When the character storage length under the target encoding format is greater than that under the original encoding format, the original field length may not be able to accommodate the complete character content, easily causing character data truncation, data write failures, or script deployment anomalies.
[0003] In the fintech business, core accounting, payment settlement, fund clearing, credit management, risk control, and transaction logs typically rely on a large number of stored procedures, triggers, and database scripts. Character fields in these scripts, such as customer names, transaction summaries, payment remarks, clearing instructions, and risk warnings, may experience insufficient capacity after database migration due to changes in encoding formats. Current technologies struggle to uniformly identify and adjust the length of character fields in scattered scripts based on differences in character storage length before and after migration, increasing the risk of missing character content during financial business data migration.
[0004] In the healthcare sector, electronic medical records, laboratory information, prescription management, medical order management, and medical data exchange also involve a large number of database scripts. Fields such as patient name, diagnosis description, examination results, medical order content, and drug instructions typically contain a significant amount of character data. During database migration across environments, differences in encoding formats may render the original field lengths insufficient for the character storage requirements of the target environment. Existing technologies struggle to simultaneously identify script encoding formats and adjust character field lengths, easily leading to incomplete medical record text, laboratory results, or medical order information after migration. Summary of the Invention
[0005] The main objective of this invention is to provide a database migration method, apparatus, device, and storage medium based on encoding format, aiming to solve the technical problem that existing technologies are unable to synchronously adjust the script field structure to address encoding differences during database cross-environment migration, which easily leads to insufficient capacity of character fields after migration.
[0006] To achieve the above objectives, the present invention provides a database migration method based on an encoding format, comprising: Obtain the script file to be migrated, the data synchronization link mapping table, the system affiliation table, the target encoding format, and the output directory; Extract the file header information and content features of the script file to be migrated, identify the original encoding format based on the file header information and the content features, and parse the script file to be migrated according to the original encoding format to generate script content; Determine the character storage length relationship of the same character under the original encoding format and the target encoding format, identify the character field definition in the script content, adjust the field length in the character field definition based on the character storage length relationship, and generate structurally optimized script content; Encode the structure optimization script content according to the target encoding format to generate a target encoding script file; Based on the link target table identifier in the data synchronization link mapping table and the system target table identifier in the system affiliation table, the system affiliation of each data synchronization link in the data synchronization link mapping table is determined, and a link matching file is generated based on the system affiliation. Output the target encoding script file and the link matching file according to the output directory to generate a database migration file set.
[0007] Furthermore, to achieve the above objectives, the present invention provides a database migration apparatus based on an encoding format, comprising: The migration data acquisition module is used to acquire the script file to be migrated, the data synchronization link mapping table, the system affiliation table, the target encoding format, and the output directory; The encoding recognition and parsing module is used to extract the file header information and content features of the script file to be migrated, identify the original encoding format based on the file header information and the content features, and parse the script file to be migrated according to the original encoding format to generate script content; The field structure optimization module is used to determine the character storage length relationship of the same character under the original encoding format and the target encoding format, identify the character field definition in the script content, adjust the field length in the character field definition based on the character storage length relationship, and generate structure-optimized script content. The target encoding generation module is used to encode the structure optimization script content according to the target encoding format and generate a target encoding script file. The link affiliation matching module is used to determine the system affiliation of each data synchronization link in the data synchronization link mapping table based on the link target table identifier in the data synchronization link mapping table and the system target table identifier in the system affiliation table, and to generate a link matching file based on the system affiliation. The migration file output module is used to output the target encoding script file and the link matching file according to the output directory, and generate a database migration file set.
[0008] Furthermore, to achieve the above objectives, the present invention also provides a computer device, the computer device including a memory, a processor, and a database migration program based on an encoding format stored in the memory and executable on the processor, wherein when the database migration program based on the encoding format is executed by the processor, it implements the steps of the database migration method based on the encoding format as described above.
[0009] Furthermore, to achieve the above objectives, the present invention also provides a computer-readable storage medium storing a database migration program based on an encoding format, wherein the database migration program based on the encoding format, when executed by a processor, implements the steps of the database migration method based on the encoding format as described above.
[0010] Beneficial Effects: This invention relates to the field of development support technology and discloses a database migration method, apparatus, device, and medium based on encoding format. The method includes: acquiring a script file to be migrated, a data synchronization link mapping table, a system affiliation table, a target encoding format, and an output directory; identifying the original encoding format of the script file and generating script content; adjusting the character field length according to the character storage length relationship between the original encoding format and the target encoding format to generate structure-optimized script content and a target encoding script file; determining the system affiliation of the data synchronization link based on the link target table identifier and the system target table identifier, generating a link matching file; and outputting the target encoding script file and the link matching file to generate a database migration file set. This invention can be applied to business scenarios such as fintech and healthcare. By identifying the encoding format and adjusting the character field length according to the character storage length relationship, the script fields meet the storage requirements of the target encoding environment; by determining the system affiliation of the data synchronization link through target table identifier matching, the method reduces insufficient character field capacity and improves the accuracy and processing efficiency of database migration. Attached Figure Description
[0011] The present invention will be further described below with reference to the accompanying drawings and embodiments. In the accompanying drawings: Figure 1 This is a schematic diagram of an application environment for a database migration method based on encoding format according to an embodiment of the present invention; Figure 2 This is a flowchart illustrating an embodiment of the database migration method based on encoding format of the present invention; Figure 3 for Figure 2 A detailed flowchart of step S10; Figure 4 for Figure 2 A detailed flowchart of step S20; Figure 5 for Figure 2A detailed flowchart of step S30; Figure 6 for Figure 2 A detailed flowchart of step S40; Figure 7 for Figure 2 A detailed flowchart of step S50; Figure 8 for Figure 2 A detailed flowchart of step S60; Figure 9 This is a schematic diagram of the functional modules of a preferred embodiment of the database migration device based on encoding format of the present invention; Figure 10 This is a schematic diagram of the structure of a computer device according to an embodiment of the present invention; Figure 11 This is another structural schematic diagram of a computer device according to one embodiment of the present invention. Detailed Implementation
[0012] It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of the invention.
[0013] The database migration method based on encoding format provided in this invention can be applied to, for example... Figure 1 In this application environment, the client communicates with the server via a network. The server can obtain the script file to be migrated, the data synchronization link mapping table, the system affiliation table, the target encoding format, and the output directory from the client; identify the original encoding format of the script file and generate script content; adjust the character field length according to the character storage length relationship between the original encoding format and the target encoding format, generating structure-optimized script content and the target encoding script file; determine the system affiliation of the data synchronization link based on the link target table identifier and the system target table identifier, generating a link matching file; output the target encoding script file and the link matching file, generating a database migration file set. This invention can be applied to business scenarios such as fintech and healthcare. By identifying the encoding format and adjusting the character field length according to the character storage length relationship, the script fields meet the storage requirements of the target encoding environment; by determining the system affiliation of the data synchronization link through target table identifier matching, the insufficient character field capacity is reduced, improving the accuracy and processing efficiency of database migration. The client can be, but is not limited to, various personal computers, laptops, smartphones, tablets, and portable wearable devices. The server can be implemented using a standalone server or a server cluster composed of multiple servers. The invention will be described in detail below through specific embodiments.
[0014] Please see Figure 2 , Figure 2This is a flowchart illustrating an embodiment of the database migration method based on encoding format provided by the present invention. It should be noted that although a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than that shown here.
[0015] like Figure 2 As shown, the database migration method based on encoding format proposed in this invention includes the following steps: S10: Obtain the script file to be migrated, data synchronization link mapping table, system ownership table, target encoding format, and output directory; In this embodiment, the database script is read based on the file location recorded in the migration task to form a script file to be migrated. The script may include stored procedures, triggers, functions, and table structure files. The read content includes the file name, file type, storage location, and file content, and different batches are distinguished by the migration task identifier.
[0016] The data synchronization mapping table records the association between data synchronization tasks and target tables, while the system affiliation table records the affiliation of business systems with target tables. Both types of tables can originate from spreadsheets, structured files, or database records, and each retains the data synchronization task identifier, business system identifier, and target table identifier during retrieval.
[0017] The target encoding format can be read from task parameters, target database configuration, or deployment requirements and converted into a unified encoding format name. The output directory is determined based on the configured storage path; if the directory does not exist, it is created; if the directory exists, the write status is checked. All data items are associated with the same migration task identifier to avoid mixing data from different migration batches.
[0018] When scripts and tables are stored together, a task configuration file is used to record the script directory, the location of the two types of tables, the target encoding format, and the output path. The corresponding data is read according to each location, and the source information is written into the same task record.
[0019] When data is stored in a scattered manner, resource identifiers are used to locate different storage areas, scripts are read through file interfaces, two types of tables are read through data query interfaces, target encoding formats are read through environment configuration, and the output directory is determined based on the result storage area identifier.
[0020] In the fintech business, the system reads account, payment, and clearing database scripts, records synchronization tasks, target tables, and two types of tables belonging to different business systems, and associates the target encoding format and output directory to avoid data mixing between different financial business migration batches.
[0021] In the healthcare business, the system reads scripts from health records, testing services, and drug distribution databases, along with corresponding synchronization mapping data and system attribution data, to ensure that the source of the data objects included in the current migration batch is clear.
[0022] This embodiment obtains the script file to be migrated, two types of tables, target encoding format, and output directory, and establishes a unified task association, which can reduce script omissions, table mismatches, mixed encoding parameters, and incorrect storage locations, thereby improving the integrity of migration input data.
[0023] S20, extract the file header information and content features of the script file to be migrated, identify the original encoding format based on the file header information and the content features, and parse the script file to be migrated according to the original encoding format to generate script content; In this embodiment, the byte data of the script file to be migrated is read, and the encoding markers, byte arrangement, and file type information are identified from the beginning area of the file to form the file header information. For files that do not carry explicit encoding markers, the byte distribution, character boundaries, line breaks, and script delimiters are extracted from the non-empty content area to form content features.
[0024] Based on the file header information, the possible encoding range is determined. Content features are used to check the integrity of character reconstruction and script structure under each encoding format. The encoding format that can stably reconstruct the file content is selected as the original encoding format. File bytes are converted into character sequences according to the original encoding format, preserving statement order, newlines, comments, and delimiters to form readable script content.
[0025] When the script file carries explicit encoding markers, the original encoding format is determined using file header recognition, and the file content is then decoded and checked. If no abnormal characters are found in the decoding result and the script statement structure is complete, the decoded character sequence is organized into the script content.
[0026] When the script file does not carry encoding markers, the file content is read using a multi-segment sampling method. The sampled bytes are restored according to the candidate encoding formats. The original encoding format is determined based on the number of abnormal characters, the integrity of character boundaries, and the integrity of the statement structure. Then, the complete file is parsed.
[0027] In the fintech business, stored procedures and triggers in account management and payment settlement databases contain Chinese content such as customer names, transaction summaries, and settlement instructions. By reading the file header and content byte characteristics of the script file, the original encoding format is determined, and then the statements, comments, and delimiters are restored to avoid garbled characters or broken statements in financial business scripts.
[0028] In the healthcare sector, health records and laboratory data scripts within regional health data platforms contain diagnostic descriptions, test results, and medication information. By identifying the encoding format through file header information and content features, and parsing the file according to the identification results, the integrity of Chinese field names and script statements can be maintained.
[0029] This embodiment identifies the original encoding format by simultaneously utilizing the file's starting area information and file content features, and restores the script characters according to the identification results. This reduces Chinese character garbled text, missing characters, and script structure damage caused by encoding judgment errors, thereby improving the accuracy of script content parsing.
[0030] S30, determine the character storage length relationship of the same character under the original encoding format and the target encoding format, identify the character field definition in the script content, adjust the field length in the character field definition based on the character storage length relationship, and generate structure-optimized script content; In this embodiment, the byte length occupied by the same character under the original encoding format and the target encoding format is read from the character encoding attribute to establish the character storage length relationship. For encoding formats with variable byte lengths, the target character storage length can be determined according to the character range involved in the script, or the maximum byte length within the corresponding character range can be used to avoid the adjusted field still being unable to accommodate the complete character.
[0031] The scan script content contains character field type identifiers, field names, and length parameters to identify character field definitions, excluding comments, string content, and non-field declaration areas. Character field definitions include character types and their length values, such as fixed-length character fields and variable-length character fields, and are not limited to specific database type names.
[0032] The number of characters a field can hold is determined based on the original field length and the original character storage length. Then, the adjusted field length is determined based on the target character storage length. The adjustment result is rounded up while ensuring it is not less than the original field length. Only the length parameter in the character field definition is replaced, while the field name, type, constraints, and script arrangement order are preserved. This generates the structure optimization script content.
[0033] When the script statement structure is complete, lexical scanning and syntax node recognition are used to locate character field definitions. The type and length parameters are read from the field declaration node, the length parameter is updated based on the character storage length relationship, and then the updated field declaration is written back to its original position. This method is suitable for character field declarations with clear structural boundaries in table structure scripts, stored procedure scripts, and trigger scripts.
[0034] In situations where script sources are scattered and statement formats are inconsistent, content partitioning and tag matching are used to identify character field definitions. Comments and string regions are masked, and field declarations are located in the remaining content based on character type identifiers, bracket boundaries, and length values. The length values are then adjusted according to character storage length relationships. This method is applicable to script files exported from different database tools.
[0035] In the fintech business, the customer name, transaction summary, and settlement description fields in payment settlement and fund clearing scripts use the length settings from the source environment. Adjusting the field lengths based on the differences in character storage lengths before and after the migration can reduce the risk of truncation of Chinese transaction data after it is written to the target database.
[0036] In the healthcare business field, the scripts of regional health record platforms include fields for diagnostic descriptions, test results, and drug instructions. Identifying character field definitions and adjusting length parameters allows these fields to maintain their original character capacity under the target encoding environment.
[0037] This embodiment establishes the storage length relationship of the same character before and after migration, and adjusts the character field length in the script accordingly. This maintains the original character capacity of the field, reduces character truncation, insufficient field capacity, and abnormal script deployment caused by encoding differences, and preserves the original script structure.
[0038] S40, Encode the structure optimization script content according to the target encoding format to generate a target encoding script file; In this embodiment, the target encoding format is used to determine the character and byte conversion method when writing the structure optimization script content to a file. Its source can be the target database environment configuration, script deployment requirements, or migration task parameters. During the encoding process, the structure optimization script content is read according to the character order, and characters, newlines, whitespace characters, and script delimiters are converted into byte data corresponding to the target encoding format, while maintaining the field definitions, statement levels, comment positions, and character order.
[0039] The structure optimization script already includes the adjusted character field definitions. The encoding operation only changes the byte representation of the character content in the file, without altering the field name, field type, field length, constraint content, or statement structure. For characters that the target encoding format cannot represent, the character position can be recorded and writing to the corresponding file can be stopped to avoid overwriting the original content with replacement characters.
[0040] The target encoding script file is formed by writing the converted byte data into a new script file. The file name and file type can be the same as the script file to be migrated, or a distinguishing identifier can be added according to the migration batch. The file after writing should retain the complete script content so that the target database environment can read the Chinese characters, field definitions, and database statements in the file according to the target encoding format.
[0041] When the content of a single script is small, a holistic encoding method is used to read the complete character sequence, convert all characters into byte data according to the target encoding format, and then write the byte data into a new script file all at once. This method reduces segment boundary checks and is suitable for stored procedures, functions, and independent triggers.
[0042] When dealing with large script volumes or numerous files, a segmented encoding method is used to divide the content into segments based on character boundaries. Each content segment is then converted and written to the script file in its original order. Content segments must not be broken within multi-byte characters, and the completed position is continuously recorded during the writing process to ensure the continuity of the file content after each segment is written.
[0043] For target encoding formats that include header encoding markers, the corresponding markers can be written before writing character byte data; for target environments that do not require header encoding markers, the byte data corresponding to the script content can be written directly. Separate write objects should be created for different script files to avoid writing the byte content of one file to another.
[0044] In the fintech business field, account management, payment settlement, and fund clearing scripts are converted into target coded script files. During the encoding process, customer names, transaction summaries, clearing instructions, and field definitions are preserved, enabling the target database environment to fully read the Chinese content in the financial business scripts.
[0045] In the healthcare business, health records, testing services, and drug distribution scripts are converted into target-encoded script files, maintaining the character order of diagnostic descriptions, test results, drug names, and script statements, thus reducing the possibility of character anomalies when the target environment reads the scripts.
[0046] This embodiment optimizes the script content by converting the structure according to the target encoding format, and maintains the character order, field definitions, and statement structure during the conversion and writing process. This enables the generated script file to meet the encoding requirements of the target database environment, reducing garbled characters, missing characters, and script reading anomalies, and improving the integrity of the script content.
[0047] S50, based on the link target table identifier in the data synchronization link mapping table and the system target table identifier in the system affiliation table, determine the system affiliation of each data synchronization link in the data synchronization link mapping table, and generate a link matching file based on the system affiliation; In this embodiment, the data synchronization task identifier and target table identifier are read from the data synchronization mapping table, and the system identifier and system target table identifier are read from the system ownership table. The target table identifier can consist of the database name, schema name, and table name, or it can use a combination of characters that can distinguish the data table. During the reading process, the correspondence between each identifier in the table is preserved to avoid mismatch between the target table and the data synchronization task or business system.
[0048] The format of the two types of target table identifiers is standardized, unifying the use of uppercase and lowercase letters, whitespace characters, separators, and qualifiers. The standardized identifiers are then compared. When the identifiers match, the corresponding data synchronization task identifier is associated with the system identifier. When a data synchronization task involves multiple target tables, the system affiliation of each target table is determined separately. When multiple target tables belong to different systems, their respective affiliation records are retained.
[0049] The matching file is used to store the data synchronization task identifier, target table identifier, system identifier, and matching results. The file can be a spreadsheet, delimited text, or structured data file, and the records are arranged according to the data synchronization task identifier or system identifier, so that the system affiliation of each data synchronization task can be read independently.
[0050] When two types of tables use the same target table naming convention, the system affiliation is determined by directly comparing the target table identifiers. The target table identifiers from both types of tables are read, leading and trailing whitespace characters are removed and their format is standardized, records with matching identifiers are associated, and then the data synchronization task identifier, target table identifier, and system identifier are written to the matching file.
[0051] When two types of tables use different target table naming conventions, segmented matching is used to determine system affiliation. The target table identifier is split into database name, schema name, and table name. For records lacking a database name, the schema name is compared with the table name. For records with qualifiers, the qualifiers are removed before comparison. After matching is complete, the corresponding system identifiers are collected for each data synchronization task, and a matching file is generated.
[0052] When a single data synchronization task corresponds to multiple target tables, the results are saved using a table-by-table attribution record method. Each target table records its corresponding system identifier to avoid losing system attribution differences after multiple target tables are merged.
[0053] In the fintech business, a data synchronization mapping table records the data synchronization tasks corresponding to account balances, payment records, and fund clearing data. A system attribution table records the relevant target tables, which belong to the account management system, payment settlement system, and fund clearing system, respectively. The system attribution of each data synchronization task is determined by comparing the target table identifiers, and the task identifier, target table identifier, and financial business system identifier are written into a matching file.
[0054] In the healthcare business domain, a data synchronization mapping table records the data synchronization tasks corresponding to health records, test results, and drug distribution data. A system attribution table records the relevant target tables, which are respectively assigned to the health record platform, the test service platform, and the drug distribution platform. The association between data synchronization tasks and platform identifiers is established based on the target table identifiers, and the attribution results are written to a matching file.
[0055] This embodiment compares the target table identifiers in the two types of tables and establishes the association between the data synchronization task and the business system. This reduces the attribution identification errors caused by differences in target table naming. Furthermore, it uses matching files to centrally store the attribution of tasks, target tables, and systems, thereby improving the integrity and readability of attribution information.
[0056] S60, output the target encoding script file and the link matching file according to the output directory to generate a database migration file set.
[0057] In this embodiment, the output directory is used to define the storage location of the target encoding script file and the matching file. It can be a local directory, a shared storage directory, or a logical storage area with a directory hierarchy. Before output, the path separator, directory hierarchy, and access location are sorted, and the target location is checked to ensure that the file can be written to, thus avoiding writing files to the wrong directory.
[0058] The target encoding script file contains script content saved according to the target encoding format, and the matching file contains the data synchronization task, target table, and system affiliation data. The write locations are determined based on the filenames and output directories of both files, and the file content is written to the corresponding locations while maintaining the file type, character encoding, and content order.
[0059] The database migration file set is used to aggregate target encoding script files and matching files generated in the same migration batch. It can be represented as a combination of files in the output directory, or as a logical set recording the names and storage locations of each file. When multiple target encoding script files are stored according to their original directory relationships, the relative directory hierarchy is preserved, allowing the file source and deployment location to be distinguished.
[0060] When outputting files to a local or shared directory, a hierarchical directory structure is used to save migration files. Script file subdirectories and matching file subdirectories are created based on the output directory. Target encoding script files are written to the script file subdirectory with their original filenames, and matching files are written to the matching file subdirectory. These are then aggregated into a database migration file set according to the same migration batch.
[0061] When the output directory uses a logical storage path, the storage location is generated by combining the batch identifier and the file name. The target encoding script file and the matching file are written to their respective logical locations, and the file name, file type, and storage location are recorded. Then, a database migration file set is formed based on the recorded content.
[0062] When multiple script files have the same file name, use relative directories or business directories to distinguish the output location, avoid files with the same name from overwriting each other, and at the same time preserve the directory relationship of each script file before migration.
[0063] In the fintech business field, account management, payment settlement, fund clearing and risk control scripts are written into the financial business script directory, and the matching files that record synchronization tasks, target tables and business systems are written into the belonging data directory. Then, they are collected into a database migration file set according to migration batches, which makes it easier to distinguish the deployment files of different financial business systems.
[0064] In the healthcare business sector, health records, testing services, and drug distribution scripts are written into their respective business directories. The target table and platform-owned data are written into the matching file directory, and a database migration file set is formed in the same output directory, so that the migration files of different healthcare businesses maintain a clear storage hierarchy.
[0065] This embodiment reduces file misplacement, name overwriting, and batch mixing by writing target encoding script files and matching files according to the output directory and aggregating files from the same migration batch into a database migration file set, thus ensuring that the storage location and ownership of migration files remain clear.
[0066] In one embodiment, such as Figure 3 As shown, step S10 above includes: S101, Obtain the migration task configuration file, and read the script directory, data synchronization link mapping table path, system affiliation table path, target encoding identifier, output path and file filtering conditions from the migration task configuration file; S102, traverse the directory hierarchy in the script directory, filter script files based on the file filtering conditions, record the file paths of the filtered script files, and read the filtered script files based on the file paths to obtain the script files to be migrated; S103, Read the link table file based on the data synchronization link mapping table path, and when the link table file contains a data synchronization link identifier field and a link target table identifier field, determine the link table file as a data synchronization link mapping table; S104, Read the attribution table file based on the system attribution table path, and when the attribution table file contains a system identifier field and a system target table identifier field, determine the attribution table file as a system attribution table; S105, Based on the target encoding identifier, determine the encoding format name from the migration task configuration file, and determine the encoding format name as the target encoding format; S106, detect whether the directory pointed to by the output path exists. If the directory pointed to by the output path exists, determine the directory pointed to by the output path as the output directory. If the directory pointed to by the output path does not exist, create a directory according to the output path and determine the created directory as the output directory.
[0067] In this embodiment, the migration task configuration file uses a parsable data structure to store the resource locations and filtering parameters required for this database migration. The content includes the script directory, data synchronization mapping table path, system-owned table path, target encoding identifier, output path, and file filtering conditions. The configuration content can be stored as a combination of fields and values. After reading, directory access parameters, table access parameters, encoding parameters, and output parameters are generated respectively, ensuring that subsequent file reads use the same batch of configuration data.
[0068] The script directory is used to locate the file storage area to be checked. Directory entries are read level by level, and subdirectories within each entry are expanded. For each file entry, the file name, file type, and file path are read. File filtering criteria can include extensions, filename formats, file content markers, and exclusion markers. Database script files are retained based on these criteria, while temporary files, backup files, and irrelevant files are excluded. After filtering, the complete file path of each script file is recorded. Then, the file's byte content is read based on the file path to obtain the script files to be migrated. Recording the file paths preserves the relationship between the script files and their original directory locations and avoids confusion caused by files with the same name.
[0069] The data synchronization mapping table path is used to locate the table file that stores the relationship between data synchronization tasks and target tables. After reading the table file, the header fields are checked. If both a data synchronization task identifier field and a target table identifier field exist in the table, it confirms that the table has a data structure describing the correspondence between the data synchronization tasks and target tables, and the table content is used as the data synchronization mapping table. Field checks can be performed by field name, field position, or preset field mapping relationships. Tables with blank headers, duplicate headers, or missing necessary fields are not included in the current migration data.
[0070] The system attribution table path is used to locate the table file that stores the attribution relationship between business systems and target tables. After reading the table file, the system identifier field and the system target table identifier field are checked. If both fields exist, the table content is used as the system attribution table. The system identifier field is used to distinguish different business systems, and the system target table identifier field is used to locate the data tables managed by each business system. The combination relationship between these two types of fields in the same data row is fully preserved.
[0071] The target encoding identifier is used to determine the encoding format required for script files from the migration task configuration file. The encoding identifier can be represented by an encoding name, encoding number, or configuration tag. After being read, it is mapped to a unified encoding format name, and this encoding format name is used as the target encoding format. This conversion eliminates differences in how different configurations represent the same encoding format.
[0072] The output path is used to determine the storage location of the migration results. After reading the output path, it checks whether the corresponding directory already exists. If the directory exists, it is directly used as the output directory; otherwise, the directory is created according to the path hierarchy, and the created directory is used as the output directory. After the directory is confirmed, write permissions and path validity can be checked to avoid subsequent files failing to be saved due to missing or unwritable directories.
[0073] This embodiment reads the script directory, two types of table paths, target encoding identifier, output path, and file filtering conditions from the same migration task configuration file, and performs script filtering, table field checking, encoding name determination, and output directory confirmation respectively. This reduces problems such as missing scripts, incorrect table selection, mixed encoding parameters, and invalid output locations, and improves the integrity and configuration consistency of migration input data.
[0074] In one embodiment, such as Figure 4 As shown, step S20 above includes: S201, Read the file header byte sequence and file content byte sequence of the script file to be migrated, identify the encoding marks and byte arrangement features in the file header byte sequence, and generate file header information based on the encoding marks and the byte arrangement features; S202, Select a content segment from the file content byte sequence, identify the byte distribution features, newline mark features and script delimiter features in the content segment, and generate content features based on the byte distribution features, newline mark features and script delimiter features; S203, obtain a set of candidate encoding formats, decode the file content byte sequence according to each candidate encoding format in the set of candidate encoding formats, record the position of invalid bytes and the position of replaced characters, identify the text structure features of the decoded content, and generate candidate decoding results; S204, determine the candidate decoding results that do not record the position of the invalid byte and the position of the replacement character as valid candidate decoding results, match the text structure features in the valid candidate decoding results with the content features to generate a structure matching result, determine the target candidate decoding result from the valid candidate decoding results based on the structure matching result and the file header information, and determine the candidate encoding format corresponding to the target candidate decoding result as the original encoding format; S205, decode the file content byte sequence according to the original encoding format to generate a decoded character sequence; S206, the decoded character sequence is divided into script content segments based on the newline marker feature and the script separator marker feature, and the script content segments are arranged according to the character arrangement order in the decoded character sequence to generate script content.
[0075] In this embodiment, the byte data of the script file to be migrated is read, the starting area of the file is divided into a header byte sequence, and the byte area carrying the script text is divided into a content byte sequence. The encoding markers in the header byte sequence indicate the character encoding format used by the file, and the byte arrangement characteristics reflect the arrangement direction of the multi-byte data. The encoding markers, arrangement direction, and corresponding byte positions are associated and saved to form the file header information. When the file does not carry explicit encoding markers, the file header information records the missing encoding marker status and identifiable arrangement characteristics for subsequent comprehensive judgment.
[0076] Content segments are selected from the beginning, middle, or non-empty areas of the file's byte sequence, avoiding reliance solely on comments or whitespace near the file header to determine the encoding format. Byte distribution characteristics include single-byte value distribution, multi-byte combination patterns, the position of the most significant byte, and consecutive zero-byte intervals. Line break characteristics include line break byte combinations, line break positions, and consecutive line break patterns. Script delimiter characteristics include statement termination characters, batch delimiters, and byte combinations near delimiters. These characteristics are then associated and saved according to byte position to form the content features.
[0077] The candidate encoding format set contains various character encoding forms that can be used to reconstruct file content. The file content byte sequence is decoded according to each candidate encoding format, and each byte combination is checked to see if it can be mapped to a valid character. Byte positions where character mapping cannot be completed are recorded as invalid byte positions, and positions where substitute characters are generated during decoding are recorded as replacement character positions. The decoded character content is then analyzed for keyword continuity, bracket closure, comment boundaries, string boundaries, newline positions, and script separator positions to form text structure features. Each candidate encoding format, its corresponding decoded content, abnormal positions, and text structure features together constitute the candidate decoding result.
[0078] Candidate decoding results without invalid byte positions or replacement character positions are retained as valid candidate decoding results. The line break positions, separator positions, and character arrangement in the text structure features are matched with the byte position relationships in the content features to form a structure matching result. When the file header information contains explicit encoding markers, candidate decoding results that are consistent with the encoding markers and meet the structure matching requirements are selected first; when encoding markers are missing or multiple valid candidate decoding results exist, the target candidate decoding result is determined based on byte arrangement consistency, text structure integrity, and structure matching degree. The candidate encoding format used by the target candidate decoding result is determined as the original encoding format.
[0079] The complete file content byte sequence is re-decoded according to the original encoding format, restoring the byte data to a decoded character sequence with defined character boundaries. Re-decoding avoids directly using partial results of content segments, ensuring that script statements, comments, whitespace characters, and delimiters in the file are all included in the parsing scope.
[0080] Based on the characteristics of line break markers and script separator markers, content boundaries are determined and script content segments are divided in the decoded character sequence. Each script content segment is arranged according to the character order in the decoded character sequence, preserving the original line breaks, comments, statement separators, and character positional relationships to form the script content.
[0081] This embodiment identifies the original encoding format by combining file header information, byte features of content segments, and decoding results under different encoding formats. This reduces encoding misjudgments caused by relying solely on file header markers. By recording invalid byte positions, replacing character positions, and checking the integrity of the text structure, encoding formats that cannot fully restore the script content can be eliminated. Parsing the complete file according to the determined original encoding format while preserving line breaks and script delimiters reduces garbled characters, missing characters, and broken statement structures, thereby improving the accuracy and completeness of script content parsing.
[0082] In one embodiment, such as Figure 5 As shown, step S30 above includes: S301, Obtain character encoding attribute data, read the original character storage length and the target character storage length of the same character under the original encoding format and the target character storage length under the target encoding format from the character encoding attribute data, and generate a character storage length relationship based on the original character storage length and the target character storage length; S302, obtain a set of character field types, identify field type identifiers from the script content, and determine the character field definition location based on the field type identifiers that match the set of character field types; S303, based on the character field definition position, identify the character field definition using byte length semantics from the script content, read the field length from the character field definition, and determine the field length as the original field length; S304, determine the field character capacity based on the original character storage length and the original field length in the character storage length relationship, determine the candidate field length based on the target character storage length and the field character capacity in the character storage length relationship, determine the field length with the larger value between the candidate field length and the original field length as the target field length, and generate field length adjustment data; S305, based on the field length adjustment data, replace the original field length in the character field definition with the target field length to generate the adjusted character field definition; S306, according to the character field definition position, replace the character field definition in the script content with the adjusted character field definition to generate structure-optimized script content.
[0083] In this embodiment, character encoding attribute data records the byte representation and storage length of characters under different encoding formats. For fixed-length encoding, the number of bytes occupied by the same character in the original and target encoding formats can be directly read; for variable-length encoding, the storage length can be read according to the actual characters, character categories, or applicable character ranges involved in the script, and the maximum storage length within that character range can be used if necessary. The original character storage length and the target character storage length are associated with the same character, making the storage occupancy before and after migration comparable.
[0084] The character field type set contains the character types for which the field capacity needs to be checked, including fixed-length character types, variable-length character types, and other field types with character storage attributes. The scan script content includes type identifiers, field names, bracket boundaries, and length parameters. Based on the matching results between the type identifiers and the character field type set, the field declaration area is located. During the identification process, field declarations, comment content, and string content are distinguished to avoid mistaking identical text in comments or statement parameters for character field definitions.
[0085] The character field definition location is used to record the character range of the type identifier and length parameter in the script content. Based on this location, the field length semantics are read, and only character field definitions that express field capacity in bytes are retained. Field definitions that express capacity in characters are not involved in length conversion to avoid repeatedly expanding field capacity. The field length is read from the length parameter and used as the original field length before migration.
[0086] The field character capacity represents the number of characters a character field can hold in its original encoding format. The field character capacity is determined based on the original field length and the original character storage length. Then, based on the field character capacity and the target character storage length, the candidate field length required to maintain the same character capacity after migration is determined. If the conversion result contains a non-integer value, it is rounded up to avoid discarding storage space insufficient for a complete character.
[0087] The candidate field length is compared with the original field length, and the field length with the larger value is selected as the target field length. This selection method increases the length when the encoding conversion requires expanding the field, and retains the original field length when the conversion result is small, avoiding the reduction of field capacity due to migration. The field length adjustment uses the character field definition position, original field length, and target field length of the data record to clarify the scope of modification for each field.
[0088] Adjust the data based on field lengths, replacing the original field lengths in the character field definitions with the target field lengths, while keeping the field names, field types, default values, constraint content, and statement arrangement unchanged. After replacing all character field definitions, combine the unchanged content and the adjusted character field definitions according to the original character order to form the structure optimization script content.
[0089] This embodiment can deduce the field length required for a character field to maintain its original character capacity in the target encoding environment by comparing the storage length of the same character under the encoding format before and after migration. By identifying the character field definition that adopts byte length semantics and selecting a target field length that is not less than the original field length, the field capacity shortage and character content truncation caused by encoding differences can be reduced, while avoiding unnecessary field reduction and maintaining the integrity of field names, constraint content and statement structure in the script.
[0090] In one embodiment, such as Figure 6 As shown, step S40 above includes: S401, Obtain target encoding parameter data corresponding to the target encoding format, and read character encoding mapping data from the target encoding parameter data; S402, Read each character according to the character arrangement order in the structure optimization script content to generate a character sequence to be encoded; S403, based on the character encoding mapping data, convert each character in the character sequence to be encoded into character encoding byte data; S404, Collect the character encoding byte data corresponding to each character according to the character arrangement order in the character sequence to be encoded, and generate the target file byte sequence; S405, decode the target file byte sequence according to the target encoding format to generate verification script content, compare the verification script content with the structure optimization script content according to the character position, and generate encoding verification result; S406, when the encoding verification result indicates that the content of the verification script is consistent with the content of the structure optimization script, read the file name and file type from the script file to be migrated, generate a target file identifier based on the file name and the file type, create a script file to be written according to the target file identifier, write the target file byte sequence into the script file to be written, and generate a target encoding script file.
[0091] In this embodiment, the target encoding parameter data is determined according to the target encoding format, recording the conversion relationship between characters and byte combinations, the length of the character encoding unit, and the arrangement of multi-byte characters. Character encoding mapping data is read from the target encoding parameter data and is used to convert the characters in the structure optimization script content into byte data recognizable by the target encoding format. For characters using variable-length encoding, different characters can be mapped to different numbers of bytes; during the conversion process, byte combinations are generated according to the mapping relationship corresponding to each character.
[0092] Characters are read according to the character order in the structure optimization script content, preserving the original positions of field definitions, script statements, comments, whitespace characters, newline characters, and separator characters to form a sequence of characters to be encoded. This sequence is read in whole-character units, avoiding splitting content from within multi-byte characters, ensuring each character has an independent encoding conversion boundary.
[0093] Based on the character encoding mapping data, the target byte representation of each character in the character sequence to be encoded is queried, and one or more bytes obtained from the query are recorded as the character encoding byte data corresponding to that character. The character encoding byte data is associated with the character position, so that the byte data of adjacent characters can be arranged in the original character order, while maintaining the statement structure and character order in the script content.
[0094] The character-encoded byte data is sequentially assembled according to the character positions in the character sequence to be encoded, forming a continuous target file byte sequence. During the assembly process, the byte order within the character-encoded byte data is not changed, nor are the byte positions corresponding to different characters swapped, thus enabling the target file byte sequence to be restored to character content consistent with the structure optimization script content according to the target encoding format.
[0095] The target file's byte sequence is decoded in reverse using the target encoding format to obtain the verification script content. This verification script content is then compared character by character with the structure optimization script content, checking for consistency in character content, number of characters, and character order. An encoding verification result is generated based on the comparison results. Character position comparison can identify character substitutions, missing characters, byte reordering, and content additions that occur during the encoding process, avoiding judgment of the encoding result solely based on file length.
[0096] If the encoding verification result indicates that the two scripts are identical, the file name and file type are read from the script file to be migrated. The file name is used to maintain the identification relationship between the target file and the source file, and the file type is used to preserve the format attributes and extensions of the script file. A target file identifier is formed based on the file name and file type, and a script file to be written is created according to the target file identifier, so that different source files have a clear writing target.
[0097] The target file's byte sequence is written to the script file to be written, with the writing order matching the byte sequence of the target file. The resulting file content is saved in the target encoding format, with the file name and file type matching the target file identifier, thus generating the target-encoded script file.
[0098] This embodiment reads the target encoding parameter data and completes the character-to-byte conversion according to the character arrangement order, which can maintain the script statements, field definitions, and character position relationships. By reverse decoding the target file byte sequence and comparing it character by character with the structure optimization script content, character substitution, missing, and out-of-order can be detected. After verification, the script file is created and written, which can reduce garbled characters, content corruption, and file type confusion caused by encoding conversion errors, and improve the accuracy and integrity of the target encoding script file.
[0099] In one embodiment, such as Figure 7 As shown, step S50 above includes: S501, Read the data synchronization link identifier and the link target table identifier from the data synchronization link mapping table, and generate link table association data based on the data synchronization link identifier and the link target table identifier; S502, Read the system identifier and system target table identifier from the system attribution table, and generate system table association data based on the system identifier and system target table identifier; S503, unify the case, whitespace, and qualifier forms of the link target table identifier in the link table association data and the system target table identifier in the system table association data, and generate standard link target table identifier and standard system target table identifier; S504, compare the standard link target table identifier with the standard system target table identifier, and generate a target table matching result based on the matching standard link target table identifier and standard system target table identifier; S505, based on the target table matching result, determine the matching link table association data and system table association data, read the data synchronization link identifier and link target table identifier from the matching link table association data, and read the system identifier and system target table identifier from the matching system table association data, establish an association between the data synchronization link identifier and the system identifier, determine the system affiliation of each data synchronization link, and generate link affiliation data based on the data synchronization link identifier, the link target table identifier, the system identifier, the system target table identifier, and the system affiliation; S506, Generate a link matching file based on the link ownership data.
[0100] In this embodiment, the data synchronization task identifier and target table identifier are read from the data synchronization mapping table, and the association between the two identifiers in the same data row is preserved. The data synchronization task identifier is used to distinguish different data transmission configurations, and the target table identifier is used to locate the database table to which the data is written. The target table identifier can consist of the database name, schema name, and table name, or it can be composed of a combination of fields that can uniquely locate the data table within the current migration scope. The read results are organized according to the data synchronization task identifier to form synchronization table association data, so that each data synchronization task includes one or more target tables with a clear source.
[0101] The system identifier and system target table identifier are read from the system attribution table, and the attribution relationship within the same data row is retained. The system identifier is used to distinguish the business systems that manage or use the target table, and the system target table identifier is used to locate the data table belonging to that business system. When a system contains multiple target tables, the association between each target table and the system identifier is recorded separately; when the same target table is associated with multiple systems, multiple system attribution records are retained to form system table association data.
[0102] The target table identifiers in the two types of tables may not be directly comparable due to differences in character case, whitespace, or qualifier formats. To address this, the letter formats in the target table identifiers are standardized, leading and trailing whitespace characters are removed, and whitespace between identifier components is corrected. Simultaneously, the qualifier formats for the database name, schema name, and table name are standardized. After formatting, standard synchronization target table identifiers and standard system target table identifiers are created. The original identifiers and their associated data are retained to avoid altering the original record content during formatting.
[0103] The standard synchronization target table identifier is compared item by item with the standard system target table identifier. When the database name, schema name, and table name contained in the two standard identifiers are the same, a target table matching record is generated, and the positions of the matching parties in the synchronization table association data and the system table association data are recorded. For identifiers that only use the table name, if the data table can be uniquely located within the current migration scope, the comparison can be completed based on the table name; if tables with the same name appear, the database name or schema name needs to be used to distinguish them.
[0104] Based on the target table matching records, locate the associated data of the corresponding synchronization table and system table, and read the data synchronization task identifier, original target table identifier, system identifier, and system target table identifier. Establish an association between the data synchronization task identifier and the system identifier to determine the system affiliation of each data synchronization task. When a single data synchronization task contains multiple target tables belonging to the same system, they are grouped into one system affiliation; when the target tables belong to different systems, the corresponding system identifier and target table identifier are retained separately. The affiliation data is formed by the data synchronization task identifier, the two types of target table identifiers, the system identifier, and the system affiliation, enabling the standard identifier matching results to be traced back to the original records in the two types of tables.
[0105] A matching file is generated based on the attribution data. The matching file records the data synchronization task identifier, target table identifier, system identifier, and system attribution. It can also record the matching status and reasons for non-matches. The file can be a spreadsheet, delimited text, or structured data file, and the records are arranged according to the data synchronization task identifier, allowing each data synchronization task and its target table attribution to be read independently.
[0106] This embodiment reduces matching omissions caused by differences in character case, whitespace, and qualifiers by unifying the character format of the two types of target table identifiers and performing item-by-item comparison. By retaining the original identifiers in the two types of associated data and establishing the association between data synchronization tasks and system identifiers, the system affiliation of each data synchronization task can be accurately determined, and the affiliation results can be traced back to the original table records. By generating a matching file containing task, target table, and system affiliation, the integrity and readability of the affiliation information can be improved.
[0107] In one embodiment, such as Figure 8 As shown, step S60 above includes: S601, Read the script file name and script file content from the target encoding script file, and read the matching file name and matching file content from the link matching file; S602, determine the script output position of the target encoding script file based on the output directory and the script file name, and determine the matching output position of the link matching file based on the output directory and the matching file name; S603, write the script file content to the script output position, and write the matching file content to the matching output position, record the script output position, the matching output position and the file writing status, and generate a file output record; S604, when the file writing status indicates that the script file content and the matching file content have been written, based on the file output record, the script output file content is read from the script output position and the matching output file content is read from the matching output position. The script output file content is compared with the script file content, and the matching output file content is compared with the matching file content to generate an output verification result. S605, when the output verification result indicates that the content of the script output file is consistent with the content of the script file and the content of the matching output file is consistent with the content of the matching file, based on the file output record, the target encoded script file output to the script output position and the link matching file output to the matching output position are collected to generate a database migration file set.
[0108] In this embodiment, the script file name and script file content are read from the target encoded script file. The script file name is used to maintain file identification relationships, and the script file content includes script byte data saved according to the target encoding format. The matching file name and matching file content are read from the matching file. The matching file content includes matching data between data synchronization tasks, target tables, and system affiliation. The names and contents of the two types of files are recorded separately to avoid mixing of write locations and write contents.
[0109] The output directory is combined with the script file name to form the script output location of the target encoded script file; the output directory is combined with the matching file name to form the matching output location of the matching file. The combination process unifies the directory separator and filename format, and checks whether the two output locations point to the same file. If a file with the same name exists, the original directory hierarchy can be preserved or a file distinguishing identifier can be added to prevent existing files from being incorrectly overwritten.
[0110] The script output position is written according to the byte order of the script file content, and the matching output position is written according to the byte order of the matching file content. The file write status is used to record whether writing to the two files has started, completed, or if a write exception has occurred. The script output position, the matching output position, and the corresponding write status together form the file output record, enabling the actual storage location and write result of each file to be located.
[0111] After both files have been written, the content is reread from the corresponding output positions based on the file output records. The content of the script output file is compared with the content of the script file before writing, according to byte position or character position. The content of the matching output file is compared with the content of the matching file before writing in the same way. The comparison results record the content length, sorting order, and whether the data at each position is consistent, forming the output verification result, which is used to identify content loss, content addition, order changes, or character anomalies that occurred during the writing process.
[0112] When the output verification results show that the contents of the two types of files are consistent, the target encoded script file in the script output location and the matching file in the matching output location are aggregated according to the file output records. The database migration file set can be composed of a combination of files in the same output directory, or it can be composed of a set of data recording file names, file types, and output locations, so that the files generated in the same migration batch have a clear range and storage relationship.
[0113] This embodiment reduces file misplacement, name overwriting, and write omissions by separately determining the output positions of the target encoding script file and the matching file and recording the file writing status. By rereading the output content and comparing it with the content before writing, it can detect missing content, changes in order, and character anomalies. After verification, the migration files are aggregated, which can improve the integrity and output accuracy of the database migration file set.
[0114] In one embodiment, taking the migration of an Oracle database to a domestically developed distributed relational database by a financial institution as an example, the resources to be migrated include stored procedure scripts, trigger scripts, table structure scripts, and other database scripts used in account management, payment settlement, fund clearing, credit management, and risk control operations. The migration task configuration file records the script directory, data synchronization mapping table path, system-owned table path, target encoding format, output path, and file filtering conditions. Based on the file filtering conditions, each business directory is traversed, excluding backup files and temporary files, and script files that meet the conditions are read. The mapping table recording the relationship between the data synchronization task and the target table, as well as the system-owned table recording the relationship between the business system and the target table, are read synchronously, and corresponding output directories are created.
[0115] For each script file, the header and content bytes are read, and the original encoding format is identified from encoding markers, byte arrangement, newline format, and script delimiter format. For scripts without explicit encoding markers, the file content is decoded according to candidate encoding formats. The original encoding format is determined based on invalid bytes, replaced characters, and script structural integrity. Then, the complete file is parsed according to the determined encoding format, preserving script statements, comments, newlines, and character arrangement to generate the script content.
[0116] The scan script content contains fixed-length and variable-length character fields, identifying field definitions that employ byte length semantics. Based on the byte length occupied by the same Chinese character in the encoding format before and after migration, the required field length to maintain its original character capacity in the target encoding environment is determined. For example, if a transaction summary field can store fifty Chinese characters in the original encoding format, and the number of bytes occupied by each Chinese character increases after migration, the field length will be increased accordingly. During the adjustment process, only the length parameter is replaced, while the field name, field type, default value, constraint content, and statement order are preserved.
[0117] The adjusted script content is converted into a byte sequence according to the target encoding format and then decoded in reverse to obtain the verification script content. When the verification script content matches the original script content character by character, a target encoded script file is generated according to the original file name and file type, thus preserving the integrity of Chinese content such as customer name, transaction summary, payment remarks, and settlement instructions.
[0118] The system retrieves the data synchronization task identifier and target table identifier from the data synchronization mapping table, and the system identifier and system target table identifier from the system affiliation table, unifying the case sensitivity, whitespace characters, and qualifier formats of the two types of target table identifiers. When the target table identifiers match, the data synchronization task is associated with the corresponding business system. For example, the payment transaction record belongs to the payment settlement system, and the settlement details table belongs to the funds settlement system. Based on this, a matching file is generated that records the data synchronization task, target table, and system affiliation.
[0119] Determine the write locations of the target encoding script file and the matching file based on the output directory, record the file write status, and reread the output file for content verification. If the output content matches the content before writing, combine the target encoding script file and the matching file under the Account Management, Payment Settlement, Fund Clearing, Credit Management, and Risk Control directories into a database migration file set.
[0120] In one embodiment, a database migration apparatus based on an encoding format is provided, which corresponds one-to-one with the database migration method based on an encoding format described in the above embodiments. (Refer to...) Figure 9 , Figure 9 This is a schematic diagram of the functional modules of a preferred embodiment of the database migration device based on encoding format of the present invention. The database migration device based on encoding format includes: a migration data acquisition module 10, an encoding recognition and parsing module 20, a field structure optimization module 30, a target encoding generation module 40, a link affiliation matching module 50, and a migration file output module 60. Detailed descriptions of each functional module are as follows: The migration data acquisition module 10 is used to acquire the script file to be migrated, the data synchronization link mapping table, the system ownership table, the target encoding format, and the output directory. The encoding recognition and parsing module 20 is used to extract the file header information and content features of the script file to be migrated, identify the original encoding format based on the file header information and the content features, and parse the script file to be migrated according to the original encoding format to generate script content; The field structure optimization module 30 is used to determine the character storage length relationship of the same character under the original encoding format and the target encoding format, identify the character field definition in the script content, adjust the field length in the character field definition based on the character storage length relationship, and generate structure-optimized script content. The target encoding generation module 40 is used to encode the structure optimization script content according to the target encoding format and generate a target encoding script file. The link affiliation matching module 50 is used to determine the system affiliation of each data synchronization link in the data synchronization link mapping table based on the link target table identifier in the data synchronization link mapping table and the system target table identifier in the system affiliation table, and to generate a link matching file based on the system affiliation. The migration file output module 60 is used to output the target encoding script file and the link matching file according to the output directory, thereby generating a database migration file set.
[0121] In one embodiment, the migration data acquisition module 10 is specifically used for: Obtain the migration task configuration file, and read the script directory, data synchronization link mapping table path, system affiliation table path, target encoding identifier, output path and file filtering conditions from the migration task configuration file; Traverse the directory hierarchy in the script directory, filter script files based on the file filtering conditions, record the file paths of the filtered script files, and read the filtered script files based on the file paths to obtain the script files to be migrated; Based on the data synchronization link mapping table path, read the link table file. When the link table file contains a data synchronization link identifier field and a link target table identifier field, determine the link table file as a data synchronization link mapping table. Based on the system belonging table path, the belonging table file is read. When the belonging table file contains a system identifier field and a system target table identifier field, the belonging table file is determined to be a system belonging table. The encoding format name is determined from the migration task configuration file based on the target encoding identifier, and the encoding format name is determined as the target encoding format; The system checks if the directory pointed to by the output path exists. If the directory pointed to by the output path exists, it is designated as the output directory. If the directory pointed to by the output path does not exist, a directory is created according to the output path, and the created directory is designated as the output directory.
[0122] In one embodiment, the encoding recognition and parsing module 20 is specifically used for: Read the file header byte sequence and file content byte sequence of the script file to be migrated, identify the encoding marks and byte arrangement features in the file header byte sequence, and generate file header information based on the encoding marks and byte arrangement features; Select content segments from the file content byte sequence, identify byte distribution features, newline mark features, and script delimiter features in the content segments, and generate content features based on the byte distribution features, newline mark features, and script delimiter features; Obtain a set of candidate encoding formats, decode the file content byte sequence according to each candidate encoding format in the set, record the positions of invalid bytes and replaced characters, identify the text structure features of the decoded content, and generate candidate decoding results; Candidate decoding results that do not record the position of the invalid byte and the position of the replacement character are determined as valid candidate decoding results. The text structure features in the valid candidate decoding results are matched with the content features to generate a structure matching result. Based on the structure matching result and the file header information, the target candidate decoding result is determined from the valid candidate decoding results. The candidate encoding format corresponding to the target candidate decoding result is determined as the original encoding format. Decode the file content byte sequence according to the original encoding format to generate a decoded character sequence; The decoded character sequence is divided into script content segments based on the newline marker feature and the script separator feature. The script content segments are arranged according to the character order in the decoded character sequence to generate script content.
[0123] In one embodiment, the field structure optimization module 30 is specifically used for: Obtain character encoding attribute data, read the original character storage length and the target character storage length of the same character under the original encoding format and the target character storage length under the target encoding format from the character encoding attribute data, and generate a character storage length relationship based on the original character storage length and the target character storage length; Obtain a set of character field types, identify field type identifiers from the script content, and determine the character field definition location based on the field type identifiers that match the set of character field types; Based on the character field definition location, identify the character field definition using byte length semantics from the script content, read the field length from the character field definition, and determine the field length as the original field length; The field character capacity is determined based on the original character storage length and the original field length in the character storage length relationship. The candidate field length is determined based on the target character storage length and the field character capacity in the character storage length relationship. The field length with the larger value between the candidate field length and the original field length is determined as the target field length, and field length adjustment data is generated. Based on the field length adjustment data, the original field length in the character field definition is replaced with the target field length to generate the adjusted character field definition; Based on the defined character field positions, the adjusted character field definitions in the script content are replaced to generate structure-optimized script content.
[0124] In one embodiment, the target encoding generation module 40 is specifically used for: Obtain target encoding parameter data corresponding to the target encoding format, and read character encoding mapping data from the target encoding parameter data; Read each character according to the character arrangement order in the optimized script content to generate a character sequence to be encoded; Based on the character encoding mapping data, each character in the character sequence to be encoded is converted into character encoding byte data; Collect the character-encoded byte data corresponding to each character according to the character arrangement order in the character sequence to be encoded, and generate the target file byte sequence; Decode the target file byte sequence according to the target encoding format to generate verification script content, compare the verification script content with the structure optimization script content according to character position, and generate encoding verification result; When the encoding verification result indicates that the content of the verification script is consistent with the content of the structure optimization script, the file name and file type are read from the script file to be migrated, a target file identifier is generated based on the file name and the file type, a script file to be written is created according to the target file identifier, the target file byte sequence is written into the script file to be written, and a target encoding script file is generated.
[0125] In one embodiment, the link home matching module 50 is specifically used for: Read the data synchronization link identifier and the link target table identifier from the data synchronization link mapping table, and generate link table association data based on the data synchronization link identifier and the link target table identifier; Read the system identifier and system target table identifier from the system attribution table, and generate system table association data based on the system identifier and system target table identifier; Unify the case, whitespace, and qualifier forms of the link target table identifier in the link table association data and the system target table identifier in the system table association data to generate standard link target table identifiers and standard system target table identifiers; The standard link target table identifier is compared with the standard system target table identifier, and a target table matching result is generated based on the matching standard link target table identifier and standard system target table identifier. Based on the target table matching results, the matching link table association data and system table association data are determined. The data synchronization link identifier and link target table identifier are read from the matching link table association data, and the system identifier and system target table identifier are read from the matching system table association data. The data synchronization link identifier and the system identifier are associated to determine the system affiliation of each data synchronization link. Link affiliation data is generated based on the data synchronization link identifier, the link target table identifier, the system identifier, the system target table identifier, and the system affiliation. A link matching file is generated based on the link attribution data.
[0126] In one embodiment, the migration file output module 60 is specifically used for: Read the script file name and script file content from the target encoding script file, and read the matching file name and matching file content from the link matching file; The script output position of the target encoding script file is determined based on the output directory and the script file name, and the matching output position of the link matching file is determined based on the output directory and the matching file name. Write the script file content to the script output location, and write the matching file content to the matching output location. Record the script output location, the matching output location, and the file writing status to generate a file output record. When the file writing status indicates that both the script file content and the matching file content have been written, based on the file output record, the script output file content is read from the script output position, and the matching output file content is read from the matching output position. The script output file content is compared with the script file content, and the matching output file content is compared with the matching file content to generate an output verification result. When the output verification result indicates that the content of the script output file is consistent with the content of the script file and the content of the matching output file is consistent with the content of the matching file, based on the file output record, the target encoded script file output to the script output position and the link matching file output to the matching output position are collected to generate a database migration file set.
[0127] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 10 As shown, the computer device includes a processor, memory, network interface, and database connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile and / or volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The network interface is used to communicate with external clients via a network connection. When the computer program is executed by the processor, it implements the functions or steps of a database migration method based on an encoding format on the server side.
[0128] In one embodiment, a computer device is provided, which may be a client, and its internal structure diagram may be as follows: Figure 11 As shown, the computer device includes a processor, memory, network interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used to communicate with an external server via a network connection. When the computer program is executed by the processor, it implements client-side functions or steps of a database migration method based on an encoding format.
[0129] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to perform the following steps: Obtain the script file to be migrated, the data synchronization link mapping table, the system affiliation table, the target encoding format, and the output directory; Extract the file header information and content features of the script file to be migrated, identify the original encoding format based on the file header information and the content features, and parse the script file to be migrated according to the original encoding format to generate script content; Determine the character storage length relationship of the same character under the original encoding format and the target encoding format, identify the character field definition in the script content, adjust the field length in the character field definition based on the character storage length relationship, and generate structurally optimized script content; Encode the structure optimization script content according to the target encoding format to generate a target encoding script file; Based on the link target table identifier in the data synchronization link mapping table and the system target table identifier in the system affiliation table, the system affiliation of each data synchronization link in the data synchronization link mapping table is determined, and a link matching file is generated based on the system affiliation. Output the target encoding script file and the link matching file according to the output directory to generate a database migration file set.
[0130] In one embodiment, a computer-readable storage medium is provided, which may be non-volatile or volatile, and a computer program is stored thereon, which, when executed by a processor, performs the following steps: Obtain the script file to be migrated, the data synchronization link mapping table, the system affiliation table, the target encoding format, and the output directory; Extract the file header information and content features of the script file to be migrated, identify the original encoding format based on the file header information and the content features, and parse the script file to be migrated according to the original encoding format to generate script content; Determine the character storage length relationship of the same character under the original encoding format and the target encoding format, identify the character field definition in the script content, adjust the field length in the character field definition based on the character storage length relationship, and generate structurally optimized script content; Encode the structure optimization script content according to the target encoding format to generate a target encoding script file; Based on the link target table identifier in the data synchronization link mapping table and the system target table identifier in the system affiliation table, the system affiliation of each data synchronization link in the data synchronization link mapping table is determined, and a link matching file is generated based on the system affiliation. Output the target encoding script file and the link matching file according to the output directory to generate a database migration file set.
[0131] It should be noted that the functions or steps that can be implemented by the computer-readable storage medium or computer device described above can be referred to the relevant descriptions on the server side and client side in the foregoing method embodiments. To avoid repetition, they will not be described one by one here.
[0132] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.
[0133] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.
[0134] It should be noted that any AI models, software tools, or components not belonging to this company appearing in the embodiments of this application are merely illustrative examples and do not represent actual use. All user personal information involved in the embodiments of this application has been authorized (with the knowledge and consent) by the relevant parties or has been fully authorized by all parties, and the executing entity may obtain it through various legal and compliant means. The collection, storage, use, processing, transmission, provision, and disclosure of the information, data, and signals involved all comply with relevant laws and regulations and do not violate public order and good morals.
[0135] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.
Claims
1. A database migration method based on encoding format, characterized in that, Includes the following steps: Obtain the script file to be migrated, the data synchronization link mapping table, the system affiliation table, the target encoding format, and the output directory; Extract the file header information and content features of the script file to be migrated, identify the original encoding format based on the file header information and the content features, and parse the script file to be migrated according to the original encoding format to generate script content; Determine the character storage length relationship of the same character under the original encoding format and the target encoding format, identify the character field definition in the script content, adjust the field length in the character field definition based on the character storage length relationship, and generate structurally optimized script content; Encode the structure optimization script content according to the target encoding format to generate a target encoding script file; Based on the link target table identifier in the data synchronization link mapping table and the system target table identifier in the system affiliation table, the system affiliation of each data synchronization link in the data synchronization link mapping table is determined, and a link matching file is generated based on the system affiliation. Output the target encoding script file and the link matching file according to the output directory to generate a database migration file set.
2. The database migration method based on encoding format as described in claim 1, characterized in that, Obtain the script file to be migrated, the data synchronization link mapping table, the system affiliation table, the target encoding format, and the output directory, including: Obtain the migration task configuration file, and read the script directory, data synchronization link mapping table path, system affiliation table path, target encoding identifier, output path and file filtering conditions from the migration task configuration file; Traverse the directory hierarchy in the script directory, filter script files based on the file filtering conditions, record the file paths of the filtered script files, and read the filtered script files based on the file paths to obtain the script files to be migrated; Based on the data synchronization link mapping table path, the link table file is read. When the link table file contains a data synchronization link identifier field and a link target table identifier field, the link table file is determined to be a data synchronization link mapping table. Based on the system belonging table path, the belonging table file is read. When the belonging table file contains a system identifier field and a system target table identifier field, the belonging table file is determined to be a system belonging table. The encoding format name is determined from the migration task configuration file based on the target encoding identifier, and the encoding format name is determined as the target encoding format; The system checks if the directory pointed to by the output path exists. If the directory pointed to by the output path exists, it is designated as the output directory. If the directory pointed to by the output path does not exist, a directory is created according to the output path, and the created directory is designated as the output directory.
3. The database migration method based on encoding format as described in claim 1, characterized in that, Extract the header information and content features of the script file to be migrated, identify the original encoding format based on the header information and content features, and parse the script file to be migrated according to the original encoding format to generate script content, including: Read the file header byte sequence and file content byte sequence of the script file to be migrated, identify the encoding marks and byte arrangement features in the file header byte sequence, and generate file header information based on the encoding marks and byte arrangement features; Select content segments from the file content byte sequence, identify byte distribution features, newline mark features, and script delimiter features in the content segments, and generate content features based on the byte distribution features, newline mark features, and script delimiter features; Obtain a set of candidate encoding formats, decode the file content byte sequence according to each candidate encoding format in the set, record the positions of invalid bytes and replaced characters, identify the text structure features of the decoded content, and generate candidate decoding results; Candidate decoding results that do not record the position of the invalid byte and the position of the replacement character are determined as valid candidate decoding results. The text structure features in the valid candidate decoding results are matched with the content features to generate a structure matching result. Based on the structure matching result and the file header information, the target candidate decoding result is determined from the valid candidate decoding results. The candidate encoding format corresponding to the target candidate decoding result is determined as the original encoding format. Decode the file content byte sequence according to the original encoding format to generate a decoded character sequence; The decoded character sequence is divided into script content segments based on the newline marker feature and the script separator feature. The script content segments are arranged according to the character order in the decoded character sequence to generate script content.
4. The database migration method based on encoding format as described in claim 1, characterized in that, Determine the character storage length relationship of the same character under the original encoding format and the target encoding format, identify the character field definitions in the script content, adjust the field length in the character field definitions based on the character storage length relationship, and generate structure-optimized script content, including: Obtain character encoding attribute data, read the original character storage length and the target character storage length of the same character under the original encoding format and the target character storage length under the target encoding format from the character encoding attribute data, and generate a character storage length relationship based on the original character storage length and the target character storage length; Obtain a set of character field types, identify field type identifiers from the script content, and determine the character field definition location based on the field type identifiers that match the set of character field types; Based on the character field definition location, identify the character field definition using byte length semantics from the script content, read the field length from the character field definition, and determine the field length as the original field length; The field character capacity is determined based on the original character storage length and the original field length in the character storage length relationship. The candidate field length is determined based on the target character storage length and the field character capacity in the character storage length relationship. The field length with the larger value between the candidate field length and the original field length is determined as the target field length, and field length adjustment data is generated. Based on the field length adjustment data, the original field length in the character field definition is replaced with the target field length to generate the adjusted character field definition; Based on the defined character field positions, the adjusted character field definitions in the script content are replaced to generate structure-optimized script content.
5. The database migration method based on encoding format as described in claim 1, characterized in that, The structure optimization script content is encoded according to the target encoding format to generate a target encoding script file, including: Obtain target encoding parameter data corresponding to the target encoding format, and read character encoding mapping data from the target encoding parameter data; Read each character according to the character arrangement order in the optimized script content to generate a character sequence to be encoded; Based on the character encoding mapping data, each character in the character sequence to be encoded is converted into character encoding byte data; Collect the character-encoded byte data corresponding to each character according to the character arrangement order in the character sequence to be encoded, and generate the target file byte sequence; Decode the target file byte sequence according to the target encoding format to generate verification script content, compare the verification script content with the structure optimization script content according to character position, and generate encoding verification result; When the encoding verification result indicates that the content of the verification script is consistent with the content of the structure optimization script, the file name and file type are read from the script file to be migrated, a target file identifier is generated based on the file name and the file type, a script file to be written is created according to the target file identifier, the target file byte sequence is written into the script file to be written, and a target encoding script file is generated.
6. The database migration method based on encoding format as described in claim 1, characterized in that, Based on the link target table identifier in the data synchronization link mapping table and the system target table identifier in the system affiliation table, the system affiliation of each data synchronization link in the data synchronization link mapping table is determined, and a link matching file is generated based on the system affiliation, including: Read the data synchronization link identifier and the link target table identifier from the data synchronization link mapping table, and generate link table association data based on the data synchronization link identifier and the link target table identifier; Read the system identifier and system target table identifier from the system attribution table, and generate system table association data based on the system identifier and system target table identifier; Unify the case, whitespace, and qualifier formats of the link target table identifier in the link table association data and the system target table identifier in the system table association data to generate standard link target table identifiers and standard system target table identifiers; The standard link target table identifier is compared with the standard system target table identifier, and a target table matching result is generated based on the matching standard link target table identifier and standard system target table identifier. Based on the target table matching results, the matching link table association data and system table association data are determined. The data synchronization link identifier and link target table identifier are read from the matching link table association data, and the system identifier and system target table identifier are read from the matching system table association data. The data synchronization link identifier and the system identifier are associated to determine the system affiliation of each data synchronization link. Link affiliation data is generated based on the data synchronization link identifier, the link target table identifier, the system identifier, the system target table identifier, and the system affiliation. A link matching file is generated based on the link attribution data.
7. The database migration method based on encoding format as described in claim 1, characterized in that, Output the target encoding script file and the link matching file according to the output directory, and generate a database migration file set, including: Read the script file name and script file content from the target encoding script file, and read the matching file name and matching file content from the link matching file; The script output position of the target encoding script file is determined based on the output directory and the script file name, and the matching output position of the link matching file is determined based on the output directory and the matching file name. Write the script file content to the script output location, and write the matching file content to the matching output location. Record the script output location, the matching output location, and the file writing status to generate a file output record. When the file writing status indicates that both the script file content and the matching file content have been written, based on the file output record, the script output file content is read from the script output position, and the matching output file content is read from the matching output position. The script output file content is compared with the script file content, and the matching output file content is compared with the matching file content to generate an output verification result. When the output verification result indicates that the content of the script output file is consistent with the content of the script file and the content of the matching output file is consistent with the content of the matching file, based on the file output record, the target encoded script file output to the script output position and the link matching file output to the matching output position are collected to generate a database migration file set.
8. A database migration device based on an encoding format, characterized in that, The database migration device based on the encoding format includes: The migration data acquisition module is used to acquire the script file to be migrated, the data synchronization link mapping table, the system affiliation table, the target encoding format, and the output directory; The encoding recognition and parsing module is used to extract the file header information and content features of the script file to be migrated, identify the original encoding format based on the file header information and the content features, and parse the script file to be migrated according to the original encoding format to generate script content; The field structure optimization module is used to determine the character storage length relationship of the same character under the original encoding format and the target encoding format, identify the character field definition in the script content, adjust the field length in the character field definition based on the character storage length relationship, and generate structure-optimized script content. The target encoding generation module is used to encode the structure optimization script content according to the target encoding format and generate a target encoding script file. The link affiliation matching module is used to determine the system affiliation of each data synchronization link in the data synchronization link mapping table based on the link target table identifier in the data synchronization link mapping table and the system target table identifier in the system affiliation table, and to generate a link matching file based on the system affiliation. The migration file output module is used to output the target encoding script file and the link matching file according to the output directory, and generate a database migration file set.
9. A computer device, characterized in that, The computer device includes a memory, a processor, and a database migration program based on an encoding format stored on the memory and executable on the processor, wherein the database migration program based on the encoding format, when executed by the processor, implements the steps of the database migration method based on an encoding format as described in any one of claims 1-7.
10. A computer-readable storage medium, characterized in that, The storage medium stores a database migration program based on an encoding format, which, when executed by a processor, implements the steps of the database migration method based on an encoding format as described in any one of claims 1-7.
