Data migration methods, apparatus, storage media and electronic devices

Abstract

This application discloses a data migration method, apparatus, storage medium, and electronic device, relating to the field of big data. The method includes: receiving data to be migrated and determining the encoding type of the data to be migrated to obtain a first type; determining a target transcoding library for storing the data of the first type, and storing the data to be migrated in a temporary table in the target transcoding library; parsing the table structure of the temporary table and generating a business table in a business database in a database cluster based on the table structure; parsing the configuration data of the data to be migrated to obtain target configuration data, and storing the target configuration data in an external table corresponding to the business database in the database cluster; transcoding the data to be migrated into encoded data of a second type based on the target configuration data to obtain target data, and storing the target data in the business table. This application solves the problem in related technologies where the converted data is prone to anomalies after encoding conversion of fixed-length data.

Description

Technical Field

[0001] This application relates to the field of big data, and more specifically, to a data migration method, apparatus, storage medium, and electronic device. Background Technology

[0002] In the fintech sector, data application has become crucial for business operations. Massive amounts of financial data have a profound impact on market insights and customer service. To handle large-scale data and operations, many financial institutions choose to use MPP (Massively Parallel Processing) distributed databases, especially in OLAP (On-Line Analytical Processing) scenarios, where their parallel processing capabilities make data querying and analysis more efficient.

[0003] However, the complexity and diversity of financial data sources, especially the issue of data encoding, pose certain challenges to data processing. Because different databases store data using different encodings, encoding conversion is necessary during data migration to ensure proper data storage. However, to improve readability and ease of use, current data is typically fixed-length. When converting fixed-length data to different encodings, the converted data may contain missing or garbled characters, leading to errors and business losses.

[0004] There is currently no effective solution to the problem that the converted data is prone to anomalies after the code conversion of fixed-length data in related technologies. Summary of the Invention

[0005] This application provides a data migration method, apparatus, storage medium, and electronic device to solve the problem in related technologies that the converted data is prone to anomalies after code conversion of fixed-length data.

[0006] According to one aspect of this application, a data migration method is provided. The method includes: receiving data to be migrated and determining the encoding type of the data to be migrated to obtain a first type; determining a target transcoding library for storing the first type of data, and storing the data to be migrated in a temporary table in the target transcoding library; parsing the table structure of the temporary table, and generating a business table in a business database in a database cluster based on the table structure, wherein the table structure of the business table is the same as the table structure of the temporary table, the business database and the target transcoding library are located in the same database cluster, and the business database is used to store encoded data of a second type; parsing the data to be migrated for configuration data to obtain target configuration data, and storing the target configuration data in an external table corresponding to the business database in the database cluster; transcoding the data to be migrated into encoded data of the second type based on the target configuration data to obtain target data, and storing the target data in the business table.

[0007] Optionally, parsing the configuration data of the data to be migrated to obtain the target configuration data and storing the target configuration data in the external table corresponding to the business database in the database cluster includes: parsing the format of the data to be migrated to obtain the format information of the data to be migrated, wherein the format information includes at least one of the following: field name, field attribute, format, alignment, and padding character of the data to be migrated; generating configuration information for each field in the data to be migrated based on the format information, wherein the configuration information includes: field length and encoding method; generating a configuration file for the data to be migrated based on the configuration information and format information, and storing the target configuration data in the configuration file in the external table according to a preset mapping relationship, wherein the preset mapping relationship is used to characterize the conversion relationship between the data type of the target configuration data and the data type corresponding to the business database.

[0008] Optionally, storing the target configuration data in the configuration file into an external table according to a preset mapping relationship includes: determining the data type of the target configuration data to obtain the target data type; obtaining the data read type of the business database and obtaining the mapping relationship between the data read type and the target data type to obtain the preset mapping relationship; converting the data type of the target configuration data into the data read type according to the preset mapping relationship, and storing the converted target configuration data into the external table.

[0009] Optionally, transcoding the data to be migrated into second-type encoded data according to the target configuration data to obtain the target data includes: splitting the data to be migrated into multiple groups of field data according to the target configuration data; obtaining the first encoding standard of the first-type encoded data, obtaining the second encoding standard of the second-type encoded data, and obtaining the conversion relationship between the first encoding standard and the second encoding standard; and performing transcoding operations on each group of field data according to the conversion relationship to obtain the target data.

[0010] Optionally, after receiving the data to be migrated and determining the encoding type of the data to be migrated to obtain a first type, the method further includes: identifying the encoded data in the data to be migrated and performing a validity check on the encoded data to obtain a check result; if the check result is normal, detecting the operating environment of the database cluster to obtain a detection result; if the detection result is normal, executing the steps of determining the target transcoding library for storing the data of the first type and storing the data to be migrated in a temporary table in the target transcoding library.

[0011] Optionally, parsing the table structure of a temporary table includes: identifying the table structure of the temporary table to obtain the table structure; or, obtaining the target configuration data of the data to be migrated, and analyzing the field attributes in the target configuration data to obtain the table structure.

[0012] Optionally, after storing the target data in the business table, the method further includes: receiving the data storage result sent by the business database; and deleting the temporary table and the external table if the data storage result indicates that the target data has been successfully stored.

[0013] According to another aspect of this application, a data migration apparatus is provided. The apparatus includes: a first receiving unit, configured to receive data to be migrated and determine the encoding type of the data to be migrated to obtain a first type; a determining unit, configured to determine a target transcoding library for storing the data of the first type, and store the data to be migrated in a temporary table in the target transcoding library; a first parsing unit, configured to parse the table structure of the temporary table and generate a business table in a business database in a database cluster according to the table structure, wherein the table structure of the business table is the same as the table structure of the temporary table, the business database and the target transcoding library are located in the same database cluster, and the business database is used to store encoded data of a second type; a second parsing unit, configured to parse the data to be migrated into configuration data to obtain target configuration data, and store the target configuration data in an external table corresponding to the business database in the database cluster; and a transcoding unit, configured to transcode the data to be migrated into encoded data of the second type according to the target configuration data to obtain target data, and store the target data in the business table.

[0014] According to another aspect of the present invention, a computer storage medium is also provided for storing a program, wherein the program, when running, controls the device where the computer storage medium is located to perform a data migration method.

[0015] According to another aspect of the present invention, an electronic device is also provided, comprising one or more processors and a memory; the memory stores computer-readable instructions, and the processor is configured to execute the computer-readable instructions, wherein the computer-readable instructions, when executed, perform a data migration method.

[0016] This application employs the following steps: receiving data to be migrated and determining its encoding type to obtain a first type; determining a target transcoding library for storing the first type of data, and storing the data to be migrated in a temporary table within the target transcoding library; parsing the table structure of the temporary table and generating a business table in the business database within the database cluster based on the table structure, wherein the business table has the same table structure as the temporary table, and the business database and the target transcoding library are located in the same database cluster, with the business database used to store the second type of encoded data; parsing the configuration data of the data to be migrated to obtain target configuration data, and storing the target configuration data in an external table corresponding to the business database within the database cluster; transcoding the data to be migrated into the second type of encoded data based on the target configuration data to obtain target data, and storing the target data in the business table. This solves the problem in related technologies where the converted data is prone to anomalies after encoding conversion of fixed-length data. By parsing the configuration data of the data to be migrated, the target configuration information of the data to be migrated is obtained. Based on the target configuration information, the data to be migrated is analyzed to determine the configuration information corresponding to each field in the data to be migrated. Then, a transcoding operation is performed according to the configuration information, so that each fixed-length field can be accurately transcoded into encoded data that can be read by the business database. The data to be migrated is stored in the target transcoding library located in the same database cluster as the business database, so that the data transcoding operation can be performed in the same database cluster. This achieves the effect of completing the data transcoding operation in the database cluster and ensuring that the transcoded data is free of anomalies. Attached Figure Description

[0017] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application. The illustrative embodiments and descriptions of this application are used to explain this application and do not constitute an undue limitation of this application. In the drawings:

[0018] Figure 1 This is a flowchart of a data migration method provided according to an embodiment of this application;

[0019] Figure 2 This is a schematic diagram of a data migration system provided according to an embodiment of this application;

[0020] Figure 3 This is a schematic diagram illustrating the connection between a data migration system and an external system according to an embodiment of this application;

[0021] Figure 4 This is a schematic diagram of a data migration apparatus provided according to an embodiment of this application;

[0022] Figure 5 This is a schematic diagram of an electronic device provided according to an embodiment of this application. Detailed Implementation

[0023] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0024] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0025] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate for the embodiments of this application described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0026] It should be noted that all information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for display, data used for analysis, etc.) involved in this disclosure are information and data authorized by the user or fully authorized by all parties. For example, this system has an interface with relevant users or organizations. Before obtaining relevant information, it is necessary to send an acquisition request to the aforementioned user or organization through the interface, and obtain the relevant information after receiving consent information from the aforementioned user or organization.

[0027] It should be noted that the data migration methods, apparatus, storage media and electronic devices defined in this disclosure can be used in the field of big data, or in any field other than big data. The application fields of the data migration methods, apparatus, storage media and electronic devices defined in this disclosure are not limited.

[0028] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, electronic medical records, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with relevant regulations and standards, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0029] Therefore, this application aims to provide a solution that can solve the above-mentioned technical problems, the details of which will be described in subsequent embodiments.

[0030] For ease of description, the following explains some of the nouns or terms used in the embodiments of this application:

[0031] MPP architecture: Massive Parallel Processing. It consists of multiple servers connected through a network of interconnected nodes to work together and complete the same task. From the user's perspective, it is a server system where each node only accesses its own local resources (memory, storage, etc.), making it a completely share-nothing structure.

[0032] Distributed databases: Distributed databases refer to database systems where data is physically distributed but logically centrally managed. Petabyte-scale (a unit of storage) distributed databases based on the Shared-Nothing architecture are increasingly widely used in the fintech field. Among them, analytical distributed databases support critical business scenarios in the financial industry such as massive data warehouses, data marts, real-time analysis, real-time decision-making, and hybrid workloads.

[0033] Data encoding: Data encoding is a convention that defines how to represent characters using a specific set of numbers or symbols. It is the foundation for processing text data in computers. For example: UTF-8: This is a variable-length character encoding standard that uses 1 to 4 bytes to represent each character. The advantage of UTF-8 encoding is that it can represent any character in the Unicode standard, including characters from various languages. GBK: This is an encoding standard primarily used to represent Chinese characters; it uses 1 or 2 bytes to represent each character. GBK encoding includes all Chinese characters, but its support for other languages ​​is not as good as UTF-8. LATIN1: Also known as ISO-8859-1, this is an encoding standard used to represent characters in Western European languages, using 1 byte per character.

[0034] Batch data processing: Batch data processing refers to the process of uniformly processing large amounts of data. This typically involves operations such as data import, data transformation, data querying, and data export.

[0035] Parallel Data Import: MPP distributed database provides a parallel import client. The parallel import function identifies the data source file by using information such as the import strategy and data format set in the external table. It then imports data from the data source file into the database in parallel using multiple data nodes (DNs), thereby improving overall import performance.

[0036] PostgreSQL Foreign Tables: An object-relational database management system, PostgreSQL foreign tables are a feature that allows you to represent data stored on an external server within a Postgres database. Foreign tables can be queried by the Postgres server just like local tables, but the data actually retrieved can be stored in any type of external data source, such as another Postgres database, other types of databases, or even a file system.

[0037] Fixed-length files: Fixed-length files are a file format where each record has a fixed length. This makes file reading and writing operations simpler and faster, but it also requires ensuring that all records conform to the preset length. If a record is not long enough, it may need to be padded with spaces or other characters.

[0038] Data governance: Data governance is a management approach aimed at ensuring that an organization's data assets are used, protected, and enhanced appropriately. Data governance typically includes the following aspects: Data quality management: Ensuring the accuracy, integrity, consistency, and availability of data. Data protection: Protecting data from unauthorized access and damage, including data security and privacy protection. Data lifecycle management: Planning and managing the entire process of data from creation to destruction. Data strategy and architecture: Planning the strategies and architecture for data acquisition, storage, processing, analysis, and use.

[0039] Metadata: Metadata describes data and helps users understand and use it. Metadata can include the following types of information: Descriptive metadata: Describes the content and characteristics of the data, such as title, author, keywords, and summary. Structured metadata: Describes the structure and format of the data, such as data type, field length, and encoding method. Administrative metadata: Describes the management and usage of the data, such as creation date, last modification date, access permissions, and copyright information. Metadata has important applications in many fields such as data governance, data management, and data analysis. For example, through metadata, users can more easily search, access, understand, and utilize data.

[0040] According to an embodiment of this application, a data migration method is provided.

[0041] Figure 1 This is a flowchart of a data migration method provided according to an embodiment of this application. For example... Figure 1 As shown, the method includes the following steps:

[0042] Step S101: Receive the data to be migrated and determine the encoding type of the data to be migrated to obtain the first type.

[0043] Specifically, after receiving the data to be migrated, it is necessary to determine whether the data needs to be transcoded. Therefore, the encoding type of the data to be migrated needs to be obtained first to obtain the first type. The data type can be the data encoding type, such as UTF-8 (Unicode Transformation Format) or GBK (Chinese Internal Code Specification). Based on the encoding type of the data to be migrated, it can be determined whether the data needs to be transcoded.

[0044] For example, if the target database stores data encoded in UTF-8, and the data to be migrated is also encoded in UTF-8, then no encoding operation is needed, and the data can be directly migrated to the target database for storage. However, if the target database stores data encoded in GBK, then the encoding of the data to be migrated needs to be converted from GBK to UTF-8 so that the data can be stored correctly in the target database.

[0045] Step S102: Determine the target transcoding library for storing the first type of data, and store the data to be migrated in a temporary table in the target transcoding library.

[0046] Specifically, when data to be migrated needs to be transcoded, it is necessary to store the data to be migrated according to its encoding type to ensure that the data stored in the database cluster does not become abnormal. Therefore, it is necessary to determine the target transcoding library for storing the data of the first type, based on the encoding type of the data to be migrated. A temporary table is then created in the target transcoding library, and the data to be migrated is stored in the temporary table. Since the data to be migrated is fixed-length, the storage length of each field in the temporary table needs to be determined based on the fixed length of the data to be migrated to avoid data loss after the data to be migrated is stored in the temporary table.

[0047] Step S103: Parse the table structure of the temporary table and generate a business table in the business database of the database cluster according to the table structure. The table structure of the business table is the same as that of the temporary table. The business database and the target transcoding database are located in the same database cluster. The business database is used to store the second type of encoded data.

[0048] Specifically, after generating a temporary table and successfully storing the data to be migrated in the temporary table, a business table can be generated in the business database of the database cluster based on the table structure of the temporary table. The business table is used to store the transcoded data to be migrated, thereby ensuring that the number of fields in the business table is the same as the number of fields in the temporary table, so that the business table can definitely store the data of all fields in the transcoded data to be migrated.

[0049] It should be noted that because the encoding length of the same data differs under different encoding schemes, the length of the data storage area in the business table cannot be preset or set to the maximum value, so that the data after encoding conversion can be displayed completely in the business table.

[0050] Step S104: Parse the configuration data of the data to be migrated to obtain the target configuration data, and store the target configuration data in the external table corresponding to the business database in the database cluster.

[0051] Specifically, since the data to be migrated is fixed-length data, and the length of each record is fixed, if the length of a record is insufficient, it needs to be padded with spaces or other characters. Therefore, when transcoding the data under each field in the data to be migrated, in order to ensure the accuracy of the transcoded data, it is necessary to determine the information of each data, such as field name, format, alignment, padding characters, etc., which is the target configuration data of the data to be migrated. Based on the target configuration data, it can be determined which characters in the data to be migrated are padding characters, which characters actually need to be transcoded, and the field information corresponding to each data. This ensures that the data can be transcoded normally while ensuring that the data is stored correctly in the business database, thus improving the data storage quality.

[0052] It should be noted that since the encoding type of the target configuration data may be inconsistent with the encoding type required in the business database, the target configuration data can be stored in an external table corresponding to the business database. The external table can be located in any database in the database cluster other than the business database, thereby ensuring the normal storage of the target configuration data and the normal reading of the target configuration data by the business database.

[0053] Step S105: Based on the target configuration data, the data to be migrated is transcoded into the second type of encoded data to obtain the target data, and the target data is stored in the business table.

[0054] Specifically, after determining the target configuration data, the configuration information of each piece of data in the data to be migrated can be determined based on the target configuration data. The data can then be modified according to the configuration information of each piece of data, such as removing padding characters. Furthermore, the data can be transcoded to obtain the second type of encoded data. Thus, each piece of data in the data to be migrated can be transcoded and stored in the business table in sequence, thereby completing the data migration operation.

[0055] The data migration method provided in this application embodiment receives data to be migrated and determines its encoding type to obtain a first type; determines a target transcoding library for storing the first type of data, and stores the data to be migrated in a temporary table in the target transcoding library; parses the table structure of the temporary table and generates a business table in the business library of the database cluster based on the table structure, wherein the table structure of the business table is the same as that of the temporary table, the business library and the target transcoding library are located in the same database cluster, and the business library is used to store the second type of encoded data; parses the configuration data of the data to be migrated to obtain target configuration data, and stores the target configuration data in an external table corresponding to the business library in the database cluster; transcodes the data to be migrated into the second type of encoded data based on the target configuration data to obtain target data, and stores the target data in the business table. This solves the problem in related technologies where the converted data is prone to anomalies after encoding conversion of fixed-length data. By parsing the configuration data of the data to be migrated, the target configuration information of the data to be migrated is obtained. Based on the target configuration information, the data to be migrated is analyzed to determine the configuration information corresponding to each field in the data to be migrated. Then, a transcoding operation is performed according to the configuration information, so that each fixed-length field can be accurately transcoded into encoded data that can be read by the business database. The data to be migrated is stored in the target transcoding library located in the same database cluster as the business database, so that the data transcoding operation can be performed in the same database cluster. This achieves the effect of completing the data transcoding operation in the database cluster and ensuring that the transcoded data is without anomalies.

[0056] Optionally, in the data migration method provided in this application embodiment, parsing the configuration data of the data to be migrated to obtain target configuration data and storing the target configuration data in the external table corresponding to the business database in the database cluster includes: parsing the format of the data to be migrated to obtain the format information of the data to be migrated, wherein the format information includes at least one of the following: field name, field attribute, format, alignment, and padding character of the data to be migrated; generating configuration information for each field in the data to be migrated based on the format information, wherein the configuration information includes: field length and encoding method; generating a configuration file for the data to be migrated based on the configuration information and the format information, and storing the target configuration data in the configuration file in the external table according to a preset mapping relationship, wherein the preset mapping relationship is used to characterize the conversion relationship between the data type of the target configuration data and the data type corresponding to the business database.

[0057] Specifically, when parsing configuration data, the format of the data to be migrated needs to be parsed first to obtain format information, such as the field name, field attribute, format, padding characters, etc. of each data in the data to be migrated. Based on the format information, the actual data content of each data can be determined, that is, the field length of each data. For example, if data A in the data to be migrated is "52300000", where the fixed length is 8, and the last 5 zeros of data A are padding characters, then the actual data content of data A can be determined to be 523 based on the format information of the data.

[0058] Furthermore, since the databases in the database cluster may not be able to directly read and use the configuration file, it is necessary to determine the data type of the configuration file and the data type that the business database can read. Based on the preset mapping relationship between the two data types, the target configuration data in the configuration file is converted into data that the business database can recognize and read, thereby ensuring the normal progress of data conversion.

[0059] Optionally, in the data migration method provided in this application embodiment, storing the target configuration data in the configuration file into an external table according to a preset mapping relationship includes: determining the data type of the target configuration data to obtain the target data type; obtaining the data reading type of the business library and obtaining the mapping relationship between the data reading type and the target data type to obtain a preset mapping relationship; converting the data type of the target configuration data into the data reading type according to the preset mapping relationship, and storing the converted target configuration data into an external table.

[0060] Specifically, when performing data type conversion, you can first obtain the data reading type of the business database, obtain the data type of the target configuration data, determine the preset mapping relationship based on the conversion relationship between the two data types, and store the target configuration data in the configuration file into an external table according to the preset mapping relationship.

[0061] Optionally, in the data migration method provided in this application embodiment, transcoding the data to be migrated into second type of encoded data according to the target configuration data to obtain the target data includes: splitting the data to be migrated into multiple groups of field data according to the target configuration data; obtaining a first encoding standard for the first type of encoded data, obtaining a second encoding standard for the second type of encoded data, and obtaining the conversion relationship between the first encoding standard and the second encoding standard; and performing transcoding operations on each group of field data according to the conversion relationship to obtain the target data.

[0062] Specifically, during data transcoding, the data to be migrated can first be split into multiple groups of field data based on the target configuration data. Each group of field data may contain one or more data entries. After splitting the data to be migrated, the code conversion relationship between the first and second encoding standards can be determined according to a preset transcoding method. Then, according to the code conversion relationship, each data entry in each group of field data is transcoded sequentially, thereby completing the conversion of the encoding type of the data in the data to be migrated from the first type to the second type.

[0063] For example, after obtaining a piece of data, the actual data content of the data can be determined by the target configuration data, and the actual data content can be converted from the first type to the original characters according to the first encoding standard, and the original characters can be converted into the second type of encoded data according to the second encoding standard, thereby completing the data transcoding operation.

[0064] Optionally, in the data migration method provided in this application embodiment, after receiving the data to be migrated and determining the encoding type of the data to be migrated to obtain a first type, the method further includes: identifying the encoded data in the data to be migrated and performing a legality check on the encoded data to obtain a check result; if the check result is normal, detecting the operating environment of the database cluster to obtain a detection result; if the detection result is normal, executing the step of determining a target transcoding library for storing the data of the first type and storing the data to be migrated in a temporary table in the target transcoding library.

[0065] Specifically, after receiving the data to be migrated, the first step is to verify the data to determine whether it can be migrated to the business database. After passing the legality verification, it is also necessary to determine the operating environment and ensure that the operating environment is in a normal state, so that the subsequent transcoding and migration operations can proceed normally.

[0066] Optionally, in the data migration method provided in this application embodiment, parsing the table structure of the temporary table includes: identifying the table structure of the temporary table to obtain the table structure; or, obtaining the target configuration data of the data to be migrated, and analyzing the field attributes in the target configuration data to obtain the table structure.

[0067] Specifically, when parsing the structure of a temporary table, the structure can be directly identified to obtain the table result, or the field attributes in the target configuration data can be analyzed to determine the number of data entries and data length under each field. Based on the field attributes, the table structure in the business table can be determined, thereby improving the accuracy of the generated business table.

[0068] Optionally, in the data migration method provided in the embodiments of this application, after storing the target data in the business table, the method further includes: receiving the data storage result sent by the business database; and deleting the temporary table and the external table if the data storage result indicates that the target data has been successfully stored.

[0069] Specifically, after the data migration operation is completed, the temporary tables and external tables generated during the data migration operation can be deleted, thereby reducing the amount of data stored in the database cluster and improving the operating performance of the database cluster.

[0070] According to an embodiment of this application, an optional data migration method is provided:

[0071] Step 1: Initialization Configuration Phase: The data platform creates a new database instance on the MPP distributed database, called the "intermediate transcoding library." This library uses GBK encoding and is only used for loading GBK encoded data. This library should be empty when there are no batch import jobs. The creation syntax is as follows:

[0072] CREATE DATABASE gbkdb ENCODING'GBK'template=template0 OWNER admin;

[0073] Subsequently, the data platform creates a server in the UTF8 service library that points to the GBK intermediate transcoding library. The address parameter is the IP address and port of the GBK encoding service library. The creation syntax is as follows:

[0074]

[0075] Created database instances and servers can be reused.

[0076] Step 2: Analysis Phase: During the R&D and testing phase, the data application parses the data to be imported and migrated. This yields configuration information for a GBK-encoded fixed-length format file (the filename can be set to gbk_fixed.txt), including the data's field names, format, alignment, padding characters, etc. Based on this analysis result, a configuration file detailing the length and encoding of each field in the data to be migrated is generated, named, for example, gbk_fixed.json.

[0077] Step 3: Fixed-length data import stage: The data application configures batch processing jobs through the big data scheduling platform, uses the GBK intermediate transcoding library, and uses the distributed data service client to import fixed-length GBK encoded data into a temporary table by setting the encoding format of the file mapped to the table. The table name is, for example, t1_mid_gbk.

[0078] Step 4: Cross-Instance Transcoding Pull Phase: The data application configures batch processing jobs through the big data scheduling platform. In the UTF8 encoding business library, a GBK-encoded external table is created using Postgres' external table mechanism, named, for example, f_t1_from_gbk_2_utf8. The field attributes of the gbk_fixed.json file from the analysis phase are loaded. The table structures of the source and target tables are declared in the external table configuration, and the configuration information is added to the external table. The external table is then mapped to a temporary table (t1_mid_gbk) in the GBK intermediate transcoding library within the same cluster. Example code:

[0079]

[0080] Based on the mapping relationship, the configuration data in the external table is used to transcode the data to be migrated in the temporary table, and the transcoded result is stored in the business table in the business database. Ultimately, this enables data to be retrieved from a GBK-encoded database to a business table in the UTF-8 business database (e.g., t1_utf8) within the same cluster via an external table. Example code:

[0081] insert into t1_utf8 select*from f_t1_from_gbk_2_utf8;

[0082] Step 5: Cleanup Phase: The data platform's cleanup service performs cleanup operations, cleaning up temporary files on the data loading server, deleting temporary tables and related temporary objects in the intermediate transcoding library, and deleting external tables in the GBK encoding format in the UTF-8 business library, thereby completing the data transcoding and data migration operations of the data to be migrated.

[0083] In this embodiment, Figure 2 This is a schematic diagram of a data migration system provided according to an embodiment of this application, such as... Figure 2 As shown, the aforementioned data migration method is executed using an optional data migration system as the execution entity. This data migration system includes at least:

[0084] Overall Control Unit 1: Used to provide overall process control and other functions. It consists of two core modules of cost tools: Configuration Parameter Management Unit 2 and Task Scheduling Unit 5. Each module is loosely coupled, and the business implementation module and parameter configuration management module are separated, which can improve the efficiency of operation and maintenance deployment and user development.

[0085] Configuration Management Unit 2 consists of Parameter Parsing Module 3 and Environment Check Module 4. It performs legality verification on input parameters, loads necessary configuration parameters, and pre-checks the availability of the environment. If the pre-verification process passes and all configuration parameters are loaded, it sends a signal to the main control unit 1, indicating that the management tool has entered the running state.

[0086] Parameter parsing module 3 is used to validate the validity of input parameters and load necessary configuration parameters.

[0087] Environment Check Module 4: Used to pre-check the availability of the environment.

[0088] Task scheduling unit 5: Composed of data file code parsing module 6, data file processing SQL statement preprocessing module 7, and data file processing command execution module 8, it is the core module for interaction between this system and the database cluster and resource scheduling system.

[0089] Data file encoding parsing module 6: Implements the business logic for encoding conversion.

[0090] Data file processing SQL statement preprocessing module 7: used to concatenate input parameters, configuration information, and templated file import or query export statements.

[0091] Data file processing command execution module 8: Used to interact with distributed database clusters and data loading servers.

[0092] Figure 3 This is a schematic diagram illustrating the connection between a data migration system and an external system according to an embodiment of this application, such as... Figure 3 As shown, it includes: Big Data Service Platform System 9, Unified Scheduling Service System 10, Data Migration System 11, Data File Transfer Server 12, Distributed Database Cluster 13, Distributed Database Coordination Node 14, and Distributed Database Data Node 15.

[0093] The data migration system and its connected external systems in this application are both included in the big data service platform system 9, which is used to provide big data processing, handling and query services to external parties.

[0094] The unified scheduling service system 10 is a distributed workflow scheduling system used to provide big data platform users with data import, processing, sorting and export of underlying data sources. Data development engineers arrange processing jobs according to business needs.

[0095] The data migration system 11 is used in the big data service platform system to specifically handle the data import submitted by users through the unified scheduling service system 10. It is directly connected to the underlying big data distributed database cluster 13, data file transfer server 12 and other modules, and provides functions such as distributed data file import cluster.

[0096] Data file transfer server 12, also known as the data file import / export transfer server, deploys data service processes to store data files to be imported. Each data transfer server is part of the big data platform's scheduling service, and each node is stateless and can serve as a backup for each other. When the data file processing capacity of the job scheduling system reaches a bottleneck, the system's performance capacity can be expanded by increasing the number of data file transfer servers.

[0097] Distributed database cluster 13, based on an MPP architecture, comprises several distributed database clusters that provide high-performance data processing capabilities for data-intensive applications. Due to differences in application business domains, different distributed database clusters exhibit variations in cluster size, version, and interface in practical applications. Data migration system 11 addresses these technical differences through configuration methods, exposing a unified interface, unified deployment, and unified operation and maintenance service to database users.

[0098] A single distributed database cluster 13 includes: a distributed database cluster coordination node 14 and a distributed database data node 15. The distributed database cluster coordination node 14 is used to receive and process SQL commands for data file processing submitted from the data migration system 11, and to send the data file processing commands to the file transfer node where the data is located. The distributed database data node 15 efficiently transfers and writes the data file to the data file transfer server 12 through the parallel export service, thereby completing the entire data migration and data processing operation. This achieves the ability to process fixed-length data files with GBK encoding and effectively import them into the MPP distributed database using UTF8 encoding.

[0099] It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0100] This application also provides a data migration apparatus. It should be noted that the data migration apparatus of this application can be used to execute the data migration method provided in this application. The data migration apparatus provided in this application will be described below.

[0101] Figure 4 This is a schematic diagram of a data migration apparatus provided according to an embodiment of this application. Figure 4 As shown, the device includes: a first receiving unit 41, a determining unit 42, a first parsing unit 43, a second parsing unit 44, and a transcoding unit 45.

[0102] The first receiving unit 41 is used to receive the data to be migrated and determine the encoding type of the data to be migrated to obtain the first type.

[0103] The determining unit 42 is used to determine the target transcoding library for storing the first type of data and to store the data to be migrated in a temporary table in the target transcoding library.

[0104] The first parsing unit 43 is used to parse the table structure of the temporary table and generate a business table in the business database in the database cluster according to the table structure. The table structure of the business table is the same as that of the temporary table. The business database and the target transcoding database are located in the same database cluster. The business database is used to store the second type of encoded data.

[0105] The second parsing unit 44 is used to parse the configuration data of the data to be migrated, obtain the target configuration data, and store the target configuration data in the external table corresponding to the business database in the database cluster.

[0106] The transcoding unit 45 is used to transcode the data to be migrated into second-type encoded data according to the target configuration data, obtain the target data, and store the target data in the business table.

[0107] The data migration apparatus provided in this application embodiment receives data to be migrated through a first receiving unit 41 and determines the encoding type of the data to be migrated to obtain a first type; a determining unit 42 determines a target transcoding library for storing the data of the first type and stores the data to be migrated in a temporary table in the target transcoding library; a first parsing unit 43 parses the table structure of the temporary table and generates a business table in the business library in the database cluster according to the table structure, wherein the table structure of the business table is the same as the table structure of the temporary table, the business library and the target transcoding library are located in the same database cluster, and the business library is used to store the encoded data of the second type; a second parsing unit 44 parses the configuration data of the data to be migrated to obtain target configuration data and stores the target configuration data in the external table corresponding to the business library in the database cluster; a transcoding unit 45 transcodes the data to be migrated into the encoded data of the second type according to the target configuration data to obtain target data and stores the target data in the business table. This invention addresses the issue of anomalies in the converted data after encoding conversion of fixed-length data in related technologies. By parsing the configuration data of the data to be migrated, the target configuration information of the data to be migrated is obtained. Based on this target configuration information, the data to be migrated is analyzed to determine the configuration information corresponding to each field in the data to be migrated. A transcoding operation is then performed based on this configuration information, ensuring that each fixed-length field is accurately transcoded into readable encoding data for the business database. Furthermore, the data to be migrated is stored in the target transcoding library, which is located in the same database cluster as the business database. This allows the data transcoding operation to be performed within the same database cluster, achieving the effect of completing the data transcoding operation within the database cluster and ensuring that the transcoded data is anomaly-free.

[0108] Optionally, in the data migration apparatus provided in this application embodiment, the second parsing unit 44 includes: a parsing module, used to parse the format of the data to be migrated to obtain the format information of the data to be migrated, wherein the format information includes at least one of the following: field name, field attribute, format, alignment, and padding of the data to be migrated; a generation module, used to generate configuration information for each field in the data to be migrated according to the format information, wherein the configuration information includes: field length and encoding method; and a storage module, used to generate a configuration file for the data to be migrated according to the configuration information and the format information, and store the target configuration data in the configuration file in an external table according to a preset mapping relationship, wherein the preset mapping relationship is used to characterize the conversion relationship between the data type of the target configuration data and the data type corresponding to the business database.

[0109] Optionally, in the data migration apparatus provided in this application embodiment, the first storage module includes: a determining submodule, used to determine the data type of the target configuration data and obtain the target data type; an obtaining submodule, used to obtain the data reading type of the business library and obtain the mapping relationship between the data reading type and the target data type to obtain a preset mapping relationship; and a storage submodule, used to convert the data type of the target configuration data into the data reading type according to the preset mapping relationship and store the converted target configuration data in an external table.

[0110] Optionally, in the data migration apparatus provided in this application embodiment, the transcoding unit 45 includes: a splitting module, used to split the data to be migrated into multiple groups of field data according to the target configuration data; an acquisition module, used to acquire a first encoding standard for the first type of encoded data, acquire a second encoding standard for the second type of encoded data, and acquire the conversion relationship between the first encoding standard and the second encoding standard; and a transcoding module, used to perform transcoding operations on each group of field data according to the conversion relationship to obtain the target data.

[0111] Optionally, in the data migration apparatus provided in this application embodiment, the apparatus further includes: an identification unit, used to identify coded data in the data to be migrated, and to perform legality verification on the coded data to obtain a verification result; a detection unit, used to detect the operating environment of the database cluster when the verification result is normal, and to obtain a detection result; and an execution unit, used to execute the step of determining a target transcoding library for storing the first type of data and storing the data to be migrated in a temporary table in the target transcoding library when the detection result is normal.

[0112] Optionally, in the data migration apparatus provided in this application embodiment, the first parsing unit 43 includes: an identification module, used to identify the table structure of the temporary table to obtain the table structure; or, an analysis module, used to obtain the target configuration data of the data to be migrated, and analyze the field attributes in the target configuration data to obtain the table structure.

[0113] Optionally, in the data migration apparatus provided in this application embodiment, the apparatus further includes: a second receiving unit, configured to receive data storage results sent by the business database; and a deletion unit, configured to delete temporary tables and external tables when the data storage results indicate that the target data has been successfully stored.

[0114] The aforementioned data migration device includes a processor and a memory. The first receiving unit 41, the determining unit 42, the first parsing unit 43, the second parsing unit 44, the transcoding unit 45, etc., are all stored in the memory as program units. The processor executes the aforementioned program units stored in the memory to realize the corresponding functions.

[0115] The processor contains a kernel, which retrieves the corresponding program units from memory. One or more kernels can be configured. By adjusting kernel parameters, the problem of anomalies in the converted data after code conversion of fixed-length data in related technologies can be solved.

[0116] The memory may include non-permanent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.

[0117] This invention provides a computer-readable storage medium storing a program thereon, which, when executed by a processor, implements the data migration method.

[0118] This invention provides a processor for running a program, wherein the program executes the data migration method during runtime.

[0119] like Figure 5 As shown, this embodiment of the invention provides an electronic device 50, which includes a processor, a memory, and a program stored in the memory and executable on the processor. When the processor executes the program, it implements the steps of the data migration method described above. The device in this document can be a server, PC, PAD, mobile phone, etc.

[0120] This application also provides a computer program product that, when executed on a data processing device, is adapted to perform the steps of initializing the data migration method described above.

[0121] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0122] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0123] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0124] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0125] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0126] Memory may include non-persistent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, like read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0127] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0128] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0129] The above are merely embodiments of this application and are not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A data migration method, characterized in that, include: Receive the data to be migrated and determine the encoding type of the data to be migrated to obtain a first type; Determine a target transcoding library for storing the first type of data, and store the data to be migrated in a temporary table in the target transcoding library; The table structure of the temporary table is parsed, and a business table is generated in the business database in the database cluster according to the table structure. The table structure of the business table is the same as that of the temporary table. The business database and the target transcoding library are located in the same database cluster. The business database is used to store the second type of encoded data. The configuration data to be migrated is parsed to obtain the target configuration data, and the target configuration data is stored in the external table corresponding to the business database in the database cluster; The data to be migrated is transcoded into the second type of encoded data according to the target configuration data to obtain the target data, and the target data is stored in the business table.

2. The method according to claim 1, characterized in that, Parsing the data to be migrated to obtain the target configuration data, and storing the target configuration data in the external table corresponding to the business database in the database cluster includes: Parse the format of the data to be migrated to obtain the format information of the data to be migrated, wherein the format information includes at least one of the following: field name, field attribute, format, alignment, and padding character of the data to be migrated; Based on the format information, configuration information for each field in the data to be migrated is generated, wherein the configuration information includes: field length and encoding method; The configuration file for the data to be migrated is generated based on the configuration information and the format information, and the target configuration data in the configuration file is stored in the external table according to a preset mapping relationship. The preset mapping relationship is used to characterize the conversion relationship between the data type of the target configuration data and the data type corresponding to the business library.

3. The method according to claim 2, characterized in that, Storing the target configuration data in the configuration file into the external table according to a preset mapping relationship includes: Determine the data type of the target configuration data to obtain the target data type; Obtain the data read type of the business database, and obtain the mapping relationship between the data read type and the target data type to obtain the preset mapping relationship; The data type of the target configuration data is converted to the data reading type according to the preset mapping relationship, and the converted target configuration data is stored in the external table.

4. The method according to claim 1, characterized in that, Based on the target configuration data, the data to be migrated is transcoded into the second type of encoded data to obtain the target data, including: Based on the target configuration data, the data to be migrated is split into multiple sets of field data; Obtain the first encoding standard for the first type of encoded data, obtain the second encoding standard for the second type of encoded data, and obtain the conversion relationship between the first encoding standard and the second encoding standard; The target data is obtained by transcoding each group of field data according to the conversion relationship.

5. The method according to claim 1, characterized in that, After receiving the data to be migrated and determining the encoding type of the data to be migrated to obtain a first type, the method further includes: Identify the encoded data in the data to be migrated, and perform a validity check on the encoded data to obtain the check result; If the verification result is normal, the operating environment of the database cluster is tested to obtain the test result; If the detection result is normal, the step of determining the target transcoding library for storing the first type of data and storing the data to be migrated in a temporary table in the target transcoding library is executed.

6. The method according to claim 1, characterized in that, Parsing the table structure of the temporary table includes: The table structure of the temporary table is identified to obtain the table structure; or, Obtain the target configuration data of the data to be migrated, and analyze the field attributes in the target configuration data to obtain the table structure.

7. The method according to claim 2, characterized in that, After storing the target data in the business table, the method further includes: Receive the data storage results sent by the service library; If the data storage result indicates that the target data has been successfully stored, delete the temporary table and the external table.

8. A data migration device, characterized in that, include: The first receiving unit is used to receive the data to be migrated and determine the encoding type of the data to be migrated to obtain a first type; A determining unit is configured to determine a target transcoding library for storing the first type of data, and store the data to be migrated in a temporary table in the target transcoding library; The first parsing unit is used to parse the table structure of the temporary table and generate a business table in the business library in the database cluster according to the table structure. The table structure of the business table is the same as that of the temporary table. The business library and the target transcoding library are located in the same database cluster. The business library is used to store the second type of encoded data. The second parsing unit is used to parse the configuration data of the data to be migrated, obtain the target configuration data, and store the target configuration data in the external table corresponding to the business database in the database cluster. The transcoding unit is used to transcode the data to be migrated into the second type of encoded data according to the target configuration data to obtain the target data, and store the target data in the business table.

9. A computer storage medium, characterized in that, The computer storage medium is used to store a program, wherein the program, when running, controls the device where the computer storage medium is located to execute the data migration method according to any one of claims 1 to 7.

10. An electronic device, characterized in that, It includes one or more processors and a memory, the memory being used to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors cause the one or more processors to implement the data migration method according to any one of claims 1 to 7.

Images