Distributed cluster data migration method and device

Abstract

The embodiment of the present application provides a distributed cluster data migration method and device, which relates to the field of big data processing and can also be used in the financial field. The method includes: transforming the data to be migrated in the source database according to preset data transformation rules, and extracting the data to be migrated after the data transformation into the target table of the source database, wherein the target table is mapped to the corresponding transfer machine through the data transmission port and IP address defined in the preset outer table; extracting the data in the target table into the outer table, and generating a transfer machine data file, so that the scheduling server migrates the transfer machine data file to the corresponding target database according to the directory parameters in the transfer machine; the present application can effectively improve the data migration efficiency and save system and human resources.

Description

Technical Field

[0001] The present application relates to the field of distributed technology and can also be used in big data processing or the financial field, and specifically relates to a distributed cluster data migration method and device. Background Art

[0002] In the context of big data distributed cluster data migration, there are currently several difficulties in the data migration scenario of large-scale data import / export:

[0003] First, data migration involves the interaction between the production and test environments, which requires a change application, resulting in a longer process.

[0004] Second, the traditional ORACLE database uses the dump method, which uses the client tool programs EXP and IMP. Based on production control requirements, data needs to be transformed before exporting. The general process is "data transformation" → "dump export file" → "transfer" → "dump import file". When dumping files, someone needs to pay attention to the dump process, discover and solve errors in time, and consume manpower; the exported files are stored on the scheduling server, which will occupy storage resources and affect the operation of other programs; the file transfer rate is low.

[0005] Third, when traditional distributed databases perform data migration, they need to send import / export tasks from the data management node to the data node and receive the processing results, which puts enormous pressure on the management node. Summary of the invention

[0006] In view of the problems in the prior art, the present application provides a distributed cluster data migration method and device, which can effectively improve the data migration efficiency and save system and human resources.

[0007] In order to solve at least one of the above problems, the present application provides the following technical solutions:

[0008] In a first aspect, the present application provides a distributed cluster data migration method, comprising:

[0009] Transform the data to be migrated in the source database according to the preset data transformation rules, and extract the data to be migrated after the data transformation into the target table of the source database, wherein the target table is mapped to the corresponding transfer machine through the data transmission port and IP address defined in the preset outer table;

[0010] The data in the target table is extracted to the outer table, and a transfer machine data file is generated, so that the scheduling server migrates the transfer machine data file to the corresponding target database according to the directory parameters in the transfer machine.

[0011] Furthermore, before transforming the data to be migrated in the source database according to the preset data transformation rule, the method further includes:

[0012] According to the location information of the data to be migrated and the data-specific delimiter defined in the preset outer table, the data to be migrated is identified on the source database to obtain the data to be migrated.

[0013] Furthermore, the step of transforming the data to be migrated in the source database according to a preset data transformation rule further includes:

[0014] The to-be-transformed table and the corresponding to-be-transformed fields in the to-be-migrated data that need to be transformed are determined according to the preset data transformation table.

[0015] Further, after determining the to-be-transformed table and the corresponding to-be-transformed field in the to-be-migrated data that need to be transformed according to the preset data transformation table, the method further includes:

[0016] The deformation function corresponding to each of the to-be-deformed fields is determined according to a preset deformation algorithm table and an encryption deformation operation is performed.

[0017] In a second aspect, the present application provides a distributed cluster data migration device, including:

[0018] A data transformation module, used to transform the data to be migrated in the source database according to a preset data transformation rule, and extract the data to be migrated after the data transformation into a target table of the source database, wherein the target table is mapped to a corresponding transfer machine through a data transmission port and an IP address defined in a preset outer table;

[0019] The data migration module is used to extract the data in the target table to the outer table and generate a transfer machine data file so that the scheduling server can migrate the transfer machine data file to the corresponding target database according to the directory parameters in the transfer machine.

[0020] Furthermore, it also includes:

[0021] The data to be migrated identification unit is used to perform a data to be migrated identification operation on the source database according to the data to be migrated location information and data specific delimiters defined in the preset table to obtain the data to be migrated.

[0022] Furthermore, it also includes:

[0023] The to-be-transformed field determining unit is used to determine the to-be-transformed table and the corresponding to-be-transformed field in the to-be-migrated data that need to be transformed according to a preset data transformation table.

[0024] Furthermore, it also includes:

[0025] The deformation function determination unit is used to determine the deformation function corresponding to each of the to-be-deformed fields according to a preset deformation algorithm table and perform an encryption deformation operation.

[0026] In a third aspect, the present application provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the distributed cluster data migration method when executing the program.

[0027] In a fourth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, which implements the steps of the distributed cluster data migration method when executed by a processor.

[0028] It can be seen from the above technical solution that the present application provides a distributed cluster data migration method and device, which maps the target table corresponding to the data to be migrated that has undergone data transformation in the source database to the corresponding transfer machine through the data transmission port and IP address defined in the external table, so that the scheduling server migrates the transfer machine data file to the corresponding target database according to the directory parameters in the transfer machine, thereby avoiding the secondary transmission of the data to be migrated from the table, which can effectively improve the data migration efficiency and save system and human resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0030] Figure 1 A schematic diagram of the process of a distributed cluster data migration method in an embodiment of the present application;

[0031] Figure 2 A structural diagram of a distributed cluster data migration device in an embodiment of the present application;

[0032] Figure 3 This is a schematic diagram of a distributed cluster data migration process in a specific embodiment of the present application;

[0033] Figure 4 This is a schematic diagram of the principle of a distributed cluster data migration system in a specific embodiment of the present application;

[0034] Figure 5 It is a schematic diagram of the structure of an electronic device in an embodiment of the present application. DETAILED DESCRIPTION

[0035] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0036] Considering the problems existing in the existing technology: First, data migration involves the interaction between the production and test environments, and a change application needs to be submitted, resulting in a long process; second, the traditional ORACLE database adopts the dump method, using the client tool programs EXP and IMP. Based on production control requirements, data needs to be transformed before exporting. The general process is "data transformation" → "dump export file" → "transmission" → "dump import file". When dumping files, someone needs to pay attention to the dump process, discover and solve errors in time, and waste manpower; the exported files are stored on the scheduling server, which will occupy storage resources and affect the operation of other programs; the file transfer rate is low; third, when traditional distributed databases perform data migration, it is necessary to send import\export tasks from the data management node to the data node and receive the processing results. The task pressure of the management node is huge. The present application provides a distributed cluster data migration method and device, which maps the target table corresponding to the data to be migrated in the source database after data deformation to the corresponding transfer machine through the data transmission port and IP address defined in the external table, so that the scheduling server migrates the transfer machine data file to the corresponding target database according to the directory parameters in the transfer machine, thereby avoiding the secondary transmission of the data to be migrated from the table, which can effectively improve the data migration efficiency and save system and human resources.

[0037] In order to effectively improve the efficiency of data migration and save system and human resources, the present application provides an embodiment of a distributed cluster data migration method. Figure 1 The distributed cluster data migration method specifically includes the following contents:

[0038] Step S101: transform the data to be migrated in the source database according to preset data transformation rules, and extract the data to be migrated after the data transformation into the target table of the source database, wherein the target table is mapped to the corresponding transfer machine through the data transmission port and IP address defined in the preset outer table.

[0039] Optionally, the data deformation is actually a deformation encryption method, which is to add a deformation function to the sensitive fields in the data to be migrated.

[0040] Specifically, due to the sensitivity of production data, in order to ensure data security and minimize the risk of data extraction to the production database, the present application can separately allocate a dbbackup library in the production database for testing and storing the extracted target data. In addition, the present application can also separately design a data transformation table data_trans_job and a transformation algorithm table data_trans_arith.

[0041] The data transformation table data_trans_job records which fields in which tables in the database need to be encrypted and transformed. The transformation algorithm table data_trans_arith records which transformation functions need to be applied to the relevant sensitive fields.

[0042] After the data deformation, the present application can extract the obtained data file into the target table of the source database, such as the dbbackup library of the production database. Thus, through the collaborative work of the data deformation table data_trans_job and the deformation algorithm table data_trans_arith, when the target data is extracted from the original dwpdata library to the dbbackup library and mapped to the transfer machine directory at the same time, all sensitive data has been encrypted and transformed.

[0043] Optionally, the present application can start the data service tool GDS and pre-create a foreign table, wherein the data file corresponding to the foreign table maps the target table in the dbbackup library to the transfer machine through the IP and port of the data transmission service file defined in the foreign table, and the data under the field of the deformation function is deformed according to the algorithm criteria to generate the transfer machine data file.

[0044] It is understandable that GDS is a distributed framework data service tool that provides data import and export functions and is deployed on a data server. The data server is located outside the database system and is connected to the system through a network. During the parallel data import process, the data in the data file is directly imported and exported to the data node through the external table, thereby realizing the "transformation and extraction" operation of the data to be migrated, thereby improving the efficiency of data migration.

[0045] The appearance stores information such as the location, file format, storage location, encoding format, and separators between data of the data source file, which can help the data service tool identify the data source file.

[0046] Therefore, after the data is extracted into files, data migration can be achieved by simply configuring the relevant information of the transfer machine on the scheduling server of the target data.

[0047] Step S102: extracting the data in the target table into the outer table, and generating a transfer machine data file, so that the scheduling server migrates the transfer machine data file to the corresponding target database according to the directory parameters in the transfer machine.

[0048] Optionally, the method of the present application is preferably applied in a production environment, and can also be used in a test environment. In the test environment, the present application starts a data service tool, creates a foreign table based on the read foreign table field information, performs an insert operation on the foreign table through the business table, and lands the data files in the transfer machine into the database.

[0049] From the above description, it can be seen that the distributed cluster data migration method provided in the embodiment of the present application can map the target table corresponding to the data to be migrated that has undergone data transformation in the source database to the corresponding transfer machine through the data transmission port and IP address defined in the external table, so that the scheduling server can migrate the transfer machine data file to the corresponding target database according to the directory parameters in the transfer machine, thereby avoiding the secondary transmission of the data to be migrated from the table, which can effectively improve the data migration efficiency and save system and human resources.

[0050] In order to accurately identify the data to be migrated from the source database, in an embodiment of the distributed cluster data migration method of the present application, before the above step S101, the following contents may be specifically included:

[0051] According to the location information of the data to be migrated and the data-specific delimiter defined in the preset outer table, the data to be migrated is identified on the source database to obtain the data to be migrated.

[0052] The table stores information such as the location of the data source file, file format, storage location, encoding format, and separators between data, which can help the data service tool identify the data source file and obtain the data to be migrated.

[0053] In order to accurately determine the data that needs to be transformed in the data to be migrated, in an embodiment of the distributed cluster data migration method of the present application, the above step S102 may further specifically include the following content:

[0054] The to-be-transformed table and the corresponding to-be-transformed fields in the to-be-migrated data that need to be transformed are determined according to the preset data transformation table.

[0055] In order to accurately determine the specific deformation function of data deformation, in an embodiment of the distributed cluster data migration method of the present application, the above step S102 may also specifically include the following content:

[0056] The deformation function corresponding to each of the to-be-deformed fields is determined according to a preset deformation algorithm table and an encryption deformation operation is performed.

[0057] Optionally, the data transformation table data_trans_job records which fields in which tables in the database need to be encrypted and transformed. The transformation algorithm table data_trans_arith records which transformation functions need to be applied to the relevant sensitive fields.

[0058] After the data deformation, the present application can extract the obtained data file into the target table of the source database, such as the dbbackup library of the production database. Thus, through the collaborative work of the data deformation table data_trans_job and the deformation algorithm table data_trans_arith, when the target data is extracted from the original dwpdata library to the dbbackup library and mapped to the transfer machine directory at the same time, all sensitive data has been encrypted and transformed.

[0059] In order to effectively improve the data migration efficiency and save system and human resources, the present application provides an embodiment of a distributed cluster data migration device for implementing all or part of the content of the distributed cluster data migration method, see Figure 2 , the distributed cluster data migration device specifically includes the following contents:

[0060] The data deformation module 10 is used to deform the data to be migrated in the source database according to the preset data deformation rules, and extract the data to be migrated after the data deformation into the target table of the source database, wherein the target table is mapped to the corresponding transfer machine through the data transmission port and IP address defined in the preset outer table.

[0061] The data migration module 20 is used to extract the data in the target table to the outer table and generate a transfer machine data file so that the scheduling server can migrate the transfer machine data file to the corresponding target database according to the directory parameters in the transfer machine.

[0062] From the above description, it can be seen that the distributed cluster data migration device provided in the embodiment of the present application can map the target table corresponding to the data to be migrated that has undergone data transformation in the source database to the corresponding transfer machine through the data transmission port and IP address defined in the external table, so that the scheduling server can migrate the transfer machine data file to the corresponding target database according to the directory parameters in the transfer machine, thereby avoiding the secondary transmission of the data to be migrated from the table, which can effectively improve the data migration efficiency and save system and human resources.

[0063] In order to accurately identify the data to be migrated from the source database, in one embodiment of the distributed cluster data migration device of the present application, the following contents are specifically included:

[0064] The data to be migrated identification unit is used to perform a data to be migrated identification operation on the source database according to the data to be migrated location information and data specific delimiters defined in the preset table to obtain the data to be migrated.

[0065] In order to accurately determine the data that needs to be transformed in the data to be migrated, in one embodiment of the distributed cluster data migration device of the present application, the following contents are specifically included:

[0066] The to-be-transformed field determining unit is used to determine the to-be-transformed table and the corresponding to-be-transformed field in the to-be-migrated data that need to be transformed according to a preset data transformation table.

[0067] In order to accurately determine the specific deformation function of data deformation, in one embodiment of the distributed cluster data migration device of the present application, the following contents are specifically included:

[0068] The deformation function determination unit is used to determine the deformation function corresponding to each of the to-be-deformed fields according to a preset deformation algorithm table and perform an encryption deformation operation.

[0069] In order to further illustrate the present solution, the present application also provides a specific application example of using the above-mentioned distributed cluster data migration device to implement the distributed cluster data migration method, see Figure 3 and Figure 4 , specifically including the following contents:

[0070] Step 1: In the production environment, transform the data in the target table and back it up to the dbbackup library of the production database. At the same time, apply the transformation function to the sensitive fields.

[0071] Step 2 starts the data service tool and creates a foreign table. The data file corresponding to the foreign table maps the target table in the dbbackup library to the transfer machine through the IP and port of the data transmission service file defined in the foreign table. The data under the field of the deformation function is deformed according to the algorithm criteria to generate a data file.

[0072] In the step 3 test environment, start the data service tool, create a foreign table based on the read foreign table field information, perform insert operations on the foreign table through the business table, and land the data files in the transfer machine into the database.

[0073] The GDS is a distributed framework data service tool that provides data import and export functions and is deployed on a data server. The data server is located outside the database system and is connected to the system through a network. During the parallel data import process, the data in the data file is directly imported and exported to the data node through the external table.

[0074] Specifically, the table stores information such as the location, file format, storage location, encoding format, and separators between data of the data source file. Its function is to help data service tools identify the data source file.

[0075] Specifically, the transfer machine is the transfer directory of the data files. By configuring the gds parameter -d (directory), the data files exported from the production dbbackup library can be stored in the target directory. The example is as follows:

[0076] . / gds_C80-d / approot1 / mpp / mppedw-p 73.16.161.143:23356-H 0.0.0.0 / 0-l / approot1 / mpp / mppedw / gds_exp6.txt-Dt 4-S 30720GB

[0077] . / gds_C80-d / approot1 / mpp / mppedw-p 73.16.161.143:23357-H 0.0.0.0 / 0-l / approot1 / mpp / mppedw / gds_exp7.txt-Dt 4-S 30720GB

[0078] . / gds_C80-d / approot1 / mpp / mppedw-p 73.16.161.143:23358-H 0.0.0.0 / 0-l / approot1 / mpp / mppedw / gds_exp8.txt-Dt 4-S 30720GB

[0079] In the above statement, gds is configured on the transit machine 73.16.161.143. Currently, there are three ports available: 23356, 23357, and 23358. The data file will be generated in the directory / approot1 / mpp / mppedw, and the gds operation log can be viewed in / approot1 / mpp / mppedw / .

[0080] Similarly, after the data is extracted into files, it is only necessary to configure the relevant information of the transfer machine on the scheduling server of the current data, thereby realizing data migration.

[0081] It is understandable that due to the sensitivity of production data, in order to ensure data security and minimize the risk of data extraction to the production database, a dbbackup library is separately allocated in the production database for testing to store the extracted target data. In addition, the data transformation table data_trans_job and the transformation algorithm table data_trans_arith are designed.

[0082] Specifically, the data transformation table data_trans_job records which fields in which tables in the database need to be encrypted and transformed.

[0083] Specifically, the deformation algorithm table data_trans_arith records which deformation functions need to be applied to the relevant sensitive fields. The following takes the table DWPDATA.T03_MTG_PROJ_INFO_H as an example:

[0084] DWPDATA.T03_MTG_PROJ_INFO_H records that the HOUSE_SELL_LICS_ORG field (d_col_name) in data_trans_job needs to be transformed using algorithm 1 (arith_id). It can be seen that the transformation function for arith_id 1 in data_trans_arith is TransName(trim( <param0>)).

[0085] Therefore, the data transformation table data_trans_job and the transformation algorithm table data_trans_arith work together. When the target data is extracted from the original dwpdata library to the dbbackup library and mapped to the transfer machine directory at the same time, all sensitive data has been encrypted and transformed.

[0086] From the above content, it can be seen that compared with the traditional dump method of Orcacle database, the data service tool GDS of this application supports high concurrency and can increase the transmission rate by about 60%; the data source is directly connected to the data node, and the data file is directly landed in the database through the external table mechanism, avoiding the secondary transmission of data from the table to the ground; the data files are stored on the transfer machine to avoid affecting the performance of the scheduling server; no additional manual monitoring is required during the data migration process, saving human resources.

[0087] From the hardware level, in order to effectively improve the data migration efficiency and save system and human resources, the present application provides an embodiment of an electronic device for implementing all or part of the content of the distributed cluster data migration method, and the electronic device specifically includes the following content:

[0088] Processor, memory, communication interface and bus; wherein the processor, memory and communication interface communicate with each other through the bus; the communication interface is used to realize information transmission between the distributed cluster data migration device and related devices such as the core business system, user terminal and related database; the logic controller can be a desktop computer, a tablet computer and a mobile terminal, etc., but the present embodiment is not limited thereto. In the present embodiment, the logic controller can be implemented with reference to the embodiment of the distributed cluster data migration method and the embodiment of the distributed cluster data migration device in the embodiment, and the contents thereof are incorporated herein, and the repeated parts are not repeated.

[0089] It is understandable that the user terminal may include a smart phone, a tablet electronic device, a network set-top box, a portable computer, a desktop computer, a personal digital assistant (PDA), a vehicle-mounted device, a smart wearable device, etc. Among them, the smart wearable device may include smart glasses, a smart watch, a smart bracelet, etc.

[0090] In practical applications, part of the distributed cluster data migration method can be executed on the electronic device side as described above, or all operations can be completed in the client device. The specific selection can be based on the processing capability of the client device and the limitations of the user's usage scenario. This application does not limit this. If all operations are completed in the client device, the client device may also include a processor.

[0091] The client device may have a communication module (i.e., a communication unit) that can communicate with a remote server to achieve data transmission with the server. The server may include a server on the task scheduling center side, and other implementation scenarios may also include a server on an intermediate platform, such as a server on a third-party server platform that has a communication link with the task scheduling center server. The server may include a single computer device, or a server cluster consisting of multiple servers, or a server structure of a distributed device.

[0092] Figure 5 FIG. 9 is a schematic block diagram of the system structure of the electronic device 9600 according to an embodiment of the present application. Figure 5 As shown, the electronic device 9600 may include a central processor 9100 and a memory 9140; the memory 9140 is coupled to the central processor 9100. It is worth noting that Figure 5 is exemplary; other types of structures may also be used to supplement or replace this structure to implement telecommunication functions or other functions.

[0093] In one embodiment, the distributed cluster data migration method function may be integrated into the central processor 9100. The central processor 9100 may be configured to perform the following control:

[0094] Step S101: transform the data to be migrated in the source database according to preset data transformation rules, and extract the data to be migrated after the data transformation into the target table of the source database, wherein the target table is mapped to the corresponding transfer machine through the data transmission port and IP address defined in the preset outer table.

[0095] Step S102: extracting the data in the target table into the outer table, and generating a transfer machine data file, so that the scheduling server migrates the transfer machine data file to the corresponding target database according to the directory parameters in the transfer machine.

[0096] From the above description, it can be seen that the electronic device provided in the embodiment of the present application maps the target table corresponding to the data to be migrated that has undergone data transformation in the source database to the corresponding transfer machine through the data transmission port and IP address defined in the external table, so that the scheduling server migrates the transfer machine data file to the corresponding target database according to the directory parameters in the transfer machine, thereby avoiding the secondary transmission of the data to be migrated from the table, which can effectively improve the data migration efficiency and save system and human resources.

[0097] In another embodiment, the distributed cluster data migration device may be configured separately from the central processor 9100. For example, the distributed cluster data migration device may be configured as a chip connected to the central processor 9100, and the distributed cluster data migration method function may be implemented under the control of the central processor.

[0098] like Figure 5 As shown, the electronic device 9600 may also include: a communication module 9110, an input unit 9120, an audio processor 9130, a display 9160, and a power supply 9170. It is worth noting that the electronic device 9600 does not necessarily have to include Figure 5 In addition, the electronic device 9600 may also include Figure 5 For components not shown, reference may be made to the prior art.

[0099] like Figure 5 As shown, the central processing unit 9100 is sometimes also referred to as a controller or an operation control, and may include a microprocessor or other processor device and / or logic device. The central processing unit 9100 receives input and controls the operation of various components of the electronic device 9600.

[0100] The memory 9140 may be, for example, one or more of a cache, a flash memory, a hard drive, a removable medium, a volatile memory, a non-volatile memory or other suitable devices. The above-mentioned information related to the failure may be stored, and a program for executing the relevant information may also be stored. The CPU 9100 may execute the program stored in the memory 9140 to implement information storage or processing, etc.

[0101] The input unit 9120 provides input to the central processing unit 9100. The input unit 9120 is, for example, a key or a touch input device. The power supply 9170 is used to provide power to the electronic device 9600. The display 9160 is used to display display objects such as images and texts. The display may be, for example, an LCD display, but is not limited thereto.

[0102] The memory 9140 may be a solid-state memory, such as a read-only memory (ROM), a random access memory (RAM), a SIM card, etc. It may also be a memory that saves information even when the power is off, can be selectively erased, and is provided with more data, examples of which are sometimes referred to as EPROMs, etc. The memory 9140 may also be some other type of device. The memory 9140 includes a buffer memory 9141 (sometimes referred to as a buffer). The memory 9140 may include an application / function storage unit 9142, which is used to store application programs and function programs or processes for executing the operation of the electronic device 9600 through the central processor 9100.

[0103] The memory 9140 may also include a data storage unit 9143 for storing data, such as contacts, digital data, pictures, sounds, and / or any other data used by the electronic device. The driver storage unit 9144 of the memory 9140 may include various drivers for communication functions of the electronic device and / or for executing other functions of the electronic device (such as messaging applications, address book applications, etc.).

[0104] The communication module 9110 is a transmitter / receiver 9110 that sends and receives signals via an antenna 9111. The communication module (transmitter / receiver) 9110 is coupled to the central processor 9100 to provide input signals and receive output signals, which may be the same as the case of a conventional mobile communication terminal.

[0105] Based on different communication technologies, multiple communication modules 9110 may be provided in the same electronic device, such as a cellular network module, a Bluetooth module and / or a wireless LAN module, etc. The communication module (transmitter / receiver) 9110 is also coupled to a speaker 9131 and a microphone 9132 via an audio processor 9130 to provide an audio output via the speaker 9131 and receive an audio input from the microphone 9132, thereby realizing a common telecommunication function. The audio processor 9130 may include any suitable buffer, decoder, amplifier, etc. In addition, the audio processor 9130 is also coupled to the central processor 9100, so that recording can be performed on the local machine through the microphone 9132, and the sound stored on the local machine can be played through the speaker 9131.

[0106] The embodiments of the present application also provide a computer-readable storage medium capable of implementing all steps of the distributed cluster data migration method in the above embodiment, where the execution subject is a server or a client. The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, all steps of the distributed cluster data migration method in the above embodiment are implemented, where the execution subject is a server or a client. For example, when the processor executes the computer program, the following steps are implemented:

[0107] Step S101: transform the data to be migrated in the source database according to preset data transformation rules, and extract the data to be migrated after the data transformation into the target table of the source database, wherein the target table is mapped to the corresponding transfer machine through the data transmission port and IP address defined in the preset outer table.

[0108] Step S102: extracting the data in the target table into the outer table, and generating a transfer machine data file, so that the scheduling server migrates the transfer machine data file to the corresponding target database according to the directory parameters in the transfer machine.

[0109] From the above description, it can be seen that the computer-readable storage medium provided in the embodiment of the present application maps the target table corresponding to the data to be migrated that has undergone data transformation in the source database to the corresponding transfer machine through the data transmission port and IP address defined in the external table, so that the scheduling server migrates the transfer machine data file to the corresponding target database according to the directory parameters in the transfer machine, thereby avoiding the secondary transmission of the data to be migrated from the table, which can effectively improve the data migration efficiency and save system and human resources.

[0110] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, devices, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0111] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (apparatus), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0112] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0113] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0114] The present invention uses specific embodiments to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A distributed cluster data migration method, characterized in that: The method comprises: Transform the data to be migrated in the source database according to the preset data transformation rules, and extract the data to be migrated after the data transformation into the target table of the source database, wherein the target table is mapped to the corresponding transfer machine through the data transmission port and IP address defined in the preset outer table; Extracting the data in the target table to the outer table and generating a transfer machine data file, so that the scheduling server can migrate the transfer machine data file to the corresponding target database according to the directory parameters in the transfer machine; The outer table stores the location, file format, storage location, encoding format, and separators between data of the data source file. The data service tool identifies the data source file through the outer table to obtain the data to be migrated. The step of transforming the data to be migrated in the source database according to a preset data transformation rule also includes: The to-be-transformed table and the corresponding to-be-transformed fields in the to-be-migrated data that need to be transformed are determined according to the preset data transformation table.

2. The distributed cluster data migration method according to claim 1, characterized in that: After determining the to-be-transformed table and the corresponding to-be-transformed field in the to-be-migrated data that need to be transformed according to the preset data transformation table, the method includes: The deformation function corresponding to each of the to-be-deformed fields is determined according to a preset deformation algorithm table and an encryption deformation operation is performed.

3. A distributed cluster data migration device, characterized in that: include: A data transformation module, used to transform the data to be migrated in the source database according to a preset data transformation rule, and extract the data to be migrated after the data transformation into a target table of the source database, wherein the target table is mapped to a corresponding transfer machine through a data transmission port and an IP address defined in a preset outer table; wherein the outer table stores the location, file format, storage location, encoding format, and delimiter of the data source file; A data migration module, used to extract the data in the target table to the outer table and generate a transfer machine data file, so that the scheduling server can migrate the transfer machine data file to the corresponding target database according to the directory parameters in the transfer machine; The distributed cluster data migration device also includes a data service tool that identifies a data source file through the external table to obtain data to be migrated; The distributed cluster data migration device further includes a to-be-transformed field determination unit, which determines a to-be-transformed table and corresponding to-be-transformed fields in the to-be-migrated data that need to be transformed according to a preset data transformation table.

4. The distributed cluster data migration device according to claim 3, characterized in that: Also includes: The deformation function determination unit is used to determine the deformation function corresponding to each of the to-be-deformed fields according to a preset deformation algorithm table and perform an encryption deformation operation.

5. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps of the distributed cluster data migration method described in any one of claims 1 to 2 are implemented.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the distributed cluster data migration method described in any one of claims 1 to 2 are implemented.

Images