Data migration method applied to golden DB database and related device

Abstract

The invention discloses a data migration method applied to a golden DB database and a related device, and relates to the field of databases, the data migration method comprises the following steps: obtaining a target name of a txt file and a storage path of the txt file in a source end database from preset configuration information; acquiring data from the txt file which is located in the storage path of the source end database and has the file name of the target name; and loading the data into the data table with the target table name. Therefore, the data migration method when the target end database is the golden DB database is realized.

Description

Technical Field

[0001] The present application relates to the field of database technology, and in particular to a data migration method and related devices applied to a golden DB database. Background Art

[0002] The data in the source database may need to be migrated to the target database. In the related art, the method for migrating the data stored in the source database to the target database includes: the source database converts the data to be migrated into a txt file; the txt file is transmitted to the target database; the target database loads the data in the txt document into the corresponding data table.

[0003] During the research process, the applicant found that if the target database is the goldenDB database, the data migration cannot be achieved through the above-mentioned related technologies because the golden DB database does not allow the source database to transfer txt files to the goldenDB database. Summary of the invention

[0004] In view of the above problems, the present application provides a data migration method and related devices applied to the golden DB database to achieve the purpose of the target. The specific scheme is as follows:

[0005] The first aspect of the present application provides a data migration method applied to a golden DB database, comprising:

[0006] Obtaining a target name of a txt file and a storage path of the txt file in a source database from preset configuration information, wherein the configuration information includes a storage path of the txt file in a source database, a name of the txt file, and a target table name of a data table corresponding to the txt file;

[0007] Get data from a txt file located in the storage path of the source database and named as the target name;

[0008] Load the data into the data table with the target table name.

[0009] In a possible implementation, the step of loading the data into the data table having the target table name includes:

[0010] Load the data into an intermediate table, and store the intermediate table in the golden DB database;

[0011] If all the data in the txt file are loaded into the intermediate table, the data in the intermediate table are loaded into the data table having the target table name.

[0012] In a possible implementation, the method further includes:

[0013] If all the data in the intermediate table are loaded into the data table with the target table name, the data in the intermediate table are cleared.

[0014] In a possible implementation, the method further includes:

[0015] Compare the data in the intermediate table with the data in the txt file;

[0016] If the data in the intermediate table is the same as the data in the txt file, the step of loading the data in the intermediate table into the data table having the target table name is executed.

[0017] In a possible implementation, the method further includes:

[0018] If the amount of data in the intermediate table is the same as the amount of data in the txt file, it is determined that the data in the intermediate table is the same as the data in the txt file.

[0019] In a possible implementation, it also includes:

[0020] receiving a query request, wherein the query request includes a query field and the target table name;

[0021] Feedback the field value of the query field in the data table having the target table name.

[0022] A second aspect of the present application provides a data migration device applied to a golden DB database, comprising:

[0023] A first acquisition module is used to acquire a target name of a txt file and a storage path of the txt file in a source database from preset configuration information, wherein the configuration information includes a storage path of the txt file in the source database, a name of the txt file, and a target table name of a data table corresponding to the txt file;

[0024] A second acquisition module is used to acquire data from a txt file located in the storage path of the source database and named with the target name;

[0025] The first loading module is used to load the data into the data table having the target table name.

[0026] The third aspect of the present application provides a computer program product, including computer-readable instructions. When the computer-readable instructions are executed on an electronic device, the electronic device implements the data migration method applied to the golden DB database according to the above-mentioned first aspect or any implementation of the first aspect.

[0027] A fourth aspect of the present application provides an electronic device, comprising at least one processor and a memory connected to the processor, wherein:

[0028] The memory is used to store computer programs;

[0029] The processor is used to execute the computer program so that the electronic device can implement the data migration method applied to the goldenDB database according to the above-mentioned first aspect or any implementation manner of the first aspect.

[0030] In a fifth aspect, the present application provides a computer storage medium, which carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement the data migration method applied to the golden DB database according to the above-mentioned first aspect or any implementation of the first aspect.

[0031] By means of the above technical solution, the present application provides a data migration method applied to the golden DB database, which obtains the target name of the txt file and the storage path of the txt file in the source database from the preset configuration information; obtains data from the txt file located in the storage path of the source database and whose file name is the target name; and loads the data into the data table with the target table name. Thus, the data migration method is realized when the target database is the golden DB database. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the accompanying drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and the originals and elements are not necessarily drawn to scale.

[0033] Figure 1 A schematic diagram of a system architecture provided for this application;

[0034] Figure 2 A schematic diagram of an optional hardware structure of a terminal provided in this application;

[0035] Figure 3 A schematic diagram of the structure of a server provided for this application;

[0036] Figure 4 A flowchart of a data migration method applied to a golden DB database provided in an embodiment of the present application;

[0037] Figure 5 A schematic diagram of incompatible objects between a source database and a target database provided in an embodiment of the present application;

[0038] Figure 6 A schematic diagram of an implementation of an interface including a JMP configuration provided in an embodiment of the present application;

[0039] Figure 7 A schematic diagram of an interface including DCP configuration provided in an embodiment of the present application;

[0040] Figure 8 A schematic diagram of a user interface for exporting DCP configuration provided in an embodiment of the present application;

[0041] Fig. 9 A schematic diagram of the code modules and business functions provided in the embodiment of the present application to sort out the functional points that need to be modified;

[0042] Fig.10 A schematic diagram of the structure of a data migration device applied to a golden DB database provided in an embodiment of the present application;

[0043] Fig.11 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0044] The following describes the embodiments of the present application in conjunction with the drawings in the embodiments of the present application. The terms used in the implementation method section of the present application are only used to explain the specific embodiments of the present application, and are not intended to limit the present application.

[0045] The embodiments of the present application are described below in conjunction with the accompanying drawings. Those skilled in the art will appreciate that, with the development of technology and the emergence of new scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0046] The terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and need not be used to describe a specific order or sequential order. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances, which is only to describe the distinction mode adopted by the objects of the same attributes when describing in the embodiments of the present application. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, so that the process, method, system, product or equipment comprising a series of units need not be limited to those units, but may include other units that are not clearly listed or inherent to these processes, methods, products or equipment.

[0047] See also Figure 1 , Figure 1A schematic diagram of a system architecture is shown. The system may include a source database 100 and a target database 200. The source database 100 may be located in an electronic device such as a server or a terminal; the target database 200 may be located in an electronic device such as a server or a terminal.

[0048] Exemplarily, the source database 100 may be an Oracle database; and the target database 200 may be a golden DB database.

[0049] Exemplarily, the source-side database 100 may run on a big data server, and the target-side database 200 may run on an application database server.

[0050] Exemplarily, the electronic device running the source database 100 may display an interface, and the electronic device may receive relevant parameters input by the user on the interface, for example, setting the name of the txt file corresponding to the data to be migrated to the target database 200. Exemplarily, the electronic device running the target database 200 may display an interface, and the electronic device may receive relevant parameters input by the user on the interface, for example, configuration information.

[0051] The target-side database 200 can interact with the source-side database 100 based on the configuration information, thereby implementing data migration.

[0052] Exemplarily, the system may further include a query terminal, which runs a mysql client, so that users or R&D personnel or business personnel can query data from the target database 200 through the query terminal.

[0053] The terminal in the embodiment of the present application can be a mobile phone, a tablet computer, a wearable device, a vehicle-mounted device, an augmented reality (AR) / virtual reality (VR) device, a laptop computer, an ultra-mobile personal computer (UMPC), a netbook, a personal digital assistant (PDA), etc., and the embodiment of the present application does not impose any restrictions on this.

[0054] Figure 2 An optional hardware structure diagram of a terminal is shown.

[0055] refer to Figure 2As shown, the terminal may include a radio frequency unit 110, a memory 120, an input unit 130, a display unit 140, a camera 150 (optional), an audio circuit 160 (optional), a speaker 161 (optional), a microphone 162 (optional), an earphone jack 163 (optional), a processor 170, an external interface 180, a power supply 190 and other components. Those skilled in the art will appreciate that Figure 2 This is merely an example of a terminal and does not constitute a limitation on the terminal. The terminal may include more or fewer components than shown in the figure, or a combination of certain components, or different components.

[0056] The input unit 130 can be used to receive input digital or character information, and generate key signal input related to the user settings and function control of the terminal. Specifically, the input unit 130 may include a touch screen 131 (optional) and / or other input devices 132. The touch screen 131 can collect the user's touch operations on or near it (such as the user's operation on or near the touch screen using any suitable object such as fingers, joints, stylus, etc.), and drive the corresponding connection device according to a pre-set program. The touch screen can detect the user's touch action on the touch screen, convert the touch action into a touch signal and send it to the processor 170, and can receive and execute the command sent by the processor 170; the touch signal at least includes the touch point coordinate information. The touch screen 131 can provide an input interface and an output interface between the terminal and the user. In addition, the touch screen can be implemented using multiple types such as resistive, capacitive, infrared and surface acoustic wave. In addition to the touch screen 131, the input unit 130 can also include other input devices. Specifically, other input devices 132 may include, but are not limited to, one or more of a physical keyboard, function keys (such as a volume control key, a switch key, etc.), a trackball, a mouse, a joystick, and the like.

[0057] Among them, the input device 132 can receive input data and the like.

[0058] The display unit 140 can be used to display information input by the user or information provided to the user, various menus of the terminal, interactive interfaces, file display and / or playback of any multimedia file. In the embodiment of the present application, the display unit 140 can be used to display interfaces, processing results, etc.

[0059] The memory 120 can be used to store instructions and data. The memory 120 can mainly include an instruction storage area and a data storage area. The data storage area can store various data, such as multimedia files, texts, etc.; the instruction storage area can store software units such as operating systems, applications, instructions required for at least one function, or their subsets and extensions. It can also include a non-volatile random access memory; provide the processor 170 with hardware, software and data resources including management of computing and processing equipment, and support control software and applications. It is also used for the storage of multimedia files, and the storage of running programs and applications.

[0060] The processor 170 is the control center of the terminal. It uses various interfaces and lines to connect various parts of the entire terminal. By running or executing instructions stored in the memory 120 and calling data stored in the memory 120, it executes various functions of the terminal and processes data, thereby controlling the terminal device as a whole. Optionally, the processor 170 may include one or more processing units; preferably, the processor 170 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface and application program, and the modem processor mainly processes wireless communication. It is understandable that the above-mentioned modem processor may not be integrated into the processor 170. In some embodiments, the processor and the memory may be implemented on a single chip, and in some embodiments, they may also be implemented separately on separate chips. The processor 170 may also be used to generate corresponding operation control signals, send them to corresponding components of the computing and processing device, read and process data in the software, especially read and process data and programs in the memory 120, so that each functional module therein performs corresponding functions, thereby controlling the corresponding components to act according to the requirements of the instructions.

[0061] The radio frequency unit 110 (optional) can be used for receiving and sending information or receiving and sending signals during a call, for example, after receiving the downlink information of the base station, it is sent to the processor 170 for processing; in addition, the designed uplink data is sent to the base station. Generally, the RF circuit includes but is not limited to an antenna, at least one amplifier, a transceiver, a coupler, a low noise amplifier (Low Noise Amplifier, LNA), a duplexer, etc. In addition, the radio frequency unit 110 can also communicate with network devices and other devices through wireless communication. The wireless communication can use any communication standard or protocol, including but not limited to Global System of Mobile communication (Global System of Mobile communication, GSM), General Packet Radio Service (General Packet Radio Service, GPRS), Code Division Multiple Access (Code Division Multiple Access, CDMA), Wideband Code Division Multiple Access (Wideband Code Division Multiple Access, WCDMA), Long Term Evolution (Long Term Evolution, LTE), email, Short Messaging Service (SMS), etc.

[0062] In this embodiment of the present application, the RF unit 110 can send data to a server and receive a processing result sent by the server.

[0063] It should be understood that the radio frequency unit 110 is optional and can be replaced by other communication interfaces, such as a network port.

[0064] The terminal also includes a power supply 190 (such as a battery) for supplying power to various components. Preferably, the power supply can be logically connected to the processor 170 through a power management system, so that the power management system can manage charging, discharging, power consumption and other functions.

[0065] The terminal also includes an external interface 180, which may be a standard Micro USB interface or a multi-pin connector, and may be used to connect the terminal to communicate with other devices, or to connect a charger to charge the terminal.

[0066] Although not shown, the terminal may also include a flashlight, a wireless fidelity (WiFi) module, a Bluetooth module, sensors with different functions, etc., which will not be described in detail here. Some or all of the methods described below may be applied in the following embodiments. Figure 2 in the terminal shown.

[0067] Figure 3A schematic diagram of the structure of a server is provided, such as Figure 3 As shown, the server includes a bus 201, a processor 202, a communication interface 203 and a memory 204. The processor 202, the memory 204 and the communication interface 203 communicate with each other via the bus 201.

[0068] The bus 201 may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 3 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0069] The processor 202 may be any one or more of a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0070] The memory 204 may include a volatile memory, such as a random access memory (RAM). The memory 204 may also include a non-volatile memory, such as a read-only memory (ROM), a flash memory, a hard disk drive (HDD), or a solid state drive (SSD).

[0071] It should be understood that the above-mentioned terminals and servers can be centralized or distributed devices, and the processors in the above-mentioned terminals and servers (such as processor 170 and processor 202) can be hardware circuits (such as application specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), general-purpose processors, digital signal processors (DSP), microprocessors or microcontrollers, etc.), or a combination of these hardware circuits. For example, the processor can be a hardware system with an instruction execution function, such as a CPU, DSP, etc., or a hardware system without an instruction execution function, such as an ASIC, FPGA, etc., or a combination of the above-mentioned hardware systems without an instruction execution function and hardware systems with an instruction execution function.

[0072] The data in the source database may need to be migrated to the target database. In the related art, the method for migrating the data stored in the source database to the target database includes: the source database converts the data to be migrated into a txt file; the txt file is transferred to the target database; the target database loads the data in the txt document into the corresponding data table. If the target database is a golden DB database, the data cannot be migrated through the above-mentioned related technology because the golden DB database does not allow the source database to transfer the txt file to the golden DB database.

[0073] In order to solve the above problems, the embodiment of the present application provides a data migration method applied to the golden DB database. The data migration method applied to the golden DB database of the embodiment of the present application is described in detail below with reference to the accompanying drawings.

[0074] Reference Figure 4 , Figure 4 A flow chart of a data migration method applied to a golden DB database provided in an embodiment of the present application is shown as follows: Figure 4 As shown, a data migration method applied to a golden DB database provided in an embodiment of the present application may include steps S401 to S403, and these steps are described in detail below.

[0075] Step S401: Obtain the target name of the txt file and the storage path of the txt file in the source database from the preset configuration information.

[0076] The configuration information includes the storage path of the txt file in the source database, the name of the txt file, and the target table name of the data table corresponding to the txt file stored in the target database.

[0077] The source database stores data that needs to be migrated to the target database. You need to set the configuration information in the target database in advance; set the target name of the txt file that the data needs to be converted into in the source database, so that after the source database converts the data into a txt file, the target name can be used as the name of the txt file.

[0078] Exemplarily, at the same time, the source database may store multiple txt files, and the file names of different txt files are different. Exemplarily, the storage paths of different txt files may be the same or different. Exemplarily, the target table names of the data tables corresponding to different txt files are different.

[0079] Golden DB is a high-performance, high-reliability distributed database system that is widely used in large-scale data storage and processing scenarios. If the source database and the target database are of different database types, for example, the source database 100 is an Oracle database and the target database 200 is a golden DB database, the target database needs to be modified for compatibility.

[0080] Exemplarily, the application database server runs a migration assessment tool, and the transformation environment is confirmed by the migration assessment tool. The confirmation process includes but is not limited to the following steps: confirming the environment of the target database and confirming the compatibility of the environment of the source database with the environment of the target database. The application database server is different from the electronic device running the source database and the electronic device running the target database. The environment includes but is not limited to: UAT (User Acceptance Test, user acceptance test environment) environment, production environment.

[0081] Exemplarily, the migration assessment tool may be a capture analysis conversion tool (cactool).

[0082] Exemplarily, the application database server is also used to confirm whether the amount of data in the target database is consistent with the production environment.

[0083] Exemplarily, the application database server performs compatibility analysis on the source database and the target database, and issues a compatibility analysis report, which includes an object compatibility overview and a summary of incompatible items. The R&D personnel make corrections to the incompatible objects according to the compatibility analysis report. If the source database is an Oracle database and the target database is a golden DB database, the object compatibility overview of the source database and the target database is shown in Table 1.

[0084] Exemplarily, the objects mentioned in this application include, but are not limited to: one or more of: data tables, indexes, functions, stored procedures, sequences, and views.

[0085] Table 1

[0086] Object Type Total number Number of compatible objects Number of incompatible objects Compatibility Function 2 0 2 0.00 procedure 40 0 40 0.00 table 68 68 0 100.00 view 3 0 3 0.00 TOTALCOMPATIBILITY 113 68 45 60.18

[0087] Table 1 is an overview of the object compatibility of the source database and the target database. Figure 5 The incompatible items are summarized as shown, and the specific contents of the incompatible objects between the source database and the target database are determined, and then changes are made. For example, R&D personnel can Figure 5 Click the "Details" button in the dialog box to view the specific contents of the incompatible objects between the source and target databases, and then make changes.

[0088] If the compatibility between the source database and the target database is 100% or the incompatible objects have been rectified, you can migrate the data tables, indexes, functions, stored procedures, sequences, views and other objects and data of the source database to the golden DB database.

[0089] Step S402: Acquire data from a txt file located in the storage path of the source database and having the target name as its file name.

[0090] Step S403: Load the data into the data table having the target table name.

[0091] It is understandable that the source database exports the data to be migrated into a txt file.

[0092] In the related art, the txt file is stored in the target database, so the process of the target database loading the data in the txt file into the data table is ETL local loading. In this application, the txt file is stored in the source database, so the process of the target database loading the data in the txt file into the data table is ETL remote loading.

[0093] In the related art, the process of "the source database exporting the data to be migrated into a txt file" is called the Export process, the process of "transferring the txt file to the target database" is called the Transfer process, and the process of "loading the data in the txt file into the data table" is called the Load process. Among them, the Export process, the Transfer process and the Load process are ETL processes. The Golden DB database is a database cloud mode service, which does not allow txt files to be placed locally in the Golden DB database. Therefore, there is no Transfer process in the embodiment of the present application, which reduces the Transfer process and improves the efficiency of the ETL process.

[0094] In order for those skilled in the art to better understand the changes in ETL when the target database is the Golden DB database, the following example is used to illustrate. Assuming that the target database of this application is changed from the Oracle database to the Golden DB database, the ETL process when the target database is the Oracle database and the ETL process when the target database is the Golden DB database are shown in Table 2.

[0095] Table 2

[0096]

[0097] It is understandable that if the target database is an Oracle database, since the Oracle database is a physical machine database, it can store txt files, so the txt files can be transferred to the target database. However, the Golden DB database is a database cloud mode service, which does not allow txt files to be placed locally in the Golden DB database. Therefore, there is no Transfer process in the embodiment of the present application, which reduces the Transfer process and improves the efficiency of the ETL process.

[0098] The sqlload configuration includes the mapping between the fields of the data in the txt file and the fields of the data table stored in the target database.

[0099] The goldenDB client is used to perform the local loading operation of loading data from txt files into the data table.

[0100] The loader tool is a remote data migration tool for GoldenDB, which is suitable for scenarios such as data backup, business migration, and data batch processing. For example, the loader tool is used to connect to the golden DB database and execute SQL (Structured Query Language) statements in batches.

[0101] The goldenDB client is used to perform remote loading operations to load data from txt files into data tables, for example, dbtool.

[0102] Exemplarily, the DCP (data configuration platform) configuration information is the configuration information in step S401.

[0103] It can be understood that in order to enable the target database to realize the above functions, the following software and programs need to be deployed on the target database. The process includes the first step to the tenth step.

[0104] Step 1: Create a corresponding golden user on the target database. For example, use the root user to execute the following command to create the golden user:

[0105] #Create a user

[0106] useradd golden

[0107] #Set the golden user password to nbcb_4286

[0108] echo"nbcb_4286"|passwd--stdin golden.

[0109] Step 2: Upload the loader tool, loader_redhat.zip, to the target database.

[0110] Unzip loader_redhat.zip on the target database, edit etc / loader.ini after decompression, and configure the db_conn_info parameter.

[0111] Example: db_conn_info=34:12.103.110.45:8886 34 is the clusterid, and 12.103.110.45:8886 is the IP port.

[0112] Step 3: Set up a query terminal for the target database. For example, the query terminal can run a MySQL client. For example, run the command: yum install-y mariadb to install the MySQL client. After the installation is complete, you can start the mariadb service by running systemctl start mariadb, and set it to start automatically when the system starts by running systemctl enablemariadb.

[0113] Step 4: Create a shell script to execute the ETL loading process. The shell script may contain commands to call ETL tools, such as SQL*Loader, Kettle, or other ETL tools.

[0114] Exemplarily, creating a shell script includes: uploading chkflg_icm_jmp.sh, common.sh, execsp_icm_jmp.sh, and sqlload2.sh files to the home / golden / etlbin directory using the root user; modifying the file ownership and authorization, for example: chown-R golden:golden / etlbin; chmod 777-R / etlbin / *; chmod777-R / etldata / *; ETL loading configuration file description, including but not limited to:

[0115] common.sh / / Shell script to connect to golden DB database client (the test environment does not have f5, so change it when it is produced / / f5 ip)

[0116] chkflg_icm_jmp.sh / / txt file and intermediate table flg data volume verification script

[0117] sqlload2.sh / / sqlload loading script

[0118] execsp_icm_jmp.sh / shell script that executes the stored procedure.

[0119] Step 5: Modify the JMP (job manager platform) configuration and DCP configuration in the target database.

[0120] Exemplarily, JMP configuration includes, but is not limited to, environment variables, dependent jobs, mutually exclusive jobs, and cross-system dependent jobs.

[0121] like Figure 6 As shown, it is a schematic diagram of an implementation method of an interface including JMP configuration provided in an embodiment of the present application.

[0122] like Figure 6 As shown, environment variables include but are not limited to: command, parm1_1, parm1_2, script1. Figure 6 The interface shown is used to modify and view environment variables, dependent jobs, mutually exclusive jobs, and cross-system dependent jobs.

[0123] Exemplarily, DCP configuration includes but is not limited to: environment variables, dependent jobs, mutually exclusive jobs, and cross-system dependent jobs; wherein, environment variables include but are not limited to: the path of the shell script.

[0124] like Figure 7 As shown, it is a schematic diagram of an interface including DCP configuration provided in an embodiment of the present application.

[0125] like Figure 7 As shown, the environment variables in the DCP configuration include but are not limited to: script1, script2, parm1_1, parm1_2, parm1_3, parm2_1, parm2_2, parm2_3.

[0126] Step 6: Export DCP configuration.

[0127] In order for those skilled in the art to better understand how to derive the DCP configuration, an example is given below to illustrate.

[0128] like Figure 8 As shown, it is a schematic diagram of the user interface for exporting DCP configuration provided in an embodiment of the present application.

[0129] like Figure 8 As shown in Figure 8 By writing or selecting the corresponding parameters in the interface shown, the DCP configuration can be exported.

[0130] The embodiment of the present application provides a data migration method applied to the golden DB database, which obtains the target name of a txt file and the storage path of the txt file in the source database from preset configuration information; obtains data from the txt file located in the storage path of the source database and named as the target name; and loads the data into the data table with the target table name. Thus, the data migration method is implemented when the target database is the golden DB database.

[0131] It is understandable that the process of ETL remotely loading the data in the txt file stored in the source database into the data table stored in the target database is very time-consuming. For example, if the txt file includes 500 data, then the time to load the data in the txt file into the data table may be as long as 5 minutes. If in the process of loading the data in the txt file into the data table, the user queries the data in the data table through the query terminal, the query time is different, and the query data may be different; for example, after loading for 1 minute, the data A111 to A113 corresponding to field 11 in the txt file are loaded into the data table. At this time, if the user queries the data of field 11, 3 data such as data A111 to data A113 can be obtained; after loading for 2 minutes, the data A111 to A120 corresponding to field 11 in the txt file have been loaded into the data table. At this time, if the user queries the data of field 11, 10 data such as data A111 to data A120 can be obtained. In order to avoid the above situation, the present application also provides the following method, which includes the following steps A1 to A2. This process is a transformation of the stored procedure of the golden DB database.

[0132] Step A1: Load the data into the intermediate table.

[0133] Exemplarily, before loading data into the intermediate table, no data is stored in the intermediate table.

[0134] For example, the data in the txt file can be loaded into the intermediate table through the JMP batch run of the target database.

[0135] Step A2: If all the data in the txt file are loaded into the intermediate table, load the data in the intermediate table into the data table with the target table name.

[0136] Exemplarily, if the target database is a golden DB database, the JMP of the target database calls a java interface to load the data in the intermediate table into the data table.

[0137] Exemplarily, if the target database is an Oracle database, the target database loads the data in the local txt file into the data table.

[0138] Exemplarily, if all the data in the intermediate table are loaded into the data table having the target table name, the data in the intermediate table are cleared.

[0139] Exemplarily, the method further includes: receiving a query request, the query request including a query field and the target table name; and feeding back a field value of the query field in the data table having the target table name.

[0140] It is understandable that both the intermediate table and the data table are located in the target database, so the process of loading the data in the intermediate table into the data table takes a very short time. For example, if the intermediate table includes 500 data, it takes 1 or 2 seconds to migrate all the data in the intermediate table to the data table. In this way, when a user queries the data in the data table through the query terminal, the query time is different and the data obtained by the query may be different.

[0141] Golden DB database is not compatible with stored procedures, and the stored procedures are executed inside the target database. For the query terminal that calls it, its internal logic is opaque, which may make it difficult to track problems. Therefore, this time, the golden DB database is transformed to de-stored procedures and the data processing logic in the stored procedures is migrated to the application side. For example, the data table in the target database is provided to the Java backend.

[0142] For example, de-storing is an important means to improve the maintainability and scalability of the GoldenDB system. By separating the business logic from the stored procedure, the coupling of the system can be reduced and the maintainability and scalability of the code can be improved.

[0143] In an optional implementation, data may be lost during the process of migrating data in a txt file to an intermediate table. Based on this, an embodiment of the present application also provides the following method: comparing the data in the intermediate table with the data in the txt file; if the data in the intermediate table is the same as the data in the txt file, executing the step of loading the data in the intermediate table into the data table having the target table name.

[0144] Exemplarily, all the data in the intermediate table can be hashed to obtain a first hash value; all the data in the txt file can be hashed to obtain a second hash value. If the first hash value and the second hash value are the same, it means that the data in the intermediate table is the same as the data in the txt file.

[0145] Exemplarily, if the amount of data in the intermediate table is the same as the amount of data in the txt file, it is determined that the data in the intermediate table is the same as the data in the txt file.

[0146] Exemplarily, if loading the data in the intermediate table into the data table fails, a retry is required. Exemplarily, when the number of retries exceeds a preset value, JMP reports an exception error.

[0147] In an optional implementation, the Golden DB database storage considerations include but are not limited to the following:

[0148] If the source database is connected to the target database for the first time or there is no infauser user in production, you need to provide the following information or operations (the following are all for the production environment): the environment of the source database, such as providing a domain name. If there is no domain name, please apply for a domain name for the database IP; create a new source database user infauser / Infauser,4286 to have query permissions for the tables in the source database; you need to open the network rule for the data bus server group 11 / 21.99.99.234-240 to access the source database port 1521, without the need for a long connection. The source database port 1521 is for an Oracle database. If the source database is not an Oracle database, change it to the corresponding database port; the test environment performs a golden DB warehousing performance test.

[0149] In an optional implementation, the application side needs to be modified, and the modification process is as follows: connection driver modification and modification of Apollo configuration to connect to the golden DB database.

[0150] For example, the code for connecting the driver transformation is as follows:

[0151]

[0152]

[0153] For example, the Apollo configuration is modified, and the code for connecting to the golden DB database is changed from code 1 to code 2.

[0154] Code 1:

[0155] spring.datasource.driver-class-name=oracle.jdbc.driver.OracleDriver

[0156] spring.datasource.druid.url =jdbc:oracle:thin:@12.99.220.206:1521:orcl

[0157] spring.datasource.druid.username=pmc

[0158] spring.datasource.druid.password=ENC(BUvHMYuawuwh+G1HKXs7EfiOlnJ0UtlK)

[0159] Code 2:

[0160] spring.datasource.driver-class-name=com.mysql.jdbc.Driver

[0161] spring.datasource.druid.url =jdbc:mysql: / / 12.103.68.45:8886 / pmc? characterEncoding=UTF-8

[0162] spring.datasource.druid.username=daifa

[0163] spring.datasource.druid.password=Nbcb,4286.

[0164] In an optional implementation, the SQL compatibility of the application side also needs to be modified. The underlying Golden DB database is a MySQL database, which is grammatically incompatible with the Oracle database. Therefore, the functional points that need to be modified are sorted out according to the code module and business function, as follows Fig. 9 shown.

[0165] In summary, in order to migrate the data in the source database to the Golden DB database in the embodiment of the present application, it is necessary to perform migration standardization actions on the Golden DB database as shown in Table 3.

[0166] Table 3

[0167]

[0168] The above introduces a data migration method applied to the golden DB database provided in an embodiment of the present application. The following will introduce a device for executing the above data migration method applied to the golden DB database.

[0169] See also Fig.10 , Fig.10 A schematic diagram of a data migration device applied to a golden DB database provided in an embodiment of the present application. Fig.10 As shown, the data migration device applied to the golden DB database includes:

[0170] The first acquisition module 601 is used to acquire the target name of the txt file and the storage path of the txt file in the source database from preset configuration information, wherein the configuration information includes the storage path of the txt file in the source database, the name of the txt file and the target table name of the data table corresponding to the txt file;

[0171] The second acquisition module 602 is used to acquire data from a txt file located in the storage path of the source database and named with the target name;

[0172] The first loading module 603 is used to load the data into the data table having the target table name.

[0173] In an optional implementation, the first loading module includes:

[0174] A first loading unit, used to load the data into an intermediate table, wherein the intermediate table is stored in the golden DB database;

[0175] The second loading unit is used to load the data in the intermediate table into the data table with the target table name if all the data in the txt file are loaded into the intermediate table.

[0176] In an optional implementation, the method further includes:

[0177] The clearing module is used to clear the data in the intermediate table if all the data in the intermediate table is loaded into the data table with the target table name.

[0178] In an optional implementation, the method further includes:

[0179] A comparison module, used to compare the data in the intermediate table with the data in the txt file;

[0180] A trigger module is used to trigger the second loading unit if the data in the intermediate table is the same as the data in the txt file.

[0181] In an optional implementation, the method further includes:

[0182] A determination module is used to determine that the data in the intermediate table is the same as the data in the txt file if the data amount of the data in the intermediate table is the same as the data amount of the data in the txt file.

[0183] In an optional implementation, it also includes:

[0184] A receiving module, used for receiving a query request, wherein the query request includes a query field and the target table name;

[0185] A feedback module is used to feed back the field value of the query field in the data table having the target table name.

[0186] The present application also provides an electronic device in an embodiment. Fig.11As shown, it shows a schematic diagram of the structure of an electronic device suitable for implementing the embodiments of the present application. The electronic device in the embodiments of the present application may include but is not limited to fixed terminals such as mobile phones, notebook computers, PDAs (personal digital assistants), PADs (tablet computers), desktop computers, etc. Fig.11 The electronic device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0187] like Fig.11 As shown, the electronic device may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 701, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 702 or a program loaded from a storage device 708 into a random access memory (RAM) 703. When the electronic device is powered on, various programs and data required for the operation of the electronic device are also stored in the RAM 703. The processing device 701, the ROM 702, and the RAM 703 are connected to each other via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0188] Typically, the following devices may be connected to the I / O interface 705: an input device 706 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 707 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 708 including, for example, a memory card, a hard disk, etc.; and a communication device 709. The communication device 709 may allow the electronic device to communicate with other devices wirelessly or by wire to exchange data. Although Fig.11 An electronic device having various devices is shown, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0189] Also provided in an embodiment of the present application is a computer program product including computer-readable instructions. When the computer-readable instructions are executed on an electronic device, the electronic device implements any one of the data migration methods applied to the golden DB database provided in the embodiments of the present application.

[0190] A computer-readable storage medium is also provided in an embodiment of the present application. The storage medium carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement any data migration method applied to the golden DB database provided in the embodiment of the present application.

[0191] It should also be noted that the device embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed over multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. In addition, in the drawings of the device embodiments provided by the present application, the connection relationship between the modules indicates that there is a communication connection between them, which may be specifically implemented as one or more communication buses or signal lines.

[0192] Through the description of the above implementation mode, the technicians in the field can clearly understand that the present application can be implemented by means of software plus necessary general hardware, and of course, it can also be implemented by special hardware including special integrated circuits, special CPUs, special memories, special components, etc. In general, all functions completed by computer programs can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be various, such as analog circuits, digital circuits or special circuits. However, for the present application, software program implementation is a better implementation mode in more cases. Based on such an understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a readable storage medium, such as a computer floppy disk, a U disk, a mobile hard disk, a ROM, a RAM, a disk or an optical disk, etc., including a number of instructions to enable a computer device (which can be a personal computer, a training device, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0193] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product.

[0194] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a website site, a computer, a training device, or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, training device, or data center. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device, a data center, etc. that includes one or more available media integrations. The available medium may be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)), etc.

Claims

1. A data migration method applied to golden DB database, characterized in that: Applied to the golden DB database, the data migration method applied to the golden DB database includes: Obtaining a target name of a txt file and a storage path of the txt file in a source database from preset configuration information, wherein the configuration information includes a storage path of the txt file in a source database, a name of the txt file, and a target table name of a data table corresponding to the txt file; Get data from a txt file located in the storage path of the source database and named as the target name; Load the data into the data table with the target table name.

2. The data migration method applied to the golden DB database according to claim 1, characterized in that: The step of loading the data into the data table having the target table name comprises: Load the data into an intermediate table, and store the intermediate table in the golden DB database; If all the data in the txt file are loaded into the intermediate table, the data in the intermediate table are loaded into the data table having the target table name.

3. The data migration method applied to the golden DB database according to claim 2 is characterized in that: Also includes: If all the data in the intermediate table are loaded into the data table with the target table name, the data in the intermediate table are cleared.

4. The data migration method applied to the golden DB database according to claim 2, characterized in that: Also includes: Compare the data in the intermediate table with the data in the txt file; If the data in the intermediate table is the same as the data in the txt file, the step of loading the data in the intermediate table into the data table having the target table name is executed.

5. The data migration method applied to the golden DB database according to claim 4 is characterized in that: Also includes: If the amount of data in the intermediate table is the same as the amount of data in the txt file, it is determined that the data in the intermediate table is the same as the data in the txt file.

6. The data migration method applied to the golden DB database according to any one of claims 2 to 5, characterized in that: Also includes: receiving a query request, wherein the query request includes a query field and the target table name; Feedback the field value of the query field in the data table having the target table name.

7. A data migration device applied to golden DB database, characterized in that: Applied to the golden DB database, the data migration device applied to the golden DB database includes: A first acquisition module is used to acquire a target name of a txt file and a storage path of the txt file in a source database from preset configuration information, wherein the configuration information includes a storage path of the txt file in the source database, a name of the txt file, and a target table name of a data table corresponding to the txt file; A second acquisition module is used to acquire data from a txt file located in the storage path of the source database and named with the target name; The first loading module is used to load the data into the data table having the target table name.

8. A computer program product, characterized in that It comprises computer-readable instructions, and when the computer-readable instructions are executed on an electronic device, the electronic device implements the data migration method applied to the golden DB database as claimed in any one of claims 1 to 6.

9. An electronic device, characterized in that: The method comprises at least one processor and a memory connected to the processor, wherein: The memory is used to store computer programs; The processor is used to execute the computer program so that the electronic device can implement the data migration method applied to the golden DB database as described in any one of claims 1 to 6.

10. A computer storage medium, characterized in that: The storage medium carries one or more computer programs, and when the one or more computer programs are executed by an electronic device, the electronic device can implement the data migration method applied to the golden DB database as described in any one of claims 1 to 6.

Images