A universal data device method based on database replication software development

By introducing data reading, conversion and assembly modules into the database replication software, the differences in different database interfaces are handled uniformly, and the problems of degraded database loading performance are solved, achieving more efficient data loading and development efficiency.

CN118626549BActive Publication Date: 2024-11-05DISJIE (BEIJING) DATA MANAGEMENT TECH CO LTD
View PDF 4 Cites 0 Cited by

Patent Information

Application Number
CN202410962820.2
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-07-18
Publication Date
2024-11-05
Estimated Expiration
2044-07-18

AI Technical Summary

Technical Problem

When existing database replication software loads multiple different databases, it needs to deal with differences between each interface, resulting in complex programming and degradation of performance. Especially in multi-process or multi-threaded mode, the optimal loading performance varies greatly, resulting in data. Loading performance dropped sharply.

Method used

A general data device method based on database replication software is proposed. Through the data reading module, the data conversion module converts data according to the target database type, the data assembly and sending module assembles the data into a stream format, and the data loading module quickly parses and converts it to the format required by the target interface, calls the target data loading task, and unifies different types of database operations by defining the database type and interface.

Benefits of technology

It improves the universality and development efficiency of database replication software, reduces maintenance costs, and data cleaning and performance optimization can be automatically applied to multiple databases during data loading. The performance optimization of the database interface on the specific target end is more convenient and the personnel division of labor is reasonable.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118626549B_ABST
    Figure CN118626549B_ABST
Patent Text Reader

Abstract

The present application proposes a general data device method based on database replication software development, the method comprising: reading the data to be loaded into the memory through the data reading module, wherein the data formats of different types read into the memory are unified into an internal general data expression format during the reading process; parsing the data read into the memory through the data conversion module, and performing data conversion on the parsed data according to the configured conversion rules and the target database type; assembling the converted data into the same stream format data through the data assembly and sending module, and sending it to the data loading module; quickly parsing the stream format data and converting it into the format required by the current environment interface of the target end through the data loading module, and calling the environment interface to obtain the target data loading task. The present application can solve the problem of how to install the data output by the source replication software to different types of databases on the target end or output it to the big data platform environment faster and better.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of database replication, and in particular to a general data device method based on database replication software development. Background Art

[0002] The data files output by the database replication software are usually directly read and parsed, and then the target database interface or the JDBC / ODBC interface of the general database is called to complete the database installation. JDBC / ODBC is a general database loading interface.

[0003] The disadvantage of this method is that each database interface has some differences in usage. If a piece of data needs to be loaded into multiple different databases, it is necessary to find a way to handle the differences between the interfaces separately, which makes the program design more complicated and prone to problems. Different database interfaces will also load applications differently in multi-process mode, which will bring more differences in program structure design. In addition, the interfaces provided by different types of databases are also different, especially in terms of optimal loading performance.

[0004] In addition, multi-process mode also corresponds to multi-thread mode, which are two different operating system internal program running modes. A process is an encapsulation of a runtime program and is the basic unit for resource scheduling and allocation in the system. The memory usage of each process does not interfere with each other. A thread is a subtask of a process, and the memory of multiple threads under a process is shared.

[0005] If the target end is only applicable to 1 to 3 database interfaces, the design and development is relatively simple. When the target end database types increase to 10 or more than 20 types, the probability of problems is very high. Frequent adjustment of the internal applicable model will directly lead to a sharp decline in data installation performance. In addition, different types of database loading interfaces are also very different. Although some databases provide multiple types of database installation interfaces, there is only one device interface with the best performance. In order to improve data installation performance, we must find a way to adapt to the best database installation interface. Summary of the invention

[0006] The present application aims to solve one of the technical problems in the related art at least to some extent.

[0007] To this end, the first objective of the present application is to propose a universal data device method based on database replication software development.

[0008] The second objective of the present application is to provide an electronic device.

[0009] The third object of the present application is to provide a computer-readable storage medium.

[0010] A fourth object of the present application is to provide a computer program product.

[0011] To achieve the above-mentioned purpose, the first embodiment of the present application proposes a general data device method based on database replication software development, including:

[0012] The data to be loaded is read into the memory through the data reading module, wherein during the reading process, the data formats of different types read into the memory are unified into an internal common data expression format;

[0013] The data conversion module is used to parse the data read into the memory, and the parsed data is converted according to the configured conversion rules and the target database type;

[0014] The converted data is assembled into the same stream format data through the data assembly and sending module, and sent to the data loading module;

[0015] The data loading module quickly parses the stream format data and converts it into the format required by the current environment interface of the target end, and calls the environment interface to obtain the target data loading task.

[0016] Optionally, the data loading module can generate a target data loading task that matches the target database according to the target database environment and the data output environment.

[0017] Optionally, parsing the data read into the memory by the data conversion module includes:

[0018] Each field data value of the data read into the memory is parsed, and dictionary information of each data table is obtained according to the field data value. The dictionary information records the storage structure, number of columns, data type of each column, and data length information of each data table.

[0019] Optionally, performing data conversion on the parsed data according to the configured conversion rules and the target database type includes:

[0020] Reading the conversion rules stored in the memory in the form of a binary tree after parsing, and performing type conversion on the fields to be converted according to the conversion rules and the target database type, wherein the type conversion process includes field truncation, field filling with other values, or numerical operation;

[0021] Fill the converted data into the new memory.

[0022] Optionally, the data assembly and sending module is a database loading client interface, and the upper layer calls the database loading client interface to indirectly complete operations on different types of databases. The method further includes:

[0023] By defining database types, unique identifiers are assigned to different types of databases in the database loading client interface;

[0024] By calling the database loading client interface to connect to the proxy database loading service, a basis is provided for subsequent data loading and data query;

[0025] Execute the database SQL submitted externally to the data assembly sending module, and send the execution query SQL to the database;

[0026] The best data loading mode is selected according to different target databases, wherein the data loading mode is loading data by client SQL statement mode, or loading by binding mode, or loading by copy or load mode.

[0027] Optionally, the data communication protocol used when the data assembly and sending module communicates with the data loading module is the TCP / IP protocol, and when sending data, dynamically encrypted network data is used according to the key returned by the database loading client interface.

[0028] To achieve the above-mentioned purpose, a second aspect of the present application provides an electronic device, comprising: a processor, and a memory communicatively connected to the processor;

[0029] The memory stores computer-executable instructions;

[0030] The processor executes the computer-executable instructions stored in the memory to implement the method as described in any one of the first aspects above.

[0031] To achieve the above-mentioned purpose, the third aspect embodiment of the present application proposes a computer-readable storage medium, in which computer-readable storage medium is stored computer execution instructions, and when the computer execution instructions are executed by a processor, they are used to implement the method as described in any one of the above-mentioned first aspects.

[0032] To achieve the above-mentioned purpose, the fourth aspect of the present application proposes a computer program product, including a computer program, which, when executed by a processor, implements the method as described in any one of the above-mentioned first aspects.

[0033] The technical solution provided by the embodiments of the present application brings at least the following beneficial effects:

[0034] The target-side loader of the database replication software developed using the present application has better versatility, especially for the types of target-side databases that need to be supported, which can often greatly improve software development efficiency and reduce software maintenance costs; the general data processing parts such as data cleaning, data conversion, and performance optimization involved in the data loading process can be automatically applied to the various different databases supported by the entire loader after one modification; the interface call part involved in the data loading process and the specific target-side database is extracted separately as an independent module for development, which can make the performance optimization of the interface side for this type of database more convenient, and the division of labor among personnel will be more convenient and reasonable, and they only need to focus on the current database processing.

[0035] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0037] Figure 1 It is a flow chart of a general data device method based on database replication software development according to an embodiment of the present application;

[0038] Figure 2 It is a block diagram of an electronic device. DETAILED DESCRIPTION

[0039] The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present application, and should not be construed as limiting the present application.

[0040] In order to adapt to more types of target-side database environments and some non-database environments, the method proposed in this application extracts a large category of public data device modules from the specific data device or output part as the data loading module in this application. The module is internally named dagent (data loading agent), and the corresponding target data loading task that is finally started is named dagentd (data loading agent daemon). Dagentd is the name of the background data loading service program developed by the dagent module. Dagentd is also the name of the program that is finally submitted to the actual user or engineering personnel for specific use. Dagent is the name of the specific internal data loading service module.

[0041] In one embodiment of the present application, if the target database is an Oracle database, the corresponding data loading service agent program name is dagentd.oracle; if the target database is a MySQL database, the corresponding data loading service agent program name is dagentd.mysql.

[0042] Currently, the data loading module proposed in this application already supports more than 20 dagentd services:

[0043] dagentd.oracle - a service program installed to the Oracle database;

[0044] dagentd.mysql - a server that is installed to the mysql database;

[0045] dagentd.db2 - a service program installed to the db2 database;

[0046] dagentd.postgresql - a server that is installed to the postgresql database;

[0047] dagentd.mssql - a daemon that is installed to the Microsoft SQL Server database;

[0048] dagentd.informix - a daemon installed to the Informix database;

[0049] dagentd.sybase - a service program installed to the Sybase database;

[0050] dagentd.dameng - the service program for the device to reach the dream database;

[0051] dagentd.gbase - a service program installed to the GBase database;

[0052] dagentd.kingbase - service program installed to the KingBase database;

[0053] dagentd.greenplum - a service program installed to the greenplum database;

[0054] dagentd.ob_mysql - a service program installed to Ocean Base for MySQL database;

[0055] dagentd.ob_oracle - a service installed to the Ocean Base for Oracle database;

[0056] dagentd.goldendb - a server that is installed to the GoldenDB database;

[0057] dagentd.tidb - a service program installed to the TiDB database;

[0058] dagentd.xgdb - the service program installed to the Xugu database;

[0059] dagentd.redis - a server that is installed to the redis memory database;

[0060] dagentd.gaussdb - a service program installed to the GaussDB database;

[0061] dagentd.antdb - a server that is installed to the AntDB database;

[0062] dagentd.kdb - a daemon that is installed to the KDB database;

[0063] dagentd.polardb_pg - a daemon installed to the Polardb for PostgreSQL database;

[0064] dagentd.hana - service program installed to HANA database;

[0065] dagentd.mongodb - a server that is installed to the mongodb database;

[0066] dagentd.text - a daemon that converts replicated data into text files;

[0067] dagentd.kafka - a server that replicates data to a large database KAFKA environment;

[0068] It is understandable that, due to the ever-changing target device environment and the more or less differences in the database device interface provided by various database products, the overall goal of data device has not changed. In order to better adapt to this change and invariance, the independent data loading module dagent of this method is to adapt to various database environments and data output environments. The data reading and conversion parts other than the dagent part will remain unchanged overall, and the research and development focus and optimization focus will also be placed on the parts other than the dagent during the actual device development process.

[0069] In addition to dagent, the application also involves several other major modules, namely the data reading module dread (data read), the data conversion module dconvert (data convert), and the database assembly and sending module dagentc (data loading agent client). These major modules cooperate with each other during the data installation process to complete the overall installation task.

[0070] The following describes a general data device method based on database replication software development according to an embodiment of the present application with reference to the accompanying drawings.

[0071] Figure 1 is a flowchart of a general data device method based on database replication software development according to an embodiment of the present application. Figure 1 As shown, the method comprises the following steps:

[0072] Step 101, read the data to be loaded into the memory through the data reading module, wherein during the reading process, the different types of data formats read into the memory are unified into an internal common data expression format.

[0073] In the embodiment of the present application, the data reading module mainly reads the data to be loaded into the memory by calling the operating system interface.

[0074] It should be noted that during the reading process, the data read into the memory is subjected to a preliminary analysis or simple conversion, so that the different types of data formats read into the memory can be converted into an internal common data expression format as much as possible.

[0075] In one embodiment of the present application, data in the memory are stored in a row-column mode, and each row of data can be quickly retrieved by row number. When data is read in, as much data of the same type as the same table is read in at one time as possible.

[0076] In one embodiment of the present application, for data such as "[T1 insert][T1 insert][T2 insert][T2 delete][T1 insert] ...", two insert records of table T1 are read for the data for the first time, and then one insert record of table T2 is read after the loading is completed. After the insertion is completed, one delete record of table T2 is read, and so on... Reading as much data of the same type in the same table as much as possible at one time can improve the overall data loading performance.

[0077] Step 102: The data read into the memory is parsed by a data conversion module, and data conversion is performed on the parsed data according to the configured conversion rules and the target database type.

[0078] In the embodiment of the present application, data conversion is mainly completed after the data is read into the memory.

[0079] First, data analysis is required before data conversion, that is, to parse out the specific data value of each field. Correspondingly, the table structure information of the table, that is, dictionary information, needs to be obtained at the beginning of data retrieval. The dictionary records the storage structure, number of columns, data type of each column, and data length information of each data table. Dictionary management will run through the entire data loading process. Without a dictionary, it is impossible to know the meaning of each field data if there is only data.

[0080] Then, the conversion rules of data conversion are read into the memory at the start of the program. After being parsed, the rules are stored in the memory in the form of a binary tree. When the data is loaded, the conversion rules can be quickly obtained by querying the memory structure. According to the conversion rules and the target database type, the fields to be converted are converted, including truncation of the fields, filling other values, or numerical operations. The converted data is filled into the new memory part, that is, the XYD_BIND structure of the interface defined by the subsequent data assembly and sending module dagentc.

[0081] Step 103, assemble the converted data into data in the same stream format through the data assembly and sending module, and send it to the data loading module.

[0082] In the embodiment of the present application, the data assembly and sending module dagentc mainly communicates with the data loading module dagent part. The data assembly and sending module is a database loading client interface. The upper layer calls the database loading client interface to indirectly complete operations on different types of databases. When copying data, the upper layer passes the data that needs to be installed to the dagentc part. The dagentc part will assemble the data that needs to be installed into a common stream data format and send it to the dagent part. After the dagent part completes the data installation, it returns the status information to dagentc, and dagentc returns the device status information to the upper layer call.

[0083] It is understandable that for the upper layer, the various databases or data platforms below become a single data target after being processed by dagentc / dagent. It is no longer important whether the final target is a database or other data platform. With a unique data loading agent client interface, data loading service development is much simpler, and there is no need to deal with various databases or data components. Data loading tasks can be divided into different people to complete, and people who are proficient in different databases can deeply develop corresponding dagent (data loading agent) modules.

[0084] It should be noted that the data assembly and sending module is a database loading client interface. The upper layer calls the database loading client interface to indirectly complete operations on different types of databases. In order to unify the access interfaces of different types of databases, this application defines the dagentc interface, which can facilitate the overall data loading development. The specific definition is as follows:

[0085] (1) Define the database type interface

[0086] In an implementation example of the present application, by defining the database type, unique identifiers are allocated to different types of databases in the database loading client interface, which is conducive to performing different special processing according to different types of databases in the later stage.

[0087] In a possible embodiment, it is implemented in C language:

[0088] enum XDA_DBT

[0089] {

[0090] XDA_DBT_ORACLE, / * Oracle database* /

[0091] XDA_DBT_DB2,

[0092] .... / * More database type definitions can be added later* /

[0093] };

[0094] (2) Define dagentc's interface for connecting to and closing the server

[0095] Similar to defining database types, implemented in C language:

[0096] extern int dagentc_db_open(dagentc_param *par, DAGENTC *c);

[0097] By calling the database loading client interface to connect to the proxy database loading service, it provides a basis for subsequent data loading and data query, and the returned data is kept in the DAGENTC * structure.

[0098] (3) Define the dagentc data query and sql execution interface.

[0099] As a possible implementation method, the dagentc data query interface is implemented in C language:

[0100] extern int dagentc_db_exec(DAGENTC *c, const char *sql);

[0101] The dagentc_db_exec function is an interface for database operations. It can execute database SQL statements submitted to dagent externally. The SQL statements are mainly DDL or insert / delete / update operations, and do not need to return actual values.

[0102] As a possible implementation method, the SQL execution interface is implemented in C language:

[0103] extern int dagentc_db_select1(DAGENTC *c, const char *sql, char *rs,int *rlen);

[0104] The definition of the SQL execution interface is: send the execution query SQL to the database, need to return a column of values, and the final returned value is stored in the rs and rlen pointers. If the function returns data less than 0, it means there is an error in the execution. If it is greater than or equal to 0, it means the total number of rows of data returned by the actual execution of the database SQL. rlen is the total length of the returned data.

[0105] (4) Define the binding method to load the sequence interface.

[0106] It should be noted that database data loading is currently divided into three categories. The first is to load data in a normal way similar to client SQL statements. The second is to load data in binding mode, which is suitable for large-scale data loading in insert / delete / update mode and has better performance. The third is to load data in copy or load mode, which is only suitable for insert mode loading and has relatively good loading performance. There may be some differences in the loading methods supported by different databases, but generally speaking, they are these three categories. In actual operation, the best data loading mode can be selected according to different target databases.

[0107] As a possible implementation, the following is the C language definition of the binding mode loading sequence interface:

[0108] extern int dagentc_stmt_open(DAGENTC *c);

[0109] extern int dagentc_stmt_prepare(DAGENTC *c, const char *sql);

[0110] extern int dagentc_stmt_bexec(DAGENTC *c, XYD_BIND *b, uint8_t *wrk,int wlen);

[0111] extern int dagentc_stmt_close(DAGENTC *c);

[0112] This sequence of interfaces completes the data opening before binding (dagentc_stmt_open()) to prepare for the actual binding data loading later. dagentc_stmt_prepare completes the parsing of the specified sql binding loading. dagentc_stmt_bexec actually sends the binding data to the dagent server to complete the data loading. dagentc_stmt_close closes the binding loading mode after the binding loading is completed.

[0113] (5) Define the copy mode loading interface

[0114] As a possible implementation, the definition of the copy loading interface is implemented in C language:

[0115] extern int dagentc_copy_open(DAGENTC *c, const char *owr, const char*tnm);

[0116] extern int dagentc_copy_bexec(DAGENTC *c, XYD_BIND *b, uint8_t *wrk,int wlen);

[0117] extern int dagentc_copy_close(DAGENTC *c);

[0118] This sequence of interfaces mainly completes the copy mode loading. dagentc_copy_open completes the opening operation before the copy mode loading, and passes in the table name and user name to prepare for the agent part of the data loading. dagentc_copy_bexec actually sends the copy data to the dagent service to complete the copy mode loading. dagentc_copy_close closes the copy mode loading after the copy mode loading is completed.

[0119] In addition, the data communication protocol used by the data assembly and sending module to communicate with the data loading module is the TCP / IP protocol. When sending data, the network data is dynamically encrypted according to the key returned by the dagentc_db_open interface to ensure the security and reliability of network data transmission when dagent and dagentc are running on different machines.

[0120] It should be noted that the data communication protocol format is mainly divided into two parts: [Data header] [Actual transmitted data]

[0121] The data header occupies 12 bytes, mainly storing the entire data packet length information (4 bytes), data operation type and other control information (occupying 4 bytes), and data checksum (4 bytes). According to the data operation type code, you can know what data is coming next, and what data is sent to the server and what actions need to be completed. The data checksum can be used to prevent the data packet from being tampered with. The hash value obtained by the fast hash operation of the subsequent data is stored in this part. After receiving the data packet, the server will first perform data verification. If the checksum is inconsistent, it will directly return an error to the client to prevent abnormal data from entering the database and causing unknown problems.

[0122] The data operation type encoding is shown in Table 1.

[0123] Table 1

[0124]

[0125] Step 104: The data loading module quickly parses the stream format data and converts it into the format required by the current environment interface of the target end, and calls the environment interface to obtain the target data loading task.

[0126] It can be understood that according to the foregoing and step 103, the data loading module quickly parses the stream format data and converts it into the format required by the current environment interface of the target end, and calls the environment interface to obtain the target data loading task. With the unique data loading agent client interface, the development of data loading services is relatively simple, and there is no need to face various different databases or data components. Data loading tasks can be divided into different people to complete, and people who are proficient in different databases can deeply develop corresponding dagents.

[0127] The target-side loader of the database replication software developed using the present application has better versatility, especially for the types of target-side databases that need to be supported, which can often greatly improve software development efficiency and reduce software maintenance costs; the general data processing parts such as data cleaning, data conversion, and performance optimization involved in the data loading process can be automatically applied to the various different databases supported by the entire loader after one modification; the interface call part involved in the data loading process and the specific target-side database is extracted separately as an independent module for development, which can make the performance optimization of the interface side for this type of database more convenient, and the division of labor among personnel will be more convenient and reasonable, and they only need to focus on the current database processing.

[0128] Figure 2 A schematic block diagram of an example electronic device 700 that can be used to implement an embodiment of the present application is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present application described and / or required herein.

[0129] like Figure 2 As shown, the device 700 includes a computing unit 701, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 702 or a computer program loaded from a storage unit 708 into a random access memory (RAM) 703. In the RAM 703, various programs and data required for the operation of the device 700 can also be stored. The computing unit 701, the ROM 702, and the RAM 703 are connected to each other via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0130] A number of components in the device 700 are connected to the I / O interface 705, including: an input unit 706, such as a keyboard, a mouse, etc.; an output unit 707, such as various types of displays, speakers, etc.; a storage unit 708, such as a disk, an optical disk, etc.; and a communication unit 709, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 709 allows the device 700 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0131] The computing unit 701 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 701 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 701 performs the various methods and processes described above, such as the voice instruction response method. For example, in some embodiments, the voice instruction response method may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 708. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 700 via the ROM 702 and / or the communication unit 709. When the computer program is loaded into the RAM 703 and executed by the computing unit 701, one or more steps of the voice instruction response method described above may be performed. Alternatively, in other embodiments, the computing unit 701 may be configured to perform the voice instruction response method in any other appropriate manner (e.g., by means of firmware).

[0132] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0133] The program code for implementing the method of the present application can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, implements the functions / operations specified in the flow chart and / or block diagram. The program code can be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0134] In the context of the present application, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0135] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0136] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), the Internet, and a blockchain network.

[0137] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship between the client and the server is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services ("Virtual Private Server", or "VPS" for short). The server may also be a server of a distributed system, or a server combined with a blockchain.

[0138] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this application can be executed in parallel, sequentially or in different orders, as long as the expected results of the technical solution of this application can be achieved, and this document is not limited here.

[0139] The above specific implementations do not constitute a limitation on the protection scope of this application. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions and improvements made within the spirit and principles of this application should be included in the protection scope of this application.

Claims

1. A general data device method based on database replication software development, characterized in that: include: The data to be loaded is read into the memory through the data reading module. During the reading process, the data formats of different types read into the memory are unified into an internal common data expression format. The data in the memory is stored in a row and column mode, and the corresponding row of data can be quickly retrieved through the row number; The data conversion module is used to parse the data read into the memory, and the parsed data is converted according to the configured conversion rules and the target database type; The converted data is assembled into the same stream format data through the data assembly and sending module, and sent to the data loading module; The data loading module quickly parses the stream format data and converts it into the format required by the current environment interface of the target end, and calls the environment interface to obtain the target data loading task; The data conversion module is used to analyze the data read into the memory, including: Parse the data value of each field of the data read into the memory, and obtain the dictionary information of each data table according to the field data value. The dictionary information records the storage structure, number of columns, data type of each column, and data length information of each data table. Without the dictionary information, only the data cannot obtain the meaning of each field data; The parsed data is converted according to the configured conversion rules and the target database type, including: Reading the conversion rules stored in the memory in the form of a binary tree after parsing, and performing type conversion on the fields to be converted according to the conversion rules and the target database type, wherein the type conversion process includes field truncation, field filling with other values, or numerical operation; Fill the converted data into the new memory; The data assembly and sending module is a database loading client interface, and the upper layer calls the database loading client interface to indirectly complete operations on different types of databases. The method also includes: By defining database types, unique identifiers are assigned to different types of databases in the database loading client interface; By calling the database loading client interface to connect to the proxy database loading service, a basis is provided for subsequent data loading and data query; Execute the database SQL submitted externally to the data assembly sending module, and send the execution query SQL to the database; The best data loading mode is selected according to different target databases, wherein the data loading mode is loading data by client SQL statement mode, or loading by binding mode, or loading by copy or load mode.

2. The method according to claim 1, characterized in that The data loading module can generate a target data loading task that matches the target database according to the target database environment and the data output environment.

3. The method according to claim 1, characterized in that The data communication protocol used by the data assembly and sending module when communicating with the data loading module is the TCP / IP protocol. When sending data, dynamic encryption network data is used according to the key returned by the database loading client interface.

4. An electronic device, characterized in that: include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory to implement the method according to any one of claims 1 to 3.

5. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, which are used to implement the method according to any one of claims 1 to 3 when executed by a processor.

6. A computer program product, characterized in that The invention comprises a computer program, which implements the method according to any one of claims 1 to 3 when being executed by a processor.

Citation Information

Patent Citations

  • Data migration method and device

    CN108073688A

  • Data synchronization method and device, equipment and medium

    CN117149907A

  • Data processing method and device for unified data management and control, electronic equipment and medium

    CN117421329A

  • Standardized database connectivity support for an event processing server in an embedded context

    US20110022618A1