A data transmission method and device, electronic equipment and storage medium

Abstract

This invention provides a data transmission method, apparatus, electronic device, and storage medium. The method includes: acquiring metadata information of a source table to be transmitted from a source database, the metadata information including: a first data volume of the source table, a preset concurrency level, and storage information of a preset type field, the storage information being used to identify whether the source table stores the preset type field; calculating a second data volume corresponding to each transmission thread based on the first data volume and the preset concurrency level; acquiring first data corresponding to each transmission thread from the source database based on the storage information and the second data volume; and transmitting the first data corresponding to each transmission thread to a target database. According to embodiments of this invention, data can be transmitted efficiently.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to a data transmission method, apparatus, electronic device, and storage medium. Background Technology

[0002] With the development of computer technology, the demand for data transmission is increasing day by day. Currently, data transmission solutions first require writing configuration files for each data table to be synchronized, transferring the source data from the source database to be queried to the disk, and then writing the source data from the disk to the target database after all the source data has been queried.

[0003] When dealing with large tables, the configuration work required is extensive and the transmission speed is slow. Therefore, the current data transmission solutions are inefficient. Summary of the Invention

[0004] This invention provides a data transmission method, apparatus, server, and storage medium to at least solve the problem of low efficiency in current data transmission schemes in related technologies.

[0005] The technical solution of the present invention is as follows:

[0006] According to a first aspect of the present invention, a data transmission method is provided, applied to a target device, comprising:

[0007] Obtain the metadata information of the source table to be transmitted from the source database. The metadata information includes: the first data volume of the source table, the preset concurrency level, and the storage information of the preset type fields. The storage information is used to identify whether the source table stores preset type fields.

[0008] Calculate the second data volume corresponding to each transmission thread based on the first data volume and the preset concurrency level;

[0009] Based on the storage information and the second data volume, retrieve the first data corresponding to each transmission thread from the source database;

[0010] The first data corresponding to each transmission thread is transmitted to the target database.

[0011] According to a second aspect of the present invention, a data transmission method is provided, applied to a target database, comprising:

[0012] Spark receives the first data corresponding to each transmission thread sent by the target device. The first data is obtained by the target device from the source database.

[0013] The first data is converted to obtain the second data in a preset compilation format;

[0014] Spark is used to write the second data corresponding to each transmission thread into the target database in parallel.

[0015] According to a third aspect of the present invention, a data transmission apparatus is provided, applied to a target device, comprising:

[0016] The first acquisition module is used to acquire metadata information of the source table to be transmitted in the source database. The metadata information includes: the first data volume of the source table, the preset concurrency level, and the storage information of the preset type fields. The storage information is used to identify whether the source table stores preset type fields.

[0017] The calculation module is used to calculate the second data volume corresponding to each transmission thread based on the first data volume and the preset concurrency level.

[0018] The second acquisition module is used to acquire the first data corresponding to each transmission thread from the source database based on the storage information and the second data volume;

[0019] The transmission module is used to transmit the first data corresponding to each transmission thread to the target database.

[0020] According to a fourth aspect of the present invention, a data transmission apparatus is provided, applied to a target database, comprising:

[0021] The receiving module is used to receive the first data corresponding to each transmission thread sent by the target device via Spark. The first data is obtained by the target device from the source database.

[0022] The conversion module is used to convert the first data to obtain the second data in a preset compilation format;

[0023] The write module is used to write the second data corresponding to each transmission thread to the target database in parallel using Spark.

[0024] According to a fifth aspect of the present invention, a server is provided, comprising: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to execute instructions to implement the view display method as described in the first aspect.

[0025] According to a sixth aspect of the present invention, a storage medium is provided that, when instructions in the storage medium are executed by a processor of a server, enables the server to perform the view display method as described in the first aspect.

[0026] According to a seventh aspect of the present invention, a computer program product is provided, which, when instructions in the computer program product are executed by a processor of a server, enables the server to perform the view display method as described in the first aspect.

[0027] The technical solutions provided by the embodiments of the present invention bring at least the following beneficial effects:

[0028] In embodiments of the present invention, metadata information of the source table to be transmitted from the source database is obtained. This metadata information includes: a first data volume of the source table, a preset concurrency level, and storage information for preset type fields. The storage information identifies whether the source table stores preset type fields. Based on the first data volume and the preset concurrency level, a second data volume corresponding to each transmission thread is calculated. Here, the data to be extracted is divided into different transmission threads to balance the concurrent transmission speed. Then, based on the different storage information of the preset type fields, the first data corresponding to each transmission thread is obtained from the source database according to the storage information and the second data volume, avoiding potential data loss during extraction and improving data transmission efficiency and accuracy. Finally, the first data corresponding to each transmission thread is transmitted to the target database, enabling efficient data transmission from the source database to the target database.

[0029] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit the invention. Attached Figure Description

[0030] Figure 1 This is a schematic diagram of an application environment for a data transmission method, apparatus, server, and storage medium provided in an embodiment of the present invention.

[0031] Figure 2 This is a flowchart of a data transmission method provided in an embodiment of the present invention.

[0032] Figure 3 This is a flowchart of another data transmission method provided in an embodiment of the present invention.

[0033] Figure 4 This is a block diagram of a data transmission device provided in an embodiment of the present invention.

[0034] Figure 5 This is a block diagram of another data transmission device provided in an embodiment of the present invention.

[0035] Figure 6 This is a block diagram of a server provided in an embodiment of the present invention.

[0036] Figure 7 This is a block diagram of a data processing device provided in an embodiment of the present invention. Detailed Implementation

[0037] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0038] like Figure 1 As shown, Figure 1 This is a schematic diagram illustrating an application environment for a data transmission method, apparatus, server, and storage medium provided in an embodiment of the present invention. The target device 100 communicates with one or more source databases 200 via network 400 for data communication or interaction. The target device 100 also communicates with one or more target databases 300 via network 400 for data communication or interaction.

[0039] The target device 100 may be a network server, database server, etc. The network 400 may be a wired or wireless network.

[0040] The above scheme obtains the metadata information of the source table to be transmitted from the source database. This metadata includes: the first data volume of the source table, the preset concurrency level, and the storage information of preset type fields. The storage information identifies whether the source table stores preset type fields. Based on the first data volume and the preset concurrency level, the second data volume corresponding to each transmission thread is calculated. Here, the data to be extracted is divided into different transmission threads to balance the concurrent transmission speed. Then, based on the different storage information of the preset type fields, the first data corresponding to each transmission thread is obtained from the source database according to the storage information and the second data volume, avoiding potential data loss during extraction and improving data transmission efficiency and accuracy. Finally, the first data corresponding to each transmission thread is transmitted to the target database, enabling efficient data transmission from the source database to the target database.

[0041] The data transmission method provided in the embodiments of the present invention will be described in detail below.

[0042] The data transmission method provided in this embodiment of the invention can be applied to the target device 100. For ease of description, unless otherwise specified, this embodiment of the invention is described with the target device 100 as the execution subject. It is understood that the execution subject does not constitute a limitation of this invention.

[0043] like Figure 2 As shown, the data transmission method may include the following steps.

[0044] Step 210: Obtain the metadata information of the source table to be transmitted in the source database. The metadata information includes: the first data volume of the source table, the preset concurrency level, and the storage information of the preset type fields. The storage information is used to identify whether the source table stores preset type fields.

[0045] Step 220: Calculate the second data volume corresponding to each transmission thread based on the first data volume and the preset concurrency level.

[0046] Step 230: Based on the storage information and the second data volume, obtain the first data corresponding to each transmission thread from the source database.

[0047] Step 240: Transmit the first data corresponding to each transmission thread to the target database.

[0048] In this embodiment of the invention, metadata information of the source table to be transmitted from the source database is obtained. This metadata information includes: a first data volume of the source table, a preset concurrency level, and storage information for preset type fields. The storage information identifies whether the source table stores preset type fields. Based on the first data volume and the preset concurrency level, a second data volume corresponding to each transmission thread is calculated. Here, the data to be extracted is divided into different transmission threads to balance the concurrent transmission speed. Then, based on the different storage information of the preset type fields, the first data corresponding to each transmission thread is obtained from the source database according to the storage information and the second data volume, avoiding potential data loss during extraction and improving data transmission efficiency and accuracy. Finally, the first data corresponding to each transmission thread is transmitted to the target database, enabling efficient data transmission from the source database to the target database.

[0049] The specific implementation methods for each of the above steps are described below.

[0050] Step 210 is involved.

[0051] Obtain the metadata information of the source table to be transmitted from the source database. The metadata information includes: the first data volume of the source table, the preset concurrency level, and the storage information of the preset type fields. The storage information is used to identify whether the source table stores preset type fields.

[0052] Before obtaining the metadata information of the source table to be transmitted from the source database as mentioned above, the following steps may also be included:

[0053] It receives user-input identification information and identifies the source table to be transmitted from the source database based on the identification information. The metadata information includes: the initial data volume of the source table, the preset concurrency level, and the storage information of preset type fields.

[0054] Among them, the first data volume of the source table refers to the total data volume of the source table; the preset concurrency level refers to the upper limit of the number of threads that can be extracted from the source table at the same time; and the storage information of the preset type fields refers to whether they include the primary key, whether they include the partition field, and whether they include the index field.

[0055] The metadata information may also include: the capacity of each source table, the number of source tables, the table name, the table description, the total number of fields, the number of primary keys, the total field descriptions, and the total field names.

[0056] Step 220 is involved.

[0057] Based on the first data volume and the preset concurrency level, calculate the second data volume corresponding to each transmission thread.

[0058] In this context, the second data volume corresponding to each transmission thread is the quotient of the first data volume and the preset concurrency level.

[0059] For example, if the first data volume is 20,000 and the preset concurrency is 5, then the second data volume corresponding to each transmission thread can be 4,000.

[0060] Here, based on the first data volume and the preset concurrency level, the second data volume corresponding to each transmission thread is calculated, so as to divide the data to be extracted into different transmission threads and balance the concurrent transmission speed.

[0061] Step 230 is involved.

[0062] Based on the storage information and the second data volume, the first data corresponding to each transmission thread is obtained from the source database.

[0063] Based on the storage information and the second data volume, an extraction scheme is matched to obtain the first data corresponding to each transmission thread from the source database according to the extraction scheme, so as to prevent data skew and data loss during the synchronization process and improve the extraction execution efficiency.

[0064] In one possible embodiment, the storage information identifier source table stores a primary key, and step 230 may specifically include the following steps:

[0065] When the primary key is an auto-incrementing primary key, the first data corresponding to each transmission thread is obtained from the source database based on the second data volume;

[0066] When the primary key is a non-auto-incrementing primary key, at least one first data range is determined based on the second data volume and the primary key;

[0067] Based on at least one first data range, obtain the first data corresponding to each transmission thread from the source database.

[0068] Among them, auto-incrementing primary keys include: 0, 1, 2, 3...; a, b, c... These are fields that can be arranged in sequence.

[0069] When the primary key is an auto-incrementing primary key, the first data corresponding to each transmission thread is retrieved from the source database based on the second data volume. For example, if the first data volume is 20,000 and the preset concurrency is 5, then the second data volume corresponding to each transmission thread can be 4,000. Retrieving the first data corresponding to each transmission thread from the source database based on the second data volume means extracting 4,000 data entries corresponding to each of the 5 transmission threads. The auto-incrementing primary key uses an offset, i.e., the second data volume, to define the boundaries, ensuring a more balanced data volume for each transmission thread.

[0070] Non-auto-incrementing primary keys, such as date and timestamp.

[0071] When the primary key is a non-auto-incrementing primary key, at least one first data range is determined based on the second data volume and the primary key. The first data range includes the upper limit value and the lower limit value of the primary key.

[0072] For example, at least one primary data range can be: data corresponding to January to March with the primary key, data corresponding to April to June with the primary key, or data corresponding to July to September with the primary key.

[0073] Since new data may be inserted during the extraction process after partitioning in the non-auto-incrementing case, which could lead to the loss of existing data in the task, it is necessary to determine the upper and lower limits of the primary key for each task before extraction to avoid data loss. In this way, even if new data is inserted during the extraction process, whether it is existing data or new data, as long as the primary key is within the first data range, it can be successfully extracted.

[0074] In one possible embodiment, step 230 may specifically include the following steps:

[0075] If the configuration information of the source table storing the information contains a splitting field, at least one second data range is determined based on the second data volume and the splitting field.

[0076] Based on at least one second data range, obtain the first data corresponding to each transmission thread from the source database.

[0077] The splitting fields are information pre-specified in the configuration information of the source table, such as user identification information, username, and user location.

[0078] If the configuration information of the source table storing the information contains a splitting field, at least one second data range is determined based on the second data volume and the splitting field.

[0079] For example, at least one first data range can be: data with the segmentation field corresponding to the user's location being location a, data with the segmentation field corresponding to the user's location being location b, or data with the segmentation field corresponding to the user's location being location c.

[0080] Since new data may be inserted during the extraction process after splitting, which may cause the original data in the task to be lost, it is necessary to determine the upper limit and lower limit of the splitting field for each task before extraction to avoid data loss.

[0081] Based on at least one second data range, the first data corresponding to each transmission thread is retrieved from the source database. This ensures that even if new data is inserted during the extraction process, whether it's existing or new data, as long as the segmentation field falls within the first data range, it can be successfully extracted.

[0082] In one possible embodiment, step 230 may specifically include the following steps:

[0083] If the source table storing the information identifier has an index field, at least one third data range is determined based on the second data volume and the index field;

[0084] Based on at least one third data interval, obtain the first data corresponding to each transmission thread from the source database.

[0085] Among them, the index fields and directory types are information independent of the source table, and each index field is used to identify a piece of data in the source table.

[0086] If the configuration information of the source table storing the information contains an index field, at least one third data range is determined based on the second data volume and the index field.

[0087] In this process, the primary key has higher priority than the splitting field, and the splitting field has higher priority than the index field. If the source table does not include the splitting field and the primary key, at least one third data range is determined based on the second data volume and the index field.

[0088] If the source table is found to be free of the sharding field, primary key, and index field, the first field in the source table can be retrieved, and the first data can be retrieved by traversing the source table.

[0089] Step 240 is involved.

[0090] The first data corresponding to each transmission thread is transmitted to the target database.

[0091] In one possible embodiment, a target table is created in a target database using a target component in the target device. The target table is used to store the first data.

[0092] Here, there is no need to write configuration files for each source table; the target table can be created in the target database through the target component in the target device.

[0093] In summary, the process involves obtaining metadata information from the source table to be transmitted in the source database. This metadata includes the initial data volume of the source table, the preset concurrency level, and the storage information for preset type fields. The storage information identifies whether the source table stores preset type fields. Based on the initial data volume and the preset concurrency level, the second data volume corresponding to each transmission thread is calculated. Here, the data to be extracted is divided into different transmission threads to balance the concurrent transmission speed. Then, based on the different storage information of the preset type fields, the initial data corresponding to each transmission thread is obtained from the source database according to the storage information and the second data volume. This avoids potential data loss during the extraction process, improves data transmission efficiency and accuracy, and finally, the initial data corresponding to each transmission thread is transmitted to the target database, enabling efficient data transmission from the source database to the target database.

[0094] Based on the above data transmission method, the present invention also provides a data transmission method applied to a target database. Specifically, in conjunction with... Figure 3 Please provide an explanation.

[0095] The data transmission method provided in this embodiment of the invention can be applied to the target database 300. For ease of description, unless otherwise specified, this embodiment of the invention is described with the target database 300 as the execution subject. It is understood that the execution subject does not constitute a limitation of this invention.

[0096] like Figure 3 As shown, the data transmission method may include the following steps.

[0097] Step 310: Receive the first data corresponding to each transmission thread sent by the target device via Spark. The first data is obtained by the target device from the source database.

[0098] Spark's runtime architecture is characterized by each application acquiring its own dedicated executor process, which remains resident throughout the application's lifecycle and runs tasks in a multi-threaded manner. Spark tasks are independent of the resource manager; as long as an executor process can be acquired and communication between them can be maintained, tasks can succeed.

[0099] Step 320: Convert the first data to obtain the second data in the preset compilation format.

[0100] The default compilation format can be JavaScript Object Notation (JSON).

[0101] During the extraction phase, Spark is used to transform the first set of data into second set of data in JSON format, which is then temporarily stored in a cache layer. JSON is a lightweight data-interchange format that uses a text format completely independent of programming languages ​​to store and represent data.

[0102] Here, by converting the first data, we obtain the second data in a preset compilation format. If the source table structure changes, the second data in the target table in JSON format supports arbitrary expansion of the key field. Since it is not a real table structure, we can arbitrarily add keys and values.

[0103] Step 330: Use Spark to write the second data corresponding to each transmission thread into the target database in parallel.

[0104] Specifically, Spark uses the second data corresponding to each transmission thread retrieved from the caching layer to write to the target database in parallel. By leveraging Spark's native memory swapping capabilities and writing concurrently to the target database, write efficiency is improved.

[0105] Additionally, before writing the second data corresponding to each transmission thread into the target database in parallel using Spark, you can use custom data mapping to ensure that the second data and its numerical format are consistent with the first data. For example, you can convert 0 to true.

[0106] Finally, data is cached by Spark and written in parallel to the distributed file system (HDFS), which allows for faster delivery of the target files. HDFS is a distributed file system designed to run on general-purpose hardware.

[0107] Here, Spark's distributed advantages are leveraged, and cached loading is used to quickly synchronize to the target library, reducing input / output (I / O) overhead. I / O overhead refers to the percentage of input / output traffic.

[0108] In one possible embodiment, metadata information of the source table is obtained from the first data;

[0109] Based on the metadata information, create a view corresponding to the first data. The content of the view is updated according to the updated data in the source database.

[0110] Based on the metadata information, a view corresponding to the first data is created, so that the user sees the latest table structure, while the underlying historical partitions of the target table in the target database retain historical data, i.e., the second data in JSON format.

[0111] For example, if the first data in the source table has a "gender" field, during synchronization, the "gender" field in the target database might be deleted, resulting in the loss of historical data; moreover, the table structure of the target database would need to be modified. Therefore, by creating a view corresponding to the first data based on metadata information, compatibility with changes to the source database can be maintained.

[0112] Here, on the one hand, Spark is used to write the second data corresponding to each transmission thread into the target database in parallel, which can ensure that no data is lost; on the other hand, based on the metadata information, a view corresponding to the first data is created and updated according to the updated data in the source database, which can display the latest table structure in a timely manner.

[0113] In summary, Spark receives the first data corresponding to each transmission thread from the target device. This first data is retrieved from the source database by the target device. The first data is then transformed to obtain second data in a pre-compiled format. This ensures no data loss even if the source database is subsequently updated. Spark then writes the second data corresponding to each transmission thread to the target database in parallel. This leverages Spark's distributed caching capabilities and allows for rapid synchronization to the target database through cache loading.

[0114] Based on the above Figure 2 In addition to the data transmission method shown, this invention also provides a data transmission device. Specifically, in conjunction with... Figure 4 Please provide an explanation.

[0115] Figure 4 This is a block diagram of a data transmission device provided in an embodiment of the present invention.

[0116] Reference Figure 4 The data transmission device 401 may include:

[0117] The first acquisition module 410 is used to acquire metadata information of the source table to be transmitted in the source database. The metadata information includes: the first data volume of the source table, the preset concurrency level and the storage information of the preset type fields. The storage information is used to identify whether the source table stores preset type fields.

[0118] The calculation module 420 is used to calculate the second data volume corresponding to each transmission thread based on the first data volume and the preset concurrency level;

[0119] The second acquisition module 430 is used to acquire the first data corresponding to each transmission thread from the source database based on the storage information and the second data volume.

[0120] The transmission module 440 is used to transmit the first data corresponding to each transmission thread to the target database.

[0121] In one possible embodiment, the storage information identifier source table stores a primary key, and the second acquisition module 430 is specifically used for:

[0122] When the primary key is an auto-incrementing primary key, the first data corresponding to each transmission thread is obtained from the source database based on the second data volume;

[0123] When the primary key is a non-auto-incrementing primary key, at least one first data range is determined based on the second data volume and the primary key;

[0124] Based on at least one first data range, obtain the first data corresponding to each transmission thread from the source database.

[0125] In one possible embodiment, the second acquisition module 430 is specifically used for:

[0126] If the configuration information of the source table storing the information contains a splitting field, at least one second data range is determined based on the second data volume and the splitting field.

[0127] Based on at least one second data range, obtain the first data corresponding to each transmission thread from the source database.

[0128] In one possible embodiment, the second acquisition module 430 is specifically used for:

[0129] If the source table storing the information identifier has an index field, at least one third data range is determined based on the second data volume and the index field;

[0130] Based on at least one third data interval, obtain the first data corresponding to each transmission thread from the source database.

[0131] In one possible embodiment, the data transmission device 401 may include:

[0132] The first creation module is specifically used to create a target table in the target database using the target components in the target device. The target table is used to store the first data.

[0133] In this embodiment of the invention, metadata information of the source table to be transmitted in the source database is obtained. This metadata information includes: a first data volume of the source table, a preset concurrency level, and storage information for preset type fields. The storage information is used to identify whether the source table stores preset type fields. Based on the first data volume and the preset concurrency level, a second data volume corresponding to each transmission thread is calculated. Here, the data to be extracted is divided into different transmission threads to balance the concurrent transmission speed. Then, based on the different storage information of the preset type fields, the first data corresponding to each transmission thread is obtained from the source database according to the storage information and the second data volume, avoiding potential data loss during extraction and improving data transmission efficiency and accuracy. Finally, the first data corresponding to each transmission thread is transmitted to the target database, enabling efficient data transmission from the source database to the target database.

[0134] Based on the above data transmission method, the present invention also provides a data transmission device. (Specifically, in conjunction with...) Figure 5 Please provide an explanation.

[0135] Figure 5 This is a block diagram of a data transmission device provided in an embodiment of the present invention.

[0136] Reference Figure 5 The data transmission device 500 may include:

[0137] The receiving module 510 is used to receive the first data corresponding to each transmission thread sent by the target device via Spark. The first data is obtained by the target device from the source database.

[0138] The conversion module 520 is used to convert the first data to obtain the second data, which is in JSON format.

[0139] The write module 530 is used to write the second data corresponding to each transmission thread to the target database in parallel using Spark.

[0140] In one possible embodiment, the data transmission device 500 may further include:

[0141] The third acquisition module is used to obtain metadata information of the source table from the first data;

[0142] The second creation module is used to create a view corresponding to the first data based on the metadata information. The content of the view is updated according to the updated data in the source database.

[0143] In this embodiment of the invention, Spark receives first data corresponding to each transmission thread sent by the target device. This first data is obtained by the target device from the source database. The first data is then converted to obtain second data in a preset compilation format. This ensures that no data is lost even if the source database is subsequently updated. Spark then writes the second data corresponding to each transmission thread to the target database in parallel. This leverages Spark's distributed caching advantages and allows for rapid synchronization to the target database through cache loading.

[0144] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0145] Figure 6 This is a block diagram of a server provided in an embodiment of the present invention. (Refer to...) Figure 6 The present invention also provides a server, including a processor 610, a communication interface 620, a memory 630 and a communication bus 640, wherein the processor 610, the communication interface 620 and the memory 630 communicate with each other through the communication bus 640.

[0146] The memory 630 is used to store instructions that can be executed by the processor 610.

[0147] When the processor 610 executes the instructions stored in the memory 630, it performs the following steps:

[0148] Obtain the metadata information of the source table to be transmitted from the source database. The metadata information includes: the first data volume of the source table, the preset concurrency level, and the storage information of the preset type fields. The storage information is used to identify whether the source table stores preset type fields.

[0149] Based on the first data volume and the preset concurrency level, calculate the second data volume corresponding to each transmission thread.

[0150] Based on the storage information and the second data volume, the first data corresponding to each transmission thread is obtained from the source database.

[0151] The first data corresponding to each transmission thread is transmitted to the target database.

[0152] In this embodiment of the invention, metadata information of the source table to be transmitted from the source database is obtained. This metadata information includes: a first data volume of the source table, a preset concurrency level, and storage information for preset type fields. The storage information identifies whether the source table stores preset type fields. Based on the first data volume and the preset concurrency level, a second data volume corresponding to each transmission thread is calculated. Here, the data to be extracted is divided into different transmission threads to balance the concurrent transmission speed. Then, based on the different storage information of the preset type fields, the first data corresponding to each transmission thread is obtained from the source database according to the storage information and the second data volume, avoiding potential data loss during extraction and improving data transmission efficiency and accuracy. Finally, the first data corresponding to each transmission thread is transmitted to the target database, enabling efficient data transmission from the source database to the target database.

[0153] Figure 7 This is a block diagram of a data processing device provided in an embodiment of the present invention. For example, the device 700 can be provided as a server. (See also...) Figure 7 Server 700 includes a processing component 722, which further includes one or more processors, and memory resources represented by memory 732 for storing instructions, such as applications, that can be executed by the processing component 722. The applications stored in memory 732 may include one or more modules, each corresponding to a set of instructions. Furthermore, processing component 722 is configured to execute instructions to perform the data transfer method described in any of the above embodiments.

[0154] The device 700 may also include a power supply component 727 configured to perform power management of the device 700, a wired or wireless network interface 750 configured to connect the device 700 to a network, and an input / output (I / O) interface 758. The device 700 can operate on an operating system stored in memory 732, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, or similar.

[0155] This invention also provides a computer program product that, when the instructions in the computer program product are executed by the processor of a server, enables the server to execute the data transmission method described in any of the above embodiments.

[0156] This invention also provides an electronic device, including: a processor, a memory, and a computer program stored in the memory and executable on the processor. When the computer program is executed by the processor, it implements the various processes of the above-described data transmission method embodiments and achieves the same technical effects. To avoid repetition, it will not be described again here.

[0157] This invention also provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it implements the various processes of the above-described data transmission method embodiments and achieves the same technical effects. To avoid repetition, it will not be described again here. The computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, etc.

[0158] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0159] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0160] The embodiments of the present invention have been described above with reference to the accompanying drawings. However, the present invention is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of the present invention without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of the present invention.

[0161] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed in this invention can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0162] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0163] In the embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0164] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0165] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0166] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, essentially, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, ROM, RAM, magnetic disks, or optical disks.

[0167] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A data transmission method, characterized in that, Applied to target devices, including: Obtain metadata information of the source table to be transmitted from the source database. The metadata information includes: the first data volume of the source table, the preset concurrency level, and the storage information of the preset type field. The storage information is used to identify whether the source table stores the preset type field. Based on the first data volume and the preset concurrency level, calculate the second data volume corresponding to each transmission thread; Based on the storage information and the second data volume, obtain the first data corresponding to each transmission thread from the source database; The first data corresponding to each transmission thread is transmitted to the target database; The storage information identifies that the source table stores a primary key. The step of retrieving the first data corresponding to each transmission thread from the source database based on the storage information and the second data volume includes: If the primary key is an auto-incrementing primary key, the first data corresponding to each transmission thread is obtained from the source database based on the second data volume; When the primary key is a non-auto-incrementing primary key, at least one first data interval is determined based on the second data volume and the primary key, wherein the first data interval includes the upper limit value and the lower limit value of the primary key; Based on the at least one first data range, the first data corresponding to each transmission thread is obtained from the source database.

2. The method according to claim 1, characterized in that, The step of obtaining the first data from the source database based on the stored information and the second data volume includes: If the configuration information identifying the source table contains a splitting field, at least one second data range is determined based on the second data volume and the splitting field. Based on the at least one second data interval, the first data corresponding to each transmission thread is obtained from the source database.

3. The method according to claim 1, characterized in that, The step of obtaining the first data from the source database based on the stored information and the second data volume includes: If the storage information indicates that the source table stores an index field, at least one third data range is determined based on the second data volume and the index field; Based on the at least one third data interval, the first data corresponding to each transmission thread is obtained from the source database.

4. The method according to claim 1, characterized in that, The method further includes: A target table is created in the target database using the target component in the target device, and the target table is used to store the first data.

5. A data transmission method, characterized in that, Applied to the target database, including: The system receives first data corresponding to each transmission thread from the target device via Spark. This first data is metadata information of the source table to be transmitted from the source database, obtained by the target device. The metadata information includes: a first data volume of the source table, a preset concurrency level, and storage information for a preset type field. This storage information identifies whether the source table stores the preset type field. Based on the first data volume and the preset concurrency level, a second data volume corresponding to each transmission thread is calculated and obtained from the source database based on the storage information and the second data volume. The storage information indicates that the source table stores a primary key. Specifically, if the primary key is an auto-incrementing primary key, the first data is obtained from the source database by the target device based on the second data volume. If the primary key is a non-auto-incrementing primary key, the first data is obtained from the source database by the target device based on at least one first data range determined by the target device based on the second data volume and the primary key. The first data range includes the upper and lower limits of the primary key. The first data is converted to obtain the second data in a preset compilation format; The second data corresponding to each transmission thread is written to the target database in parallel using Spark.

6. The method according to claim 5, characterized in that, The method further includes: Obtain the metadata information of the source table from the first data; Based on the metadata information, a view corresponding to the first data is created, and the content of the view is updated according to the updated data in the source database.

7. A data transmission device, characterized in that, Applied to target devices, including: The first acquisition module is used to acquire metadata information of the source table to be transmitted in the source database. The metadata information includes: the first data volume of the source table, the preset concurrency level, and the storage information of the preset type field. The storage information is used to identify whether the source table stores the preset type field. The calculation module is used to calculate the second data volume corresponding to each transmission thread based on the first data volume and the preset concurrency level; The second acquisition module is used to acquire the first data corresponding to each transmission thread from the source database based on the storage information and the second data volume; The transmission module is used to transmit the first data corresponding to each transmission thread to the target database; The storage information identifies that the source table stores a primary key. The second acquisition module is specifically used to: when the primary key is an auto-incrementing primary key, acquire the first data corresponding to each transmission thread from the source database based on the second data volume; when the primary key is a non-auto-incrementing primary key, determine at least one first data range based on the second data volume and the primary key; and acquire the first data corresponding to each transmission thread from the source database based on the at least one first data range, wherein the first data range includes the upper limit and lower limit of the primary key.

8. A data transmission device, characterized in that, Applied to the target database, including: The receiving module is used to receive first data corresponding to each transmission thread sent by the target device via Spark. The first data is metadata information of the source table to be transmitted from the source database obtained by the target device. The metadata information includes: a first data volume of the source table, a preset concurrency level, and storage information of a preset type field. The storage information is used to identify whether the source table stores the preset type field. Based on the first data volume and the preset concurrency level, a second data volume corresponding to each transmission thread is calculated, and the data is obtained from the source database based on the storage information and the second data volume. The storage information indicates that the source table stores a primary key. Specifically, if the primary key is an auto-incrementing primary key, the first data is obtained by the target device from the source database based on the second data volume; if the primary key is a non-auto-incrementing primary key, the first data is obtained by the target device based on the second data volume and the primary key, determining at least one first data range, and obtaining the data from the source database based on the at least one first data range. The first data range includes the upper limit and lower limit of the primary key. The conversion module is used to convert the first data to obtain the second data, wherein the data format of the second data is JSON; The write module is used to write the second data corresponding to each transmission thread to the target database in parallel using Spark.

9. An electronic device, characterized in that, include: A processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the steps of the data transmission method as described in any one of claims 1 to 6.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the data transmission method as described in any one of claims 1 to 6.

Images