Data storage method, storage medium, program product and electronic device

Abstract

The present application relates to the field of information technology, and in particular to a data storage method, a readable storage medium, a program product, and an electronic device. The data storage method is applied to a data integration system. When the data integration system stores the source data of the source end in the storage partition corresponding to the target table of the target end as a partition coverage business, if the target end does not have a corresponding instruction to execute the partition coverage business, a temporary table can be established on the target end. The partitioning rules of the temporary table and the target table are the same, so the data integration system can store the source data in the partition corresponding to the temporary table, and then replace the partition storing the source data in the temporary table with the data of the partition corresponding to the target table, thereby completing the partition coverage business. Therefore, the data integration system does not need to determine the storage partition corresponding to the source data to complete the partition coverage business, thereby reducing the complexity of the data integration system in performing the partition coverage business and saving the computing resources of the data integration system.

Description

Technical Field

[0001] The present application relates to the field of data integration, and in particular to a data storage method, a readable storage medium, a program product and an electronic device. Background Art

[0002] Data integration refers to reading source data from an engine (source end) (such as online analytical processing (OLAP) engine, online transaction processing (OLTP) engine) through a data integration system, and storing the source data source data in another engine (target end) after arrangement. Among them, the data integration system provides a unified application system for users to access multiple effective and heterogeneous data sources, so that users can truly focus on the specific results they want, without having to worry about how to obtain these results.

[0003] In some business scenarios, users need to store source data in the source engine to the target engine. Since some storage areas (such as storage space of a memory) of the target engine already store old data, when storing source data in the above storage areas, it is necessary to delete the corresponding old data in the storage areas, and then store the source data in the storage areas. This way of storing data is called partition overwriting.

[0004] In order to respond to partition overwrite services more quickly, some engines define partition overwrite instructions for responding to partition overwrite services. For example, in some target engines, such as Spark, Flink, and data warehouse management systems (Hive), partition overwrite writing can be performed through the insert overwrite command. The data integration system can write source data to the engine by sending the defined insert overwrite command to this part of the engine.

[0005] However, for some engines (such as relational database service (RDS)), when the data integration system writes data to the engine through partition overlay, it is necessary to determine the partitioning rules of the target engine, and determine the partition where the source data is to be stored (hereinafter referred to as the target partition) based on the partitioning rules, and then delete the old data in the target partition before storing the source data in the target partition. The partitioning rules are rules for determining the storage area corresponding to the source data based on a certain part of the source data.

[0006] In data integration, the amount of source data is large. When the data integration system stores source data to the target end, it needs to determine the storage area corresponding to each source data according to the partitioning rules of the target end. Therefore, the data integration system has high complexity in completing the partition coverage business. In addition, some partitioning rules of the target end are user-defined rules, which makes it impossible for the data integration system to accurately partition the source data. Summary of the invention

[0007] In view of this, the present application provides a data storage method, a readable storage medium, a program product, and an electronic device.

[0008] In a first aspect, the present application provides a data storage method, the method comprising: detecting a first instruction, wherein the first instruction is used to instruct first data of a first electronic device to be stored in at least one first storage partition in a first storage partition table of a second electronic device; sending a second instruction to the second electronic device, wherein the second instruction is used to instruct the second electronic device to establish a second storage partition table corresponding to the first storage partition table, and the data partitioning rules of the first storage partition table and the second storage partition table are the same, and there are multiple second storage partitions in the second storage partition table corresponding to each first storage partition in the first storage partition table; in response to detecting that the second electronic device has completed the establishment of the first storage partition table, sending a third instruction to the second electronic device, the third instruction is used to instruct the second electronic device to store the first data in the second storage partition table corresponding to the second storage partition table based on the data partitioning rule; sending a fourth instruction to the second electronic device, the fourth instruction is used to instruct the second electronic device to replace the data in at least one second storage partition in the second storage partition table storing the first data with the data in the first storage partition table corresponding to the first storage partition.

[0009] Exemplarily, in some embodiments of the present application, the data storage method can be applied to a data integration system or an electronic device that deploys a data integration system. The first instruction can be, for example, a partition overwrite service. The partition overwrite service is, for example, storing the first data (also referred to as source data) in the first electronic device (also referred to as the source end) in the second electronic device (also referred to as the target end), and the second electronic device is deployed with a corresponding engine for storing the first data. If the engine of the target end does not support the instruction of the partition overwrite service (such as the insert overwrite command), the data integration system can send a second instruction to the target end, instructing the engine of the target end to establish a second storage partition table (also referred to as a temporary table) with the same data partition rule (also referred to as the partition rule) as the first storage partition table (also referred to as the target table), and the second storage partition table corresponds to the second storage partition. No data is stored, so the data integration system can store the first data in each second storage partition without using the insert overwrite command. Then the data integration system can send a fourth instruction (also referred to as a data replacement instruction) to the engine of the target end to replace the data in the second storage partition with the data in the corresponding first storage partition, thereby completing the partition overwrite service.

[0010] It should be understood that in the process of the data integration system performing the partition coverage service, the data integration system does not need to determine the first storage partition corresponding to each first data based on the data partition rule of the target end, thereby reducing the complexity of the data integration system completing the partition coverage service and saving the computing resources of the data integration system completing the partition coverage service.

[0011] In a possible implementation of the first aspect above, the data partitioning rule includes a partition column and a partition expression, wherein the partition column is used to indicate a column in at least one row of data stored in a storage partition table that is used to determine a storage partition corresponding to each row of data; and the partition expression is used to indicate a correspondence between content in a partition column in each row of data in at least one row of data and a storage partition storing the corresponding row of data.

[0012] Exemplarily, taking the first data as row data with an identifier of 6, a name of F, and a date of 1990-01-01 as an example, the partition column can be, for example, a column corresponding to the date, that is, the content of the partition column corresponding to the row data is 1990-01-01, and the partition expression can be, for example, the date of the same month is the same partition, that is, the storage partition corresponding to the above row data is the second storage partition corresponding to January.

[0013] It should be understood that, since the first storage partition table and the second storage partition table have the same data partitioning rule, the second storage partition to which the first data is stored based on the second storage partition table corresponds to the first storage partition to which the first data is stored based on the first storage partition table. Exemplarily, if the data stored in the second storage partition is data with a date of January, then the first storage partition corresponding to the second storage partition also stores data with a date of January.

[0014] In a possible implementation of the first aspect, the method further includes: after the second electronic device executes the fourth instruction, deleting the second storage partition table and the data in each second storage partition.

[0015] That is to say, after the second electronic device executes the fourth instruction, the second partition table and the data stored in the second partition can be deleted, thereby releasing the storage space of the second electronic device.

[0016] In a possible implementation of the first aspect above, the fourth instruction includes at least one sub-instruction corresponding to each second storage partition storing first data in at least one second storage partition, and each sub-instruction is used to instruct the second electronic device to replace the data in the second storage partition corresponding to the sub-instruction with the first storage partition corresponding to the second storage partition corresponding to the sub-instruction in the first storage partition table.

[0017] That is to say, for the second storage partition storing the first data, the data integration system will send a sub-instruction to the engine corresponding to the second electronic device (i.e., the target end), and the sub-instruction is used to complete the replacement of the data corresponding to the second storage partition with the data corresponding to the first storage partition. That is, one sub-instruction only completes the replacement of the data of the second storage partition with the data of the corresponding first storage partition once.

[0018] It should be understood that in other embodiments, the data integration system may also determine multiple second storage partitions storing the first data, and then complete the data replacement between the multiple second storage partitions and the corresponding first storage partitions through one sub-instruction.

[0019] In a possible implementation of the first aspect, the sending of the fourth instruction to the second electronic device includes: traversing each second storage partition corresponding to the second storage partition table, and when data is stored in the second storage partition currently traversed, sending a sub-instruction for the second storage partition currently traversed to the second electronic device.

[0020] That is to say, the data integration system will traverse each second storage partition corresponding to the second storage partition table. If the currently traversed second storage partition stores data, the data integration system will send a sub-instruction to the second electronic device to replace the data in the current second storage partition with the data in the corresponding first storage partition.

[0021] In a possible implementation of the first aspect above, when data is stored in the second storage partition currently traversed, the sub-instruction for the second storage partition currently traversed is sent to the second electronic device, including: when the number of rows of data stored in the second storage partition currently traversed is not 0, the sub-instruction for the second storage partition currently traversed is sent to the second electronic device.

[0022] That is to say, based on the row data of the data stored in the second storage partition currently traversed, it can be determined whether the corresponding second storage partition stores the first data. When the number of rows of stored data in the second storage partition is not 0, it means that the first data is stored in the current second storage partition, and the data integration system can send a sub-instruction for the current second storage partition to the second electronic device.

[0023] In a possible implementation of the first aspect above, the sub-instruction includes a partition exchange instruction for the second storage partition corresponding to the sub-instruction.

[0024] For example, if some engines support the exchange partition command, the sub-instruction that the data integration system may send to the engine at the target end is the exchange partition command to complete the data replacement between the second storage partition and the first storage partition.

[0025] In a possible implementation of the first aspect above, the above-mentioned sub-instruction includes a deletion instruction for deleting the data of the first storage partition corresponding to the sub-instruction in the first storage partition table, and an insertion instruction for inserting the data in the second storage partition corresponding to the sub-instruction in the second storage partition table into the first storage partition corresponding to the sub-instruction in the first storage partition table.

[0026] Exemplarily, the sub-instructions may also be, for example, a truncate command and an insert into command. The truncate command deletes the data in the first storage partition corresponding to the second storage partition, and then the insert into command inserts the data in the second storage partition into the corresponding first storage partition, thereby completing the data replacement between the second storage partition and the first storage partition.

[0027] In a second aspect, the present application provides an electronic device, the electronic device comprising: a memory for storing instructions; at least one processor for executing instructions so that the device implements the first aspect and any possible implementation method of the first aspect. The beneficial effects that can be achieved in the second aspect can refer to the beneficial effects of the method provided in any implementation of the first aspect, and will not be repeated here.

[0028] In a third aspect, the present application provides a computer-readable storage medium, in which instructions are stored, and when the instructions are executed by a device, the computer implements the method provided in the first aspect and any possible implementation of the first aspect. The beneficial effects that can be achieved in the third aspect can refer to the beneficial effects of the method provided in any implementation of the first aspect, and will not be repeated here.

[0029] In a fourth aspect, the present application provides a computer program product, which, when running on a device, enables the device to implement the method provided in the first aspect and any possible implementation of the first aspect. The beneficial effects that can be achieved in the fourth aspect can refer to the beneficial effects of the method provided in any implementation of the first aspect, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 According to some embodiments of the present application, a scenario diagram of a data integration system performing data integration is shown;

[0031] Figure 2 According to some embodiments of the present application, a flow chart of a data integration system writing data to a target end is shown;

[0032] Figure 3 According to some embodiments of the present application, a scene graph of data integration is shown;

[0033] Figure 4 According to some embodiments of the present application, an interactive process of a data integration system writing source data to a target end is shown. Figure 1 ;

[0034] Figure 5 According to some embodiments of the present application, an interactive process of a data integration system writing source data to a target end is shown. Figure 2 ;

[0035] Figure 6 According to some embodiments of the present application, a schematic structural diagram of a host is shown. DETAILED DESCRIPTION

[0036] The illustrative embodiments of the present application include, but are not limited to, a data storage method, a readable storage medium, a program product, and an electronic device.

[0037] In order to make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0038] Figure 1 According to some embodiments of the present application, a scenario diagram of a data integration system performing data integration is shown.

[0039] like Figure 1 As shown in FIG, the scenario includes a source end, a target end, and a data integration system. The data integration system is used to obtain source data from the source end and then store the source data in the target end.

[0040] Exemplarily, both the source end and the target end can be physical machines, such as computers or servers. The physical machine is configured with at least one data structure engine, such as Hive, ClickHouse, MySQL, data lake insight (DLI), etc.

[0041] It should be understood that the target end and the source end may also include other types of engines. When executing data integration services, the data integration system needs to obtain source data from an engine on the source end, and then write the source data to an engine on the target end in a partition overwrite manner. The engine on the source end and the engine on the target end may be of the same type or of different types.

[0042] Generally speaking, the engine on the target side stores data based on a partition table. The partition table is divided into multiple partitions according to the partition rules, and each partition corresponds to a partition value (for example, the first partition, the second partition, etc.). A partition corresponds to a storage area (for example, different storage spaces of the same storage device), and each storage area stores all the data of the corresponding partition. Among them, the partition rules include partition columns and partition expressions. For the data of the partition column corresponding to the source data, the partition value of the source data can be determined based on the partition expression, and the source data with the same partition value is the data of the same partition.

[0043] Exemplarily, the data integration system may be a computer or a server, and the data integration system is connected to the source end and the target end via a communication interface. Exemplarily, the communication interface may include a network interface.

[0044] It should be understood that when performing data integration, the user can obtain source data from an engine on the source side based on the data integration system, and then store the source data in a partition corresponding to a target table of an engine on the target side.

[0045] For example, when performing data integration, there is a partition overwrite write mode, that is, when new data is written to the target end, the partition involved in the target end is first deleted, and then the new data is inserted. If some engines support the insert overwrite command to perform partition overwrite writing, the data integration system can write the source data obtained from the source end to the partition corresponding to the engine of the target end by sending the insert overwrite command to these engines. For example, Table 1 shows the data table of the target end before the partition overwrite is performed.

[0046] Table 1

[0047] Logo name date 1 A 1990-01-01 2 B 1990-01-01 3 C 1990-01-01 4 D 1990-02-01 5 E 1990-03-01

[0048] As shown in Table 1, Table 1 includes 5 rows of data. The first column of each row of data is the identifier of the data, the second column is the name of the data, and the third column is the date of the data. Exemplarily, in this embodiment, the partitioning rule is, for example, to use the date of the data as the partition column and the distribution area of ​​the date as the partition expression. For example, the partition expression is that the dates of the same month are the same partition.

[0049] Referring to Table 1, in Table 1, the data with a date of 1990-01-01 is a partition (A, B, and C are a partition), and its partition value corresponds to 1 (the first partition). The data with a date of 1990-02-01 is a partition (D is a partition), and its partition value corresponds to 2 (the second partition). The data with a date of 1990-03-01 is a partition (E is a partition), and its partition value corresponds to 3 (the third partition).

[0050] It should be understood that in other embodiments, Table 1 may include more or less data. And the partition column of Table 1 may also be other columns. For example, in other embodiments, the partition column is a column of names, and the partition expression is that data with the same name is the same partition.

[0051] It should be understood that in other embodiments, the data in Table 1 may also include more columns, such as an age column, a gender column, etc. The partitioning expressions of the same data in different tables may also be different. For example, in a table of Hive and MySQL, the age column of the data is used as the partitioning column. In the Hive table, the partitioning expression is that the data with the same age is one partition. In the MySQL table, the partitioning expression is that the age in the preset range is one partition. For example, data aged between 0 and 10 is one partition, data aged between 11 and 20 is one partition, and so on.

[0052] When the partition overwrite write mode is executed in Table 1, for example, the source data written is identified as 6, the name is F, and the date is 1990-01-01. Based on the partition expression, it can be determined that the partition value of the target partition of the written source data is 1. Therefore, when the partition overwrite write mode is executed, it is necessary to delete all the data in the target partition with a partition value of 1, and then write the source data to the target partition of the target end. Exemplarily, after writing the source data, the data on the target end is shown in Table 2.

[0053] Table 2

[0054] Logo name date 6 F 1990-01-01 4 D 1990-02-01 5 E 1990-03-01

[0055] Referring to Table 2, the data in the target partition with a partition value of 1 includes A, B, and C. Therefore, after deleting A, B, and C, F can be written to the target partition. It should be understood that when the above-mentioned partition overwrite write operation is performed on some engines that support the insert overwrite command, the data integration system can directly send the insert overwrite command to the corresponding engine to perform the partition overwrite business. However, some engines do not support the insert overwrite command, so the data integration system is required to obtain the partition rules of the target table corresponding to the target table engine on the target side. For example, determine the partition column and partition expression of the target table. The data integration system then calculates the data of the partition column corresponding to the source data based on the partition expression to determine the partition value of the source data. Finally, the data integration system deletes the old data in the target partition corresponding to the partition value of the target table, and then writes the source data to the target partition, thereby completing the partition overwrite business.

[0056] For example, Figure 2 According to some embodiments of the present application, a flow chart of a data integration system writing data to a target end is shown.

[0057] For example, when the data integration system writes data to the target end, the source data is first obtained from an engine on the source end, and then the data is written to the target end. If the engine on the target end does not support the insert overwrite command, the execution Figure 2 The process shown.

[0058] It should be understood that when performing data integration, the engine on the source side, the engine on the target side, and the target table corresponding to the target side engine have been determined (determined by the user). The data integration system only needs to obtain the source data from the determined source side engine, and then write the source data into the target table of the determined target side engine.

[0059] Exemplarily, the execution subject of each of the following processes is a data integration system, which may be, for example, an electronic device such as a server or a computer. When introducing each of the following processes, the execution subject of each step will not be repeatedly introduced.

[0060] It should be understood that the following process only introduces the process of the data integration system writing data to the target table on the target end, and the process of the data integration system obtaining data from the source end is not described in detail.

[0061] like Figure 2 As shown, the process includes:

[0062] S201, obtaining row data.

[0063] Exemplarily, the source data that the data integration system needs to write to the target end is, for example, a data list. When writing data to the target end, each row of data in the data list is first obtained as row data, and then the operation of writing the row data to the target end is performed. For example, the above writes the row data with the identifier 6, the name F, and the date 1990-01-01.

[0064] S202, determining the partition column of the row data.

[0065] Exemplarily, when the data integration system selects the target end, it needs to determine the partitioning rule of the target end, and the partitioning rule includes the partitioning column and partitioning expression of the data. After obtaining the row data, the data integration system determines the partitioning column of the row data according to the partitioning rule of the target end. For example, for the row data with the identifier 6, the name F, and the date 1990-01-01. The corresponding partitioning column is the date, and the partitioning expression is that the dates of the same month are in the same partition.

[0066] S203: Determine a partition value of the row data based on the partition column.

[0067] For example, after determining the partition column of the row data, the partition value can be calculated based on the data of the partition column corresponding to the row data and the partition expression. For example, the row data with the identifier 6, the name F, and the date 1990-01-01 has the data of the corresponding partition column 1990-01-01. Based on the partition expression, the partition value of the row data can be determined to be 1.

[0068] S204: Determine the target partition of the target end corresponding to the row data based on the partition value.

[0069] For example, after the partition value is determined, the target partition can be determined. For example, the target partition with the partition value of 1 is the partition with the date in 1990-01.

[0070] S205: Determine whether the original data stored in the target partition has been deleted.

[0071] Exemplarily, after determining the target partition, first determine whether the data before the target partition corresponding to the target partition (if there is data before) has been deleted. The determination method is, for example, to determine based on the mark of whether to delete data in the target partition. For example, if the mark of the target partition is true, it means that the original data has been deleted. If the target partition is not marked as true, the data needs to be deleted.

[0072] If the judgment result is yes, for example, the target partition has a true mark, S207 is executed to write the row data.

[0073] If the judgment result is no, for example, the target partition does not have a true mark, then S206 is executed to delete the original data of the target partition and set the target partition to a data deleted state.

[0074] It should be understood that whether the data of the target partition has been deleted may also be marked in other ways, such as marked as 0 or marked as 1, and the embodiments of the present application do not limit the marking of whether the data of the target partition has been deleted.

[0075] S206, deleting the original data of the target partition, and setting the target partition to a data deleted state.

[0076] Exemplarily, when the target partition stores original data, the data integration system deletes the original data of the target partition, and sets the target partition to a data-deleted state, waiting for new data to be written.

[0077] It should be understood that setting the target partition to a data deleted state is, for example, marking the target partition as true, indicating that the target partition has deleted the original data and does not need to delete the data in the subsequent data writing process.

[0078] S207, write row data.

[0079] Exemplarily, when the target partition does not store the original data, the data integration system writes the row data into the target partition, thereby completing the partition coverage business.

[0080] It should be understood that in the above scheme, the target end does not support the insert overwrite command. When the data integration system writes the source data to the target end, it needs to determine the partitioning rules of the target end and determine the partition value of the source data according to the partitioning rules to write the source data into the partition corresponding to the target end. Since the amount of data is large during data integration, the data integration system needs to determine the partition value of each row of data it writes. Therefore, the complexity of the data integration system to complete the partition overwrite service is high, and the data integration system needs to consume more computing resources to complete the partition overwrite service.

[0081] In some engines, a data replacement instruction is defined, which is used to replace the content in a specified partition in a table with the partition corresponding to the specified partition in another table (such as the target table). For example, the partition exchange command (exchange partition command) is used to exchange data in specified partitions of different tables. In other embodiments, the data replacement instruction can also be a command to delete data and insert data. For example, the data of the specified partition of the target table is first deleted through the truncate command, and then the corresponding data is written to the specified partition of the target table through the insert into command.

[0082] Based on the above-mentioned data replacement instruction, the present application provides a data processing method, which is applied to a data integration system. The method includes that when the data integration system detects an instruction to store the source data in the partition corresponding to the target table of the target end by partition overwriting, if the target end does not support the insert overwrite command, but the above-mentioned data replacement instruction is defined, a temporary table with the same partitioning rules as the partitioning rules of the target table can be established in real time on the target end, wherein the storage areas corresponding to the partitions of the temporary table are different from the storage areas corresponding to the partitions of the target table, and no data is stored in the storage areas corresponding to the partitions of the temporary table. Since no data is stored in the storage areas corresponding to the partitions of the temporary table, the source data can be stored in the partition corresponding to the temporary table without the insert overwrite command. Then, the data integration system can send a data replacement instruction to the target end, replace the data of the corresponding partition in the target table with the source data in the partition where the source data exists in the temporary table, and store the source data in the target table, thereby completing the partition coverage business of the corresponding partition of the target table.

[0083] In some embodiments, after the partition overwriting service is completed, the temporary table may be deleted.

[0084] Through this solution, when the data integration system performs partition coverage business, it does not need to write the source data directly into the partition corresponding to the target table on the target end. Therefore, it is not necessary to determine the partition rules of the target table corresponding to the target end, and there is no need to partition the source data. The data integration system sends the source data to the target end, and the target end writes the source data into the partition corresponding to the temporary table. After completing the writing of the source data, when the data integration system determines that there is data in the corresponding partition in the temporary table, it can execute the data replacement instruction to allow the data of the corresponding partition in the temporary table to replace the data of the partition corresponding to the target table, thereby completing the partition coverage. In this way, the data integration system does not need to determine the partition rules of the target end and the partition value of the source data each time when writing data to the target end, thereby reducing the complexity of the partition coverage business and reducing the computing resources consumed by the data integration system.

[0085] It should be understood that since the data integration system does not need to determine the partitioning rules of the target side when executing the partition coverage business, even if the engine of the target side contains custom partitioning rules, the source data can be accurately partitioned (the partitioning process is performed by the corresponding engine).

[0086] In some embodiments, the data replacement instruction may be, for example, a partition exchange command, which is used to exchange data of a partition corresponding to a temporary table with data of a partition corresponding to a target table. If the target end does not support the partition exchange command, a delete command for data of the partition corresponding to the target table is executed, and then the data of the partition corresponding to the temporary table is inserted into the partition corresponding to the target table, thereby completing the partition overwriting service.

[0087] The following describes a scenario for executing data integration services.

[0088] Figure 3 According to some embodiments of the present application, a scene graph of data integration is shown.

[0089] like Figure 3 As shown, in the embodiment of the present application, the data integration system, engine A as the source end, and engine B as the target end are respectively run in device 1, device 2, and device 3. The data integration system obtains source data from engine A and stores the source data in engine B.

[0090] It should be understood that device 1, device 2 and device 3 can be electronic devices such as servers, mobile phones or computers. Other types of engines, such as engine C, can also be run in device 2 and device 3.

[0091] Exemplarily, device 1 and device 2, and device 1 and device 3 may communicate via a communication interface. The communication interface may include, for example, a network interface.

[0092] Exemplarily, in other embodiments, the data integration system may run together with the engine at the source end in one device, or the data integration system may run together with the engine at the target end in one device.

[0093] Below, according to Figure 3 The above data integration scenario shows the process of writing source data to the target end by the data integration system.

[0094] Figure 4 According to an embodiment of the present application, an interactive process of a data integration system writing source data to a target end is shown. Figure 1 .

[0095] Exemplarily, in this embodiment, when the data integration system writes data to the target end, it first obtains the source data from an engine A on the source end, and then writes the data to an engine B on the target end.

[0096] It should be understood that when performing data integration, the engine A on the source side, the engine B on the target side, and the target table corresponding to the engine B on the target side have been determined (determined by the user). The data integration system only needs to obtain the source data from the determined engine A on the source side, and then write the source data to the determined engine B on the target side.

[0097] Exemplarily, the execution subject of each of the following processes is a data integration system, which may be, for example, an electronic device such as a server or a computer. When introducing each of the following processes, the execution subject of each step will not be repeatedly introduced.

[0098] It should be understood that the following process only introduces the process of the data integration system writing data to the target table on the target end, and the process of the data integration system obtaining data from the source end is not described in detail.

[0099] like Figure 4 As shown, the process includes:

[0100] S401, the data integration system 420 detects a write request for partition overlay.

[0101] Exemplarily, when the data integration system 420 detects a partition overwrite write request (i.e., the first instruction), it will execute the partition overwrite service. The partition overwrite write request may, for example, include an identifier of the source data on which the partition overwrite service needs to be executed, engine A at the source end 410 for acquiring the source data, engine B at the target end 430 for writing the source data, and an identifier of the target table corresponding to the source data in engine B.

[0102] Exemplarily, the source data identifier may be, for example, the identifier of each data in the source table in the engine of the source end 410, for example, the source data identifier is 1-100, with a total of 100 data. The target table identifier may be, for example, xxx, which is used in engine B to uniquely specify the target table.

[0103] Exemplarily, the write request for partition coverage may be sent by, for example, a client of the corresponding data integration system 420. The user sends the write request for partition coverage to the data integration system 420 through the application in the device 1.

[0104] S402, the data integration system 420 sends a request to the source end 410 to obtain source data.

[0105] Exemplarily, the write request for partition coverage includes the identifier of the source data and the identifier of the source end 410. The data integration system 420 can determine the engine A of the source end 410 according to the identifier of the source end 410, and then send a request to the source end 410 to obtain the source data corresponding to the identifier of the source data. The request may include the identifier of the source data, for example.

[0106] S403, the source end 410 sends source data to the data integration system 420.

[0107] Exemplarily, after receiving the request for acquiring source data sent by the data integration system, the source end 410 may send the source data to the data integration system 420 based on the identifier of the source data in the request.

[0108] S404, the data integration system 420 determines whether the target end 430 supports the partition overlay instruction.

[0109] Exemplarily, in some embodiments, the partition overwrite instruction is, for example, an insert overwrite command. The data integration system 420 stores a list of identifiers of engines that support the insert overwrite command. If the identifier of engine B of the target end 430 is in the above list, it can be determined that engine B supports the insert overwrite command, otherwise engine B does not support the insert overwrite command.

[0110] If the judgment result is yes, for example, it is detected that the target end 430 supports the insert overwrite command, S405 is executed, and the data integration system 420 sends the source data and the partition overwrite instruction for the source data to the target end 430 .

[0111] If the judgment result is no, for example, it is detected that the target end 430 does not support the insert overwrite command, S407 is executed, and the data integration system 420 sends an instruction to the target end 430 to create a temporary table.

[0112] It should be understood that in other embodiments, the data integration system 420 can also determine whether the engine B of the target end 430 supports the insert overwrite command in other ways. The embodiments of the present application do not limit the method of determining whether the engine supports the insert overwrite command.

[0113] S405 , the data integration system 420 sends the source data and a partition overwriting instruction for the source data to the target end 430 .

[0114] Exemplarily, the partition overwrite instruction is, for example, an insert overwrite command. If engine B supports the insert overwrite command, the data integration system 420 may send the source data and the insert overwrite command to engine B to write the source data into each partition corresponding to the target table.

[0115] S406, the target end 430 executes the partition overwrite instruction to store the source data into the corresponding partition.

[0116] For example, after receiving the insert overwrite command, the engine B of the target end 430 can complete the partition overwrite service based on the command, that is, delete the data of the partition corresponding to the source data, and then write the source data.

[0117] For example, taking the source data with the written identifier 6, name F, and date 1990-01-01 as an example, the partition overwrite write command can be, for example, insert overwrite xxx values(6, "F", '1990-01-01'). Where xxx is the identifier of the target table. It should be understood that in other engines, the partition overwrite write command can also be in other forms, and the embodiments of the present application do not limit the partition overwrite write command.

[0118] S407, the data integration system 420 sends an instruction to create a temporary table to the target end 430.

[0119] Exemplarily, if the data integration system 420 determines that the engine B of the target end 430 does not support the insertoverwrite command, an instruction (ie, the second instruction) for creating a temporary table is sent to the target end 430 .

[0120] Exemplarily, the instruction to create a temporary table may include, for example, the type of the created table, and the type of the table corresponds to the target table.

[0121] S408, the target end 430 executes the instruction to create a temporary table to create a temporary table.

[0122] Exemplarily, after receiving the instruction to create a temporary table, the target end 430 may create a temporary table having the same partition rule as the target table based on the instruction to create the temporary table.

[0123] It should be understood that the storage areas corresponding to the partitions in the temporary table are different from the storage areas corresponding to the partitions in the target table, and no data is stored in the partitions corresponding to the temporary table.

[0124] S409, the target end 430 sends a message to the data integration system 420 indicating that the temporary table creation is complete.

[0125] Exemplarily, after the target end 430 has established the temporary table, it will return a message to the data integration system 420 indicating that the establishment of the temporary table is complete.

[0126] S410, the data integration system 420 sends source data to the target end 430, as well as a storage instruction to store the source data in a temporary table.

[0127] Exemplarily, after receiving the message that the temporary table is created, the data integration system 420 may send the source data and a storage instruction (ie, the third instruction) to store the source data in the temporary table to the target end 430. The storage instruction may include identification information of the temporary table.

[0128] S411, the target end 430 executes the storage instruction to store the source data into the partition corresponding to the temporary table.

[0129] Exemplarily, after receiving the storage instruction, the engine B of the target end 430 executes the storage instruction to store the source data in the partition corresponding to the temporary table.

[0130] It should be understood that since the temporary table established in S408 is a newly established table, there is no data stored in the partitions corresponding to the temporary table. Therefore, the target end 430 can directly write the source data into the partitions corresponding to the temporary table according to the partitioning rules of the temporary table. The partitioning rules of the temporary table are the same as the partitioning rules of the target table. For example, for the row data with the identifier 6, the name F, and the date 1990-01-01. The corresponding partition column is the date, and the partition expression is that the dates of the same month are the same partition. Then the row data of the row data with the identifier 6, the name F, and the date 1990-01-01 is written into the partition with the partition value of 1 of the temporary table.

[0131] It should be understood that the storage area corresponding to the partition in the temporary table is different from the storage area corresponding to the partition in the target table.

[0132] It should be understood that in other embodiments, the target end 430 may also write the source data into corresponding partitions in the temporary table in batches.

[0133] S412, the data integration system 420 obtains the partition information corresponding to the kth partition in the temporary table.

[0134] Exemplarily, after the target end 430 writes the source data into the partition corresponding to the temporary table, the partition information of the corresponding partition in the temporary table will change.

[0135] Exemplarily, the partition information may include, for example, the number of rows of stored data in the corresponding partition. For example, if a partition with a partition value of 1 stores 3 rows of data, the partition information corresponding to the partition with a partition value of 1 may be 3. If the corresponding partition does not store data, the partition information may be 0, for example.

[0136] It should be understood that the partition information may be, for example, metadata information of the corresponding partition, and the metadata information is used to indicate information such as the amount of data stored in the corresponding partition and the data format.

[0137] Exemplarily, after the source data is stored in the temporary table, the data integration system 420 will traverse the partition information of all partitions in the temporary table, for example, it can traverse from the partition with a partition value of 1 to the last partition in order of partition values. In other embodiments, the data integration system 420 can also traverse from the last partition to the partition with a partition value of 1. The embodiments of the present application do not limit the manner in which the data integration system 420 traverses the partitions of the temporary table.

[0138] Exemplarily, the initial value of k may be 1, for example, and k=k+1 after each partition is traversed by the data integration system 420. In other embodiments, the initial value of k may also be the maximum value of the partitions in the temporary table, and k=k-1 after each partition is traversed by the data integration system 420. The embodiments of the present application do not limit the manner in which the data integration system 420 records the partitions traversed.

[0139] S413, the data integration system 420 determines whether there is data in the kth partition based on the partition information of the kth partition.

[0140] Exemplarily, after the data integration system 420 obtains the partition information of the kth partition of the temporary table, it determines whether the number of rows of data stored in the partition is greater than 0. If the number of rows of data in the partition is greater than 0, it means that new data is stored in the partition, and the data of the kth partition needs to be replaced with the kth partition corresponding to the target table.

[0141] If the judgment result is yes, for example, if it is judged that the number of rows of data in the kth partition is greater than 0, S414 is executed, and the data integration system 420 sends a data replacement instruction for the kth partition of the temporary table and the target table to the target end 430 .

[0142] If the result of the determination is negative, for example, if the number of rows of data in the kth partition is 0, S416 is executed, and the data integration system 420 determines whether the kth partition is the last partition traversed in the temporary table.

[0143] S414 , the data integration system 420 sends a data replacement instruction for the kth partition of the temporary table and the target table to the target end 430 .

[0144] Exemplarily, when the data integration system 420 determines that the number of rows of data in the kth partition of the temporary table is greater than 0, it can be determined that the partition stores new data. At this time, the data integration system 420 sends a data replacement instruction (i.e., the fourth instruction) to the target end 430 to replace the data of the temporary table corresponding to the partition with the data of the target table corresponding to the partition, thereby completing the partition coverage business of the partition.

[0145] S415 , the target end 430 executes the data replacement instruction to replace the partition data of the kth partition corresponding to the temporary table with the kth partition corresponding to the target table.

[0146] Exemplarily, in some embodiments, the data replacement instruction is, for example, a partition exchange command. For example, after receiving the partition exchange command exchange partition for the kth partition sent by the data integration system 420, the engine B can exchange the data corresponding to the kth partition in the temporary table with the data corresponding to the partition in the target table to complete the partition coverage business for the kth partition of the target table.

[0147] In other embodiments, if the engine B of the target end 430 does not support the partition exchange service, the data integration system 420 sends a command truncate to delete the data of the kth partition of the target table to the engine B of the target end 430, and the engine B executes the deletion of the data of the kth partition of the target table. Then, the data integration system 420 sends a command insert into to insert the data of the kth partition in the temporary table into the partition corresponding to the target table to the engine B of the target end 430, and the engine B executes the above command to insert the data of the kth partition corresponding to the temporary table into the partition corresponding to the target table, thereby completing the partition coverage service for the kth partition.

[0148] S416, the data integration system 420 determines whether the kth partition is the last partition traversed in the temporary table.

[0149] Exemplarily, if the number of rows of data in the kth partition is 0, it means that no new data is stored in the partition, and the execution of the data replacement instruction on the partition is skipped, and it is determined whether the partition is the last partition traversed in the temporary table.

[0150] Alternatively, after the data integration system 420 executes S413, S416 is also executed to determine whether the kth partition is the last partition traversed in the temporary table.

[0151] If the judgment result is yes, for example, the kth partition is the last partition traversed in the temporary table, S417 is executed, and the data integration system 420 completes the partition coverage business.

[0152] If the judgment result is no, for example, the kth partition is not the last partition traversed in the temporary table, k is updated so that k=k+1, and then go to S412, the data integration system 420 obtains the partition information corresponding to the kth partition in the temporary table.

[0153] It should be understood that there is no order in which S416, S414, and S415 are executed. In some embodiments, the data integration system 420 first executes S414, then S415, and finally S416. In other embodiments, the data integration system 420 may also first execute S416, then S414 and S415.

[0154] S417, the data integration system 420 completes the partition coverage business.

[0155] Exemplarily, when the data replacement is completed for the last partition in the temporary table, the data integration system 420 completes the partition coverage service.

[0156] It can be understood that in the embodiments of the present application, if the engine B on the target side does not support the partition overwrite instruction, the data integration system does not need to determine the partition value of each row of data when writing data to the target side, and can directly send the source data to the target side, which writes the source data to the partition corresponding to the temporary table. After completing the writing of the source data, the data integration module traverses the partition information of all partitions in the temporary table, and replaces the corresponding data in the partition where the partition information indicates that there is data with the corresponding partition in the target table, thereby completing the partition overwrite service. Through the embodiments of the present application, the data integration system completes the partition overwrite service with low complexity, saving the computing resources of the data integration system. And because the data integration system does not need to determine the partition value of the source data, the data integration system can also accurately perform the partition overwrite service for tables whose target side partition expressions are custom rules.

[0157] In other embodiments, when traversing all partitions of the temporary table of the target end 430, the data integration system 420 can simultaneously obtain the partition information of all partitions in the temporary table, and then determine the partitions with data in the temporary table based on the partition information, and then send a data replacement instruction to the target end 430 to replace the partitions with data in the temporary table determined by the data integration system with the partitions corresponding to the target table, thereby completing the partition coverage business.

[0158] For example, Figure 5 According to some embodiments of the present application, an interactive process of a data integration system 420 writing source data to a target end 430 is shown. Figure 2 .like Figure 5 As shown, the process includes:

[0159] S501, the data integration system 420 detects a write request for partition overlay.

[0160] S502, the data integration system 420 sends a request to the source end 410 to obtain source data.

[0161] S503, the source end 410 sends source data to the data integration system 420.

[0162] S504, the data integration system 420 determines whether the target end 430 supports the partition overlay instruction.

[0163] S505 , the data integration system 420 sends the source data and a partition overwriting instruction for the source data to the target end 430 .

[0164] S506, the target end 430 executes the partition overwrite instruction to store the source data into the corresponding partition.

[0165] S507, the data integration system 420 sends an instruction to create a temporary table to the target end 430.

[0166] S508, the target end 430 executes the instruction to create a temporary table to create a temporary table.

[0167] S509, the target end 430 sends a message to the data integration system indicating that the temporary table creation is complete.

[0168] S510, the data integration system 420 sends source data to the target end 430, as well as a storage instruction to store the source data in a temporary table.

[0169] S511, the target end 430 executes the storage instruction to store the source data into the partition corresponding to the temporary table.

[0170] Exemplarily, the above process from S501 to S511 may refer to the process from S401 to S411.

[0171] S512, the data integration system 420 sends an instruction to the target end 430 to obtain the partition information of each partition in the temporary table.

[0172] S513, the target end 430 sends the partition information of each partition in the temporary table to the data integration system 420.

[0173] S514, the data integration system 420 determines the partition storing the source data based on the partition information of each partition.

[0174] S515, the data integration system 420 sends a data replacement instruction for the partition storing the source data to the target end 430.

[0175] Exemplarily, the data replacement instruction may include identification information of a partition storing source data.

[0176] S516 , the target end 430 executes the data replacement instruction to replace the partition of the temporary table storing the source data with the partition corresponding to the target table.

[0177] Exemplarily, the data replacement process refers to S415.

[0178] S517, the data integration system 420 completes the partition coverage business.

[0179] It can be understood that in some embodiments of the present application, the data integration system 420 can also directly obtain the partition information of each partition corresponding to the temporary table, and determine the partition where the temporary table stores the source data based on the partition information, and then send a data replacement instruction for the partition storing the active data, thereby completing the partition coverage business.

[0180] An embodiment of the present application also provides an electronic device, which may be, for example, a source end, a data integration system, or a physical machine such as a server, a mobile phone, a computer, etc. at a target end.

[0181] The following takes the host as the target end as an example to introduce the electronic device in the embodiment of the present application.

[0182] Figure 6 According to some embodiments of the present application, a schematic structural diagram of a host is shown.

[0183] like Figure 6 As shown, the host 100 includes at least one processor 101 , at least one memory 102 , and a communication interface 103 .

[0184] The processor 101 may be a CPU, configured to execute the aforementioned data replacement instruction to implement the data processing method provided in each embodiment of the present application.

[0185] It should be understood that each processor 101 may be connected to at least one memory 102 for storing execution and data during the operation of the host 100. In some embodiments, the memory 102 may be a volatile memory, such as a random access memory (RAM), a static random access memory (SRAM), a dynamic random access memory (DRAM), a synchronous dynamic random access memory (SDRAM), a double data rate synchronous dynamic random access memory (DDR SDRAM), an enhanced synchronous dynamic random access memory (ESDRAM), a synchronous link dynamic random access memory (SLDRAM), etc.

[0186] It should be understood that the data corresponding to the partition in the aforementioned target table may be stored in the aforementioned memory 102 , for example.

[0187] Optionally, in some embodiments, the host 100 may further include a non-volatile memory for storing instructions and data involved in the methods provided in the embodiments of the present application.

[0188] The communication interface 103 is used for the host 100 to communicate with other electronic devices (such as a data integration system). The communication interface 103 may include, for example, a network interface.

[0189] It should be understood that Figure 6 The structure of the host 100 shown is only an example. In other embodiments, the host 100 may also include more or fewer modules, which is not limited here.

[0190] The various embodiments of the mechanism disclosed in the present application can be implemented in hardware, software, firmware or a combination of these implementation methods. The embodiments of the present application can be implemented as a computer program or program code executed on a programmable system, which includes at least one processor, a storage system (including volatile and non-volatile memory and / or storage elements), at least one input device and at least one output device.

[0191] Program code can be applied to input instructions to perform the functions described in this application and generate output information. The output information can be applied to one or more output devices in a known manner. For the purposes of this application, a processing system includes any system having a processor such as, for example, a digital signal processor, a microcontroller, an application specific integrated circuit, or a microprocessor.

[0192] Program code can be implemented with high-level programming language or object-oriented programming language to communicate with the processing system. When necessary, program code can also be implemented with assembly language or machine language. In fact, the mechanism described in this application is not limited to the scope of any specific programming language. In either case, the language can be a compiled language or an interpreted language.

[0193] In some cases, the disclosed embodiments may be implemented in hardware, firmware, software, or any combination thereof. The disclosed embodiments may also be implemented as instructions carried or stored on one or more temporary or non-temporary machine-readable (e.g., computer-readable) storage media, which may be read and executed by one or more processors. For example, instructions may be distributed over a network or by other computer-readable media. Therefore, a machine-readable medium may include any mechanism for storing or transmitting information in a machine (e.g., computer) readable form, including but not limited to, a floppy disk, an optical disk, an optical disk, a read-only memory, a magneto-optical disk, a random access memory, an erasable programmable read-only memory, an electrically erasable programmable read-only memory, a magnetic card or an optical card, a flash memory, or a tangible machine-readable memory for transmitting information (e.g., carrier waves, infrared signal digital signals, etc.) using the Internet in electrical, optical, acoustic, or other forms of propagation signals. Therefore, a machine-readable medium includes any type of machine-readable medium suitable for storing or transmitting electronic instructions or information in a machine (e.g., computer) readable form.

[0194] In the accompanying drawings, some structural or method features may be shown in a specific arrangement and / or order. However, it should be understood that such a specific arrangement and / or order may not be required. Instead, in some embodiments, these features may be arranged in a manner and / or order different from that shown in the illustrative drawings. In addition, the inclusion of structural or method features in a particular figure does not mean that such features are required in all embodiments, and in some embodiments, these features may not be included or may be combined with other features.

[0195] It should be noted that the units / modules mentioned in the various device embodiments of the present application are all logical units / modules. Physically, a logical unit / module can be a physical unit / module, or a part of a physical unit / module, or can be implemented as a combination of multiple physical units / modules. The physical implementation method of these logical units / modules themselves is not the most important. The combination of functions implemented by these logical units / modules is the key to solving the technical problems proposed by the present application. In addition, in order to highlight the innovative part of the present application, the above-mentioned device embodiments of the present application do not introduce units / modules that are not closely related to solving the technical problems proposed by the present application, which does not mean that there are no other units / modules in the above-mentioned device embodiments.

[0196] It should be noted that in the examples and description of this patent, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "including one" do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0197] Although the present application has been illustrated and described with reference to certain preferred embodiments thereof, it will be apparent to those skilled in the art that various changes in form and details may be made therein without departing from the spirit and scope of the present application.

Claims

1. A data storage method, characterized in that: The method comprises: A first instruction is detected, wherein the first instruction is used to instruct to store first data of the first electronic device into at least one first storage partition in the first storage partition table of the second electronic device; sending a second instruction to a second electronic device, wherein the second instruction is used to instruct the second electronic device to establish a second storage partition table corresponding to the first storage partition table, and The first storage partition table and the second storage partition table have the same data partitioning rules, and the second storage partition table contains a plurality of second storage partitions corresponding to the first storage partitions in the first storage partition table; In response to detecting that the second electronic device completes the establishment of the first storage partition table, sending a third instruction to the second electronic device, wherein the third instruction is used to instruct the second electronic device to store the first data in the second storage partition corresponding to the second storage partition table based on the data partition rule; A fourth instruction is sent to the second electronic device, wherein the fourth instruction is used to instruct the second electronic device to replace data in at least one of the second storage partitions storing the first data in the second storage partition table with data in the corresponding first storage partition in the first storage partition table.

2. The method according to claim 1, characterized in that The data partitioning rule includes a partition column and a partition expression, wherein the partition column is used to indicate a column in at least one row of data stored in a storage partition table for determining a storage partition corresponding to each row of data; and the partition expression is used to indicate a correspondence between the content of the partition column in each row of data in the at least one row of data and a storage partition storing the corresponding row of data.

3. The method according to claim 1, characterized in that The method further comprises: After the second electronic device executes the fourth instruction, the second storage partition table and the data in each of the second storage partitions are deleted.

4. The method according to claim 1, characterized in that: The fourth instruction includes at least one sub-instruction corresponding to each second storage partition in the at least one second storage partition storing the first data, and each of the sub-instructions is used to instruct the second electronic device to replace the data in the second storage partition corresponding to the sub-instruction with the first storage partition corresponding to the second storage partition corresponding to the sub-instruction in the first storage partition table.

5. The method according to claim 4, characterized in that The sending a fourth instruction to the second electronic device includes: The second storage partitions corresponding to the second storage partition table are traversed, and when data is stored in the second storage partition currently traversed, the sub-instruction for the second storage partition currently traversed is sent to the second electronic device.

6. The method according to claim 5, characterized in that In the case that data is stored in the second storage partition currently traversed, sending the sub-instruction for the second storage partition currently traversed to the second electronic device includes: When the number of rows of data stored in the second storage partition currently traversed is not 0, the sub-instruction for the second storage partition currently traversed is sent to the second electronic device.

7. The method according to any one of claims 4 to 6, characterized in that: The sub-instruction includes a partition exchange instruction for the second storage partition corresponding to the sub-instruction.

8. The method according to any one of claims 4 to 6, characterized in that: The sub-instruction includes a delete instruction for deleting the data of the first storage partition corresponding to the sub-instruction in the first storage partition table, and an insert instruction for inserting the data in the second storage partition corresponding to the sub-instruction in the second storage partition table into the first storage partition corresponding to the sub-instruction in the first storage partition table.

9. An electronic device, characterized in that: The electronic device comprises a memory for storing instructions; At least one processor is used to execute the instructions to enable the electronic device to implement the method according to any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that: The readable storage medium stores instructions, and when the instructions are executed on a computer, the computer is caused to execute the method according to any one of claims 1 to 8.

11. A computer program product, characterized in that When the computer program product is executed on a device, the device is enabled to implement the method according to any one of claims 1 to 8.

Images