Data integration method and related equipment

Abstract

The embodiment of the invention discloses a data integration method and related equipment, which are used for improving the data integration efficiency. The method comprises the steps of obtaining a data integration instruction between a source system and a target system, wherein the data integration instruction comprises a query condition; on the basis of the query condition, sequentially reading multiple pieces of data to be integrated from the source system; if any piece of to-be-integrated data is read, integrating the to-be-integrated data to the target system, determining the read to-be-integrated data as read data, and updating a first read data abstract based on the identification information of each piece of read data, the first read data abstract is used for identifying the uniqueness of each piece of read data; if the data integration between the source system and the target system is interrupted, integration recovery information is generated, and the integration recovery information comprises the query condition and the first read data abstract.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of data integration, and in particular to a data integration method and related equipment. Background Art

[0002] When doing system integration for large enterprise customers, it often involves the integration of large-scale data sets. A data integration may involve tens of millions of rows of data, with a data volume of hundreds of GB or even more. Such large-scale data integration may encounter errors during its execution, and it is often necessary to pause or interrupt it, eliminate the error factors, and then resume the integration.

[0003] In the existing technical solution, full re-execution can be adopted. Generally, it is necessary to obtain all the data to be integrated from the source system, and re-integrate each piece of data to be integrated into the target system. In addition, since part of the data to be integrated has been integrated into the target system before the integration is interrupted, in the process of re-integrating each piece of data to be integrated into the target system, it is also necessary to combine the detection of the data status of the target system and ignore the duplicate integrated data.

[0004] When the amount of data to be integrated is extremely large, if the above-mentioned integration recovery solution of full re-execution is adopted, each piece of data to be integrated processed before the integration interruption needs to be reprocessed one by one, which will cost a lot of time and result in low data integration efficiency from the source system to the target system. Summary of the invention

[0005] The embodiments of the present application provide a data integration method and related devices for improving data integration efficiency.

[0006] A first aspect of an embodiment of the present application provides a data integration method, including:

[0007] Obtaining a data integration instruction between a source system and a target system, wherein the data integration instruction includes a query condition;

[0008] Based on the query condition, multiple pieces of data to be integrated are read from the source system in sequence;

[0009] If any of the data to be integrated is read, the data to be integrated is integrated into the target system, and the read data to be integrated is determined as the read data, and based on the identification information of each piece of the read data, a first digest of the read data is updated, where the first digest of the read data is used to identify the uniqueness of each piece of the read data;

[0010] If the data integration between the source system and the target system is interrupted, integration recovery information is generated, and the integration recovery information includes the query condition and the first read data summary.

[0011] In a specific implementation, the first read data digest is generated based on a preset digest algorithm, and after generating the integrated recovery information, the method further includes:

[0012] receiving an integrated recovery instruction corresponding to the data integration instruction, the integrated recovery instruction having the corresponding integrated recovery information;

[0013] Based on the query condition, multiple pieces of data to be queried are read sequentially from the source system;

[0014] If any of the to-be-checked data is read, then based on the identification information of each of the to-be-checked data that has been read and the preset summary algorithm, a second digest of the read data is updated, wherein the second digest of the read data is used to identify the uniqueness of each of the to-be-checked data that has been read;

[0015] If the second read data digest is consistent with the first read data digest in the integrated recovery information, each piece of data to be checked that is subsequently read is integrated into the target system.

[0016] In a specific implementation, the integrated recovery information further includes a first read quantity, and the method further includes:

[0017] If the step of determining the read data to be integrated as read data is performed, a first read quantity is updated, where the first read quantity is used to identify the total quantity of the data to be integrated read from the source system.

[0018] In a specific implementation, the first read data digest is generated based on a preset digest algorithm, and after generating the integrated recovery information, the method further includes:

[0019] receiving an integrated recovery instruction corresponding to the data integration instruction, the integrated recovery instruction having the corresponding integrated recovery information;

[0020] Based on the query condition, multiple pieces of data to be queried are read sequentially from the source system;

[0021] If any of the to-be-checked data is read, a second digest of the read data is updated based on the identification information of each of the to-be-checked data that has been read and the preset digest algorithm, and a second read quantity is updated, wherein the second digest of the read data is used to identify the uniqueness of each of the to-be-checked data that has been read, and the second read quantity is used to identify the total quantity of the to-be-checked data read from the source system;

[0022] If the second read data summary is consistent with the first read data summary in the integrated recovery information, and the second read quantity is consistent with the first read quantity in the integrated recovery information, each piece of data to be checked that is read thereafter is integrated into the target system.

[0023] If the second read count is consistent with the first read count in the integrated recovery information, and the second read data digest is inconsistent with the first read data digest in the integrated recovery information, then the integrated recovery operation is interrupted.

[0024] In a specific implementation, updating the first read data digest based on the identification information of each piece of the read data includes:

[0025] Inputting the identification information of the first piece of read data into a preset digest algorithm, and using the first data digest output by the preset digest algorithm as the first read data digest;

[0026] The identification information of the i-th read data is input into a preset summary algorithm to obtain the i-th data summary determined based on the i-1-th first read data summary and the identification information of the i-th read data, and the first read data summary is updated to the i-th data summary.

[0027] In a specific implementation, the query condition includes a filtering condition and a sorting condition.

[0028] A second aspect of an embodiment of the present application provides a data integration system, including:

[0029] An acquisition unit, used to acquire a data integration instruction between a source system and a target system, wherein the data integration instruction includes a query condition;

[0030] A reading unit, configured to sequentially read a plurality of pieces of data to be integrated from the source system based on the query condition;

[0031] The reading unit is further configured to integrate the data to be integrated into the target system if any of the data to be integrated is read, determine the read data to be integrated as read data, and update a first read data digest based on identification information of each piece of the read data, wherein the first read data digest is used to identify the uniqueness of each piece of the read data;

[0032] A recovery unit is used to generate integration recovery information if the data integration between the source system and the target system is interrupted, wherein the integration recovery information includes the query condition and the first read data summary.

[0033] In a specific implementation, the first read data digest is generated based on a preset digest algorithm, and after the integrated recovery information is generated, the data integration system further includes: a receiving unit;

[0034] The receiving unit is used to receive an integrated recovery instruction corresponding to the data integration instruction, wherein the integrated recovery instruction has the corresponding integrated recovery information;

[0035] The reading unit is further used to sequentially read multiple pieces of to-be-queried data from the source system based on the query condition;

[0036] The recovery unit is further configured to update a second digest of the read data based on the identification information of each piece of the read data to be checked and the preset digest algorithm if any piece of the data to be checked is read, wherein the second digest of the read data is used to identify the uniqueness of each piece of the read data to be checked;

[0037] The recovery unit is further configured to integrate each piece of to-be-queried data subsequently read into the target system if the second read data digest is consistent with the first read data digest in the integrated recovery information.

[0038] In a specific implementation, the integrated recovery information also includes a first read quantity, and the reading unit is further used to update the first read quantity if the step of determining the read data to be integrated as read data is executed, and the first read quantity is used to identify the total number of data to be integrated read from the source system.

[0039] In a specific implementation, the first read data digest is generated based on a preset digest algorithm, and after the integrated recovery information is generated, the data integration system further includes: a receiving unit;

[0040] The receiving unit is used to receive an integrated recovery instruction corresponding to the data integration instruction, wherein the integrated recovery instruction has the corresponding integrated recovery information;

[0041] The reading unit is further used to sequentially read multiple pieces of to-be-queried data from the source system based on the query condition;

[0042] The reading unit is further configured to update a second read data digest based on identification information of each piece of the read data to be checked and the preset digest algorithm, and update a second read quantity if any piece of the data to be checked is read, wherein the second read data digest is used to identify the uniqueness of each piece of the read data to be checked, and the second read quantity is used to identify the total quantity of the data to be checked read from the source system;

[0043] The recovery unit is also used to integrate each piece of data to be checked that is subsequently read into the target system if the second read data summary is consistent with the first read data summary in the integrated recovery information, and the second read quantity is consistent with the first read quantity in the integrated recovery information.

[0044] The recovery unit is further configured to interrupt the integrated recovery operation if the second read quantity is consistent with the first read quantity in the integrated recovery information and the second read data summary is inconsistent with the first read data summary in the integrated recovery information.

[0045] In a specific implementation, the reading unit is specifically configured to input the identification information of the first read data into a preset digest algorithm, and use the first data digest output by the preset digest algorithm as the first read data digest;

[0046] The identification information of the i-th read data is input into a preset summary algorithm to obtain the i-th data summary determined based on the i-1-th first read data summary and the identification information of the i-th read data, and the first read data summary is updated to the i-th data summary.

[0047] In a specific implementation, the query condition includes a filtering condition and a sorting condition.

[0048] A third aspect of an embodiment of the present application provides a data integration device, including:

[0049] CPU, memory and input / output interface;

[0050] The memory is a short-term storage memory or a persistent storage memory;

[0051] The central processing unit is configured to communicate with the memory and execute instructions in the memory to perform the method described in the first aspect.

[0052] A fourth aspect of the embodiments of the present application provides a computer program product comprising instructions, and when the computer program product is run on a computer, the computer is caused to execute the method described in the first aspect.

[0053] A fifth aspect of an embodiment of the present application provides a computer storage medium, wherein the computer storage medium stores instructions, and when the instructions are executed on a computer, the computer executes the method described in the first aspect.

[0054] It can be seen from the above technical solution that the embodiment of the present application has the following advantages: In the embodiment of the present application, only by using the integration recovery information including the query condition and the first read data summary, the integrated data among the multiple pieces of data to be integrated and whether these data have been modified can be accurately determined. In addition, during the entire integration recovery process, the data to be integrated that was integrated into the target system before the integration was interrupted does not need to be repeatedly integrated, which helps to improve the overall data integration efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 A system architecture diagram of the data integration method disclosed in the embodiment of the present application;

[0056] Figure 2 A schematic diagram of a flow chart of the data integration method disclosed in an embodiment of the present application;

[0057] Figure 3 A schematic diagram of the structure of the data integration system disclosed in the embodiment of the present application;

[0058] Figure 4 A schematic diagram of the structure of a computer device disclosed in an embodiment of the present application. DETAILED DESCRIPTION

[0059] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0060] The embodiments of the present application provide a data integration method and related devices for improving data integration efficiency.

[0061] In order to better implement the data integration method of the embodiment of the present application, the present application provides the following Figure 1 The data integration architecture shown in the figure includes a source system, a target system and a data integration system. The source system, the target system and the data integration system can be deployed on the same, completely different or not completely the same computer devices. The purpose of data integration is to integrate the data to be integrated in the source system into the target system.

[0062] It should be noted that the source system and target system in data integration refer to two different data management systems, which usually have different or non-identical fields. For example, business data is migrated from an old data management system to a new data management system. The source system and target system in data synchronization and data migration usually refer to two identical business systems or data management systems. Data synchronization and migration generally migrate data from the source end where system A is deployed to the target end where system A is deployed. It can be seen that in the actual data integration process, it is first necessary to read multiple pieces of data to be integrated from the source system, and then continue to perform data conversion operations such as field mapping or field calculation on each piece of data to be integrated, and then the converted data to be integrated can be integrated into the target system. See the following embodiments for details.

[0063] in, Figure 1 The data shown in D1, D2, Di, Di+1, Di+2, Dn, etc. are all data to be integrated read from the source system and are based on Figure 1 The data in the solid-line box of the reading process shown in the figure are the read data, and the data in the dotted-line box are the data to be integrated.

[0064] Based on the aforementioned data integration architecture, the present application embodiment provides the following data integration method, including:

[0065] 201. Obtain data integration instructions between a source system and a target system, where the data integration instructions include query conditions.

[0066] When the integration system receives a data integration instruction between a source system and a target system, it indicates that the integration system needs to obtain the data to be integrated from the source system, and convert the obtained data to be integrated and store it in the target system.

[0067] It is understandable that in order to complete the instructions of the data integration instruction, the data to be integrated should first be obtained from the source system, and there is usually a large amount of business data in the source system. In the data integration scenario, all business data in the source system are generally not regarded as data to be integrated into the target system. Therefore, in order to accurately obtain the data to be integrated from the business system that needs to be integrated into the target system, the data integration instruction will include query conditions. Generally, the query conditions include but are not limited to at least one of the filtering conditions and the sorting conditions. Among them, the filtering conditions are used to filter the data to be integrated from the business system that needs to be integrated into the target system, and the sorting conditions are used to arrange the data to be integrated obtained from the business system according to certain rules, such as arranging from large to small according to the field value of a field of the data to be integrated. The specific rules used for sorting can be set by default or as needed, and are not specifically limited here.

[0068] 202. Based on the query condition, multiple pieces of data to be integrated are read from the source system in sequence.

[0069] Based on the above embodiments, it can be known that the role of the query condition is to accurately obtain the data to be integrated in the business system that needs to be integrated into the target system. Therefore, each business data found in the source system that meets the query condition should be the data to be integrated. In addition, if the query condition does not clearly define the sorting condition of the acquired data to be integrated, considering that the data to be integrated is obtained from the database of the source system, therefore, multiple data to be integrated have a relative physical order in the database table, which can be used as the default sorting of multiple data to be integrated in the embodiment of the present application.

[0070] In practical applications, in order to free up memory space in the data integration system, it is not necessary to read all the data to be integrated that meets the query conditions at once and integrate them into the integration system. Instead, a streaming reading method can be used to obtain only part of the data to be integrated each time.

[0071] 203. If any data to be integrated is read, the data to be integrated is integrated into the target system, and the read data to be integrated is determined as the read data, and based on the identification information of each read data, the first read data summary is updated, and the first read data summary is used to identify the uniqueness of each read data.

[0072] In order to ensure accuracy, the data integration system usually needs to process multiple data to be integrated one by one. Therefore, for each piece of data to be integrated that is read, in addition to performing the integration operation on it, it is also necessary to generate the latest first read data summary based on its own identification information and the identification information of all the read data read before it, and replace the previous one with the latest first read data summary. In other words, after each piece of data to be integrated is read, it is necessary to obtain the identification information of each piece of read data read at the current moment and before (the data to be integrated becomes the read data after being read) to update the first read data summary.

[0073] In order to ensure that the first read data digest can be used to accurately confirm the read data in multiple pieces of data to be integrated during the recovery process of data integration, the updated first read data digest must at least be able to identify the uniqueness of each piece of read data.

[0074] The identification information includes the corresponding data to be integrated or the field values ​​of some preset fields in the data to be integrated, such as at least one of a timestamp field and a data row hash value field, which is not limited here.

[0075] 204. If the data integration between the source system and the target system is interrupted, generate integration recovery information, where the integration recovery information includes a query condition and a summary of the first read data.

[0076] Based on the above embodiments, it can be known that the query condition determines the data to be integrated that can be obtained from the source system, and the first read data summary accurately identifies the uniqueness of multiple read data. Therefore, generating integration recovery information including the query condition and the first read data summary helps the data integration system to obtain multiple data to be checked based on the query condition after analyzing and eliminating the factors that cause the integration interruption, and determine the data that has been read based on the first read data summary, so as to avoid repeated integration of the same data.

[0077] It should be noted that, as can be seen from the aforementioned embodiments, in a complete data integration process, it is necessary to first perform a data reading operation, and then perform a data integration operation on the read data. Similarly, in the integrated recovery process, it is necessary to first perform a data reading operation, and then perform a data integration operation on the read data. The difference is that, in order to avoid repeatedly performing data integration operations, the embodiments of the present application propose integrated recovery information that focuses on each read data. Through the query conditions in the integrated recovery information and the first read data summary, the read data in the previously interrupted integration can be found, and the integration operation is not performed on these data again, thereby avoiding the reduction in integration efficiency caused by repeated integration.

[0078] In actual applications, there is also a data recovery strategy called local full cache. Specifically, the data set generation logic (for example, database SELECT statements, data query API calls, etc.) is first executed to obtain the source data. Secondly, starting from the first row of data, the data is temporarily stored on the server's storage device. Finally, when the data integration is suspended due to an exception, the position of the processed data row is recorded. When it is re-executed next time, when reading data from the local storage device, the data is read from the last recorded position.

[0079] Among them, the data format of the integrated recovery information can adopt a standard data format, such as: JSON string, XML string, etc., or a dedicated private format can be used; the storage facility of the integrated recovery information can be a database, a local disk file, a distributed file system, etc., which is not limited here.

[0080] Based on the above content, it can be known that the local full cache increases the hardware cost of the server and occupies a large amount of additional I / O traffic because it is necessary to temporarily store the data set to be integrated on the storage device of the server; in a distributed environment, in order to ensure that each node in the cluster can access the temporarily stored data, it is necessary to use an independent network storage facility, which places higher requirements on the network bandwidth. In the embodiment of the present application, only by including the first read data summary and the integrated recovery information of the query conditions, it is possible to accurately determine the read data in multiple pieces of data to be integrated and whether these data have been modified. And in the entire integrated recovery process, the data to be integrated that was integrated into the target system before the integration was interrupted does not need to be integrated repeatedly, which helps to improve the overall data integration efficiency.

[0081] On the basis of the aforementioned embodiment, after the aforementioned step 204, the present application can specifically realize the recovery of data integration by simulating the data reading operation in data integration, as detailed in the following steps: receiving an integrated recovery instruction corresponding to the data integration instruction, the integrated recovery instruction having corresponding integrated recovery information; based on the query conditions, reading multiple pieces of data to be checked from the source system in sequence; if any piece of data to be checked is read, then updating a second digest of the read data based on the identification information of each piece of data to be checked that has been read and a preset digest algorithm, the second digest of the read data being used to identify the uniqueness of each piece of data to be checked that has been read; if the second digest of the read data is consistent with the first digest of the read data in the integrated recovery information, then integrating each piece of data to be checked that is subsequently read into the target system.

[0082] First, after receiving an integrated recovery instruction for a certain data integration, integrated recovery information can be obtained from the integrated recovery instruction, wherein the integrated recovery instruction for a certain data integration includes an integrated recovery instruction corresponding to the data integration instruction.

[0083] Secondly, after the integration recovery information is obtained, multiple pieces of data to be checked can be obtained from the source system according to the query conditions in the integration recovery information. It is understandable that if the data in the source system has not changed since the last data integration interruption and before the integration recovery instruction is received, then the multiple pieces of data to be checked queried in the source system based on the aforementioned query conditions should be completely consistent with the multiple pieces of data to be integrated queried in step 202.

[0084] Finally, the operation is similar to data integration. In the process of data recovery, it is equivalent to repeating similar steps of data integration. The difference is that it is not necessary to perform integration operations for each piece of data to be checked that is read. Specifically, if any data to be checked is read, the second read data summary is updated based on the identification information of each piece of data to be checked that has been read and the preset summary algorithm. The second read data summary is used to identify the uniqueness of each piece of data to be checked that has been read. Among them, the preset summary algorithm used to update the second read data summary is the same as the preset summary algorithm used to update the first read data summary. Therefore, if a certain piece of data to be checked is read, the updated second read data summary is consistent with the first read data summary in the integrated recovery information, then this piece of data to be checked is the last data read before the integration is interrupted. That is to say, in the embodiment of the present application, for each piece of data to be checked that is read after this piece of data to be checked, it is necessary to start the integration operation.

[0085] It should be noted that when the second read data summary is consistent with the first read data summary in the integrated recovery information, each piece of data to be checked that is read thereafter needs to be processed as data to be integrated, just like the first data integration. The specific processing method is detailed in the aforementioned related embodiments and will not be repeated here.

[0086] In addition, if the second read data digest after each update is different from the first read data digest until all the data to be checked are traversed, it means that the multiple pieces of data to be checked obtained from the source system according to the query conditions are different from the multiple pieces of data to be integrated obtained in step 202, and data recovery without duplication integration is no longer possible. You can choose to send a recovery termination notification, which is used to indicate that the aforementioned data recovery instruction cannot be executed. However, after this, you can choose to initiate a new data integration instruction to re-integrate the data instead of recovering the data.

[0087] Furthermore, if the integrated recovery information also includes a first read number, then for each read data to be integrated, an embodiment of the present application also includes: if the step of determining the read data to be integrated as read data is executed, the first read number is updated, wherein the first read number is used to identify the total number of read data.

[0088] Specifically, like the update of the summary of the first data that has been read, if any of the data to be integrated is read, the data to be integrated will definitely be determined as the read data. Therefore, in addition to increasing the first read quantity by 1 after each data to be integrated is read, the update of the first read quantity can also be triggered by the execution of the step of "determining the read data to be integrated as the read data". The first read quantity update operation in the embodiment of the present application can be configured as needed, and it is only necessary to ensure the atomicity between the three transactions of updating the first read quantity, updating the summary of the first data that has been read, and determining the read data to be integrated as the read data, that is, the three must occur at the same time, or not occur at the same time.

[0089] Similarly, when the integrated recovery information also includes a first read quantity, the embodiment of the present application can complete the data recovery for the data to be checked with higher recovery efficiency, which can be specifically implemented in the following manner: receiving an integrated recovery instruction corresponding to the data integration instruction, the integrated recovery instruction having corresponding integrated recovery information; based on the query conditions, reading multiple data to be checked from the source system in sequence; if any data to be checked is read, then based on the identification information of each data to be checked that has been read and a preset summary algorithm, a second read data summary is updated, and the second read quantity is updated, the second read data summary is used to identify the uniqueness of each data to be checked that has been read, and the second read quantity is used to identify the total number of data to be checked read from the source system; if the second read data summary is consistent with the first read data summary in the integrated recovery information, and the second read quantity is consistent with the first read quantity in the integrated recovery information, then each data to be checked that is read thereafter is integrated into the target system; if the second read quantity is greater than the first read quantity, the integrated recovery operation is interrupted.

[0090] Specifically, the method of data recovery in the embodiment of the present application is similar to the method of data recovery when the integrated recovery information does not contain the first read quantity, except that if any data to be checked is read, the second read quantity needs to be updated. Similar to the update of the first read quantity, the second read quantity is increased by 1 after each data to be checked is read. Therefore, if the updated second read quantity is consistent with the first read quantity in the integrated recovery information after a certain data to be checked is read, it means that this data to be checked may be the last data read before the integration interruption. However, whether this data to be checked is determined to be the last data read before the integration interruption, and whether this data to be checked and the data to be checked read before it have not changed between the last integration interruption and the current integrated recovery, it is necessary to further rely on the consistency between the second read data summary and the first read data summary to judge. Among them, when the last data read before the integration interruption is determined, and this data and the data read before it have not changed between the last integration interruption and the current integrated recovery, it means that each data to be checked read thereafter needs to perform a data integration operation.

[0091] That is to say, when and only when the second read data summary is consistent with the first read data summary, and the second read quantity is consistent with the first read quantity, it can be determined that the most recently read data to be checked is the last data read before the integration is interrupted. If the second read quantity is greater than the first read quantity, it means that after exceeding the number of data to be integrated read before the integration is interrupted, the first read data summary that is the same as the second read data summary has not been updated, which means that the data to be checked is not completely the same as the data to be integrated obtained in step 202. At this time, there is no possibility of integrated recovery, and the integrated recovery operation can be directly interrupted. Among them, in addition to the step of triggering the interruption of the integrated recovery operation by judging that the second read quantity is greater than the first read quantity, it can also be triggered by the second read quantity being equal to the first read quantity and the second integrated data summary being not equal to the first integrated data summary, which is not limited in the embodiment of the present application.

[0092] In the embodiment of the present application, the triggering of the interruption of the integrated recovery operation is realized through judgment. When faced with a very large number of data to be checked, it is not necessary to traverse all the data to be checked before determining that the data to be checked is not completely the same as the data to be integrated obtained in step 202. When the second reading number is greater than the first reading number, the integrated recovery operation can be interrupted, thereby improving the integrated recovery efficiency and avoiding a large number of "invalid" readings.

[0093] In another specific implementation method, for the technical effect that the first read data summary can identify the uniqueness of multiple read data, the first read data summary can be updated in the following manner: input the identification information of the first read data into a preset summary algorithm, and use the first data summary output by the preset summary algorithm as the first read data summary; input the identification information of the i-th read data into the preset summary algorithm, obtain the i-th data summary determined based on the i-1th first read data summary and the identification information of the i-th read data, and update the first read data summary to the i-th data summary.

[0094] Since the embodiment of the present application can adopt streaming to read the data to be checked and the data to be integrated, in order to maximize the efficiency of data integration and data recovery, the embodiment of the present application adopts an incremental digest algorithm that supports continuous acceptance of new inputs (i.e., identification information of the data that has been read) in the data stream and updates the hash value based on the new input and the original hash value. Specifically, the incremental digest algorithm can be implemented by adjusting the flow of conventional digest algorithms to meet the needs of incremental digests, such as incremental MD5, incremental SHA-256 and other digest algorithms obtained by improving conventional digest algorithms such as MD5 and SHA-256; or it can be a digest algorithm such as XXH3 and TTH (Tiger Tree Hash) that is designed for incremental calculation.

[0095] Specifically, in the initial state where no identification information is entered, there will be a default summary value in the preset summary algorithm. After any identification information of the read data is entered, the summary value in the preset summary algorithm will be updated, and this updated summary value will be used as the updated summary of the read data (such as the first read data summary or the second read data summary).

[0096] In addition, in application scenarios where computing power is sufficient and the requirements for integration and recovery efficiency are not high, summary algorithms other than the incremental summary algorithm can also be used. By recording the identification information of each integrated data and inputting the recorded identification information of each integrated data into the preset summary algorithm each time an update is required, an updated summary of the read data can be obtained, which can also achieve the technical effect required by the embodiments of the present application.

[0097] See also Figure 3 , the embodiment of the present application provides a data integration system, including:

[0098] An acquisition unit 301 is used to acquire a data integration instruction between a source system and a target system, where the data integration instruction includes a query condition;

[0099] A reading unit 302 is used to sequentially read multiple pieces of data to be integrated from a source system based on a query condition;

[0100] The reading unit 302 is further configured to integrate the data to be integrated into the target system if any data to be integrated is read, and determine the read data to be integrated as the read data, and update the first read data digest based on the identification information of each read data, where the first read data digest is used to identify the uniqueness of each read data;

[0101] The recovery unit 303 is configured to generate integration recovery information if data integration between the source system and the target system is interrupted, wherein the integration recovery information includes a query condition and a summary of the first read data.

[0102] In a specific implementation, the first read data digest is generated based on a preset digest algorithm, and after the integrated recovery information is generated, the data integration system further includes: a receiving unit;

[0103] A receiving unit, configured to receive an integrated recovery instruction corresponding to the data integration instruction, wherein the integrated recovery instruction has corresponding integrated recovery information;

[0104] The reading unit 302 is further used to sequentially read multiple pieces of to-be-queried data from the source system based on the query condition;

[0105] The recovery unit 303 is further configured to update a second read data digest based on the identification information of each read data to be checked and a preset digest algorithm if any data to be checked is read, wherein the second read data digest is used to identify the uniqueness of each read data to be checked;

[0106] The recovery unit 303 is further configured to integrate each piece of to-be-queried data subsequently read into the target system if the second read data digest is consistent with the first read data digest in the integrated recovery information.

[0107] In a specific implementation, the integrated recovery information also includes a first read quantity, and the reading unit 302 is also used to update the first read quantity if the step of determining the read data to be integrated as the read data is executed. The first read quantity is used to identify the total number of data to be integrated read from the source system.

[0108] In a specific implementation, the first read data digest is generated based on a preset digest algorithm, and after the integrated recovery information is generated, the data integration system further includes: a receiving unit;

[0109] A receiving unit, configured to receive an integrated recovery instruction corresponding to the data integration instruction, wherein the integrated recovery instruction has corresponding integrated recovery information;

[0110] The reading unit 302 is further used to sequentially read multiple pieces of to-be-queried data from the source system based on the query condition;

[0111] The reading unit 302 is further configured to update a second read data digest based on identification information of each read data to be checked and a preset digest algorithm, and update a second read quantity if any data to be checked is read, wherein the second read data digest is used to identify the uniqueness of each read data to be checked, and the second read quantity is used to identify the total quantity of data to be checked read from the source system;

[0112] The recovery unit 303 is further configured to integrate each piece of data to be checked that is subsequently read into the target system if the second read data summary is consistent with the first read data summary in the integrated recovery information, and the second read quantity is consistent with the first read quantity in the integrated recovery information.

[0113] The recovery unit 303 is further configured to interrupt the integrated recovery operation if the second read quantity is consistent with the first read quantity in the integrated recovery information and the second read data summary is inconsistent with the first read data summary in the integrated recovery information.

[0114] In a specific implementation, the reading unit 302 is specifically configured to input the identification information of the first read data into a preset digest algorithm, and use the first data digest output by the preset digest algorithm as the first read data digest;

[0115] The identification information of the i-th read data is input into a preset summary algorithm to obtain the i-th data summary determined based on the i-1-th first read data summary and the identification information of the i-th read data, and the first read data summary is updated to the i-th data summary.

[0116] In a specific implementation, the query condition includes a filtering condition and a sorting condition.

[0117] Figure 4 It is a schematic diagram of a computer device structure provided in an embodiment of the present application. The computer device 400 may include one or more central processing units (CPU) 401 and a memory 405. The memory 405 stores one or more application programs or data.

[0118] The memory 405 may be a volatile storage or a persistent storage. The program stored in the memory 405 may include one or more modules, each of which may include a series of instruction operations in the computer device. Furthermore, the central processor 401 may be configured to communicate with the memory 405 and execute a series of instruction operations in the memory 405 on the computer device 400.

[0119] The computer device 400 may also include one or more power supplies 402, one or more wired or wireless network interfaces 403, one or more input and output interfaces 404, and / or one or more operating systems, such as Windows Server 2000 or Windows Server 2003. TM , Mac OS X TM , Unix TM ,Linux TM , FreeBSD TM wait.

[0120] The CPU 401 can execute the aforementioned Figures 1 to 3 The operations performed by the data integration system in the illustrated embodiment will not be described in detail here.

[0121] It should be noted that, although the steps in the flowcharts involved in the embodiments are drawn in sequence according to the instructions of the arrows, unless otherwise clearly stated in this document, there is no strict order restriction on the execution of these steps, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the embodiments may include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0122] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0123] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0124] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0125] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0126] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, read-only memory), random access memory (RAM, random access memory), disk or optical disk and other media that can store program code.

[0127] An embodiment of the present application also provides a computer program product comprising instructions, and when the computer program product is run on a computer, the computer executes the data integration method as described above.

Claims

1. A data integration method, characterized in that: include: Obtaining a data integration instruction between a source system and a target system, wherein the data integration instruction includes a query condition; Based on the query condition, multiple pieces of data to be integrated are read from the source system in sequence; If any of the data to be integrated is read, the data to be integrated is integrated into the target system, and the read data to be integrated is determined as the read data, and based on the identification information of each piece of the read data, a first digest of the read data is updated, where the first digest of the read data is used to identify the uniqueness of each piece of the read data; If the data integration between the source system and the target system is interrupted, integration recovery information is generated, and the integration recovery information includes the query condition and the first read data summary.

2. The data integration method according to claim 1, characterized in that: The first read data digest is generated based on a preset digest algorithm. After generating the integrated recovery information, the method further includes: receiving an integrated recovery instruction corresponding to the data integration instruction, the integrated recovery instruction having the corresponding integrated recovery information; Based on the query condition, multiple pieces of data to be queried are read sequentially from the source system; If any of the to-be-checked data is read, then based on the identification information of each of the to-be-checked data that has been read and the preset summary algorithm, a second digest of the read data is updated, wherein the second digest of the read data is used to identify the uniqueness of each of the to-be-checked data that has been read; If the second read data digest is consistent with the first read data digest in the integrated recovery information, each piece of data to be checked that is subsequently read is integrated into the target system.

3. The data integration method according to claim 1, characterized in that: The integrated recovery information further includes a first read quantity, and the method further includes: If the step of determining the read data to be integrated as the read data is performed, the first read quantity is updated, and the first read quantity is used to identify the total quantity of the read data.

4. The data integration method according to claim 3, characterized in that: The first read data digest is generated based on a preset digest algorithm. After generating the integrated recovery information, the method further includes: receiving an integrated recovery instruction corresponding to the data integration instruction, the integrated recovery instruction having the corresponding integrated recovery information; Based on the query condition, multiple pieces of data to be queried are read sequentially from the source system; If any of the to-be-checked data is read, a second digest of the read data is updated based on the identification information of each of the to-be-checked data that has been read and the preset digest algorithm, and a second read quantity is updated, wherein the second digest of the read data is used to identify the uniqueness of each of the to-be-checked data that has been read, and the second read quantity is used to identify the total quantity of the to-be-checked data read from the source system; If the second read data digest is consistent with the first read data digest in the integrated recovery information, and the second read quantity is consistent with the first read quantity in the integrated recovery information, then each piece of data to be checked that is subsequently read is integrated into the target system; If the second read quantity is greater than the first read quantity, the integrated recovery operation is interrupted.

5. The data integration method according to any one of claims 1 to 4, characterized in that: The updating of the first read data summary based on the identification information of each piece of the read data comprises: Inputting the identification information of the first piece of read data into a preset digest algorithm, and using the first data digest output by the preset digest algorithm as the first read data digest; The identification information of the i-th read data is input into a preset summary algorithm to obtain the i-th data summary determined based on the i-1-th first read data summary and the identification information of the i-th read data, and the first read data summary is updated to the i-th data summary.

6. The method according to claim 1, characterized in that The query conditions include filtering conditions and sorting conditions.

7. A data integration system, characterized in that: include: An acquisition unit, used to acquire a data integration instruction between a source system and a target system, wherein the data integration instruction includes a query condition; A reading unit, configured to sequentially read a plurality of pieces of data to be integrated from the source system based on the query condition; The reading unit is further configured to integrate the data to be integrated into the target system if any of the data to be integrated is read, determine the read data to be integrated as read data, and update a first read data digest based on identification information of each piece of the read data, wherein the first read data digest is used to identify the uniqueness of each piece of the read data; A recovery unit is used to generate integration recovery information if the data integration between the source system and the target system is interrupted, wherein the integration recovery information includes the query condition and the first read data summary.

8. A computer device, characterized in that: include: CPU, memory and input / output interface; The memory is a transient storage memory or a persistent storage memory; The central processing unit is configured to communicate with the memory and execute instructions in the memory to perform the method according to any one of claims 1 to 6.

9. A computer program product comprising instructions, characterized in that When the computer program product is run on a computer, the computer is caused to execute the method according to any one of claims 1 to 6.

10. A computer storage medium, characterized in that: The computer storage medium stores instructions, and when the instructions are executed on a computer, the computer is caused to execute the method according to any one of claims 1 to 6.

Images