Data integration method and related apparatus

Abstract

Embodiments of the present application disclose a data integration method and related equipment, which are used for improving data integration efficiency. The method comprises: obtaining a data integration instruction between a source system and a target system, the data integration instruction comprising a query condition; sequentially reading a plurality of to-be-integrated data from the source system based on the query condition; if any of the to-be-integrated data is read, integrating the to-be-integrated data into the target system, determining the read to-be-integrated data as read data, and updating a first read data digest based on identification information of each of the read data, the first read data digest being used for identifying uniqueness of each of the read data; if the data integration between the source system and the target system is interrupted, generating integration recovery information, the integration recovery information comprising the query condition and the first read data digest.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to the field of data integration, and in particular, to a data integration method and related device. BACKGROUND

[0002] When doing system integration for large enterprise customers, large-scale data set integration is often involved, and one data integration may involve tens of millions of rows of data, with a data volume of several hundred GB or even more. In such large-scale data integration, errors may occur during execution, and at this time, the integration often needs to be paused or interrupted, and then resumed after the error factors are excluded.

[0003] In the prior art, full re-execution can be used. Generally, all the to-be-integrated data is obtained from the source system, and each piece of to-be-integrated data is integrated into the target system again. Moreover, since part of the to-be-integrated data has been integrated into the target system before the integration is interrupted, the detection of the data state of the target system is combined in the process of integrating each piece of to-be-integrated data into the target system again, and the data that needs to be integrated repeatedly is ignored.

[0004] In the case of a large amount of to-be-integrated data, the integration recovery scheme of full re-execution described above needs to reprocess each piece of to-be-integrated data processed before the interruption, which will cost a large amount of time and result in low efficiency of data integration from the source system to the target system. SUMMARY

[0005] Embodiments of the present application provide a data integration method and related device to improve the efficiency of data integration.

[0006] A first aspect of embodiments of the present application provides a data integration method, comprising:

[0007] obtaining a data integration instruction between a source system and a target system, the data integration instruction containing a query condition;

[0008] sequentially reading multiple pieces of to-be-integrated data from the source system based on the query condition;

[0009] if any of the to-be-integrated data is read, integrating the to-be-integrated data into the target system, determining the read to-be-integrated data as read data, and updating a first read data digest based on the identification information of each piece of the read data, the first read data digest being used to identify the uniqueness of each piece of the read data;

[0010] if the data integration between the source system and the target system is interrupted, generating integration recovery information, the integration recovery information including the query condition and the first read data digest.

[0011] In an implementation manner, the first read data digest is generated based on a preset digest algorithm, and after the integrated recovery information is generated, the method further comprises:

[0012] receiving an integrated recovery instruction corresponding to the data integration instruction, the integrated recovery instruction having the corresponding integrated recovery information;

[0013] sequentially reading a plurality of to-be-searched data from the source system based on the query condition;

[0014] If any of the to-be-searched data is read, updating a second read data digest based on the identification information of each read to-be-searched data and the preset digest algorithm, the second read data digest being used to identify the uniqueness of each read to-be-searched data;

[0015] If the second read data digest is consistent with the first read data digest in the integrated recovery information, integrating each to-be-searched data read thereafter into the target system.

[0016] In an implementation manner, the integrated recovery information further comprises a first read quantity, and the method further comprises:

[0017] If the step of determining the read to-be-integrated data as read data is executed, updating the first read quantity, the first read quantity being used to identify the total quantity of to-be-integrated data read from the source system.

[0018] In an implementation manner, the first read data digest is generated based on a preset digest algorithm, and after the integrated recovery information is generated, the method further comprises:

[0019] receiving an integrated recovery instruction corresponding to the data integration instruction, the integrated recovery instruction having the corresponding integrated recovery information;

[0020] sequentially reading a plurality of to-be-searched data from the source system based on the query condition;

[0021] If any of the to-be-searched data is read, updating a second read data digest based on the identification information of each read to-be-searched data and the preset digest algorithm, and updating a second read quantity, the second read data digest being used to identify the uniqueness of each read to-be-searched data, and the second read quantity being used to identify the total quantity of to-be-searched data read from the source system;

[0022] If the second read data digest is consistent with the first read data digest in the integrated recovery information, and the second read quantity is consistent with the first read quantity in the integrated recovery information, then each piece of to-be-searched data read later is integrated into the target system.

[0023] If the second read quantity is consistent with the first read quantity in the integrated recovery information, and the second read data digest is inconsistent with the first read data digest in the integrated recovery information, then the integrated recovery operation is interrupted.

[0024] In a specific implementation manner, the first read data digest is updated based on the identification information of each piece of the read data, including:

[0025] The identification information of the first piece of read data read is input into a preset digest algorithm, and the first data digest output by the preset digest algorithm is taken as the first read data digest;

[0026] The identification information of the i-th piece of read data read is input into a preset digest algorithm, an i-th data digest determined based on an (i-1)-th first read data digest and the identification information of the i-th piece of read data is obtained, and the first read data digest is updated to the i-th data digest.

[0027] In a specific implementation manner, the query condition includes a filtering condition and a sorting condition.

[0028] The second aspect of the embodiment of the application provides a data integration system, including:

[0029] An acquisition unit is configured to acquire a data integration instruction between a source system and a target system, the data integration instruction including a query condition;

[0030] A reading unit is configured to read a plurality of pieces of to-be-integrated data from the source system in sequence based on the query condition;

[0031] The reading unit is further configured to, if any piece of the to-be-integrated data is read, integrate the to-be-integrated data into the target system, determine the to-be-integrated data read as read data, and update a first read data digest based on the identification information of each piece of the read data, the first read data digest being used to identify the uniqueness of each piece of the read data;

[0032] A recovery unit is configured to, if the data integration between the source system and the target system is interrupted, generate integrated recovery information, the integrated recovery information including the query condition and the first read data digest.

[0033] In a specific implementation, the first read data digest is generated based on a preset digest algorithm, and after the integrated recovery information is generated, the data integration system further comprises a receiving unit;

[0034] The receiving unit is configured to receive an integrated recovery instruction corresponding to the data integration instruction, the integrated recovery instruction having the corresponding integrated recovery information;

[0035] The reading unit is further configured to sequentially read a plurality of to-be-searched data from the source system based on the search condition;

[0036] The recovery unit is further configured to, if any of the to-be-searched data is read, update a second read data digest based on the identification information of each read to-be-searched data and the preset digest algorithm, the second read data digest being used to identify the uniqueness of each read to-be-searched data;

[0037] The recovery unit is further configured to, if the second read data digest is consistent with the first read data digest in the integrated recovery information, integrate each to-be-searched data read thereafter into the target system.

[0038] In a specific implementation, the integrated recovery information further comprises a first read quantity, and the reading unit is further configured to, if the step of determining the read to-be-integrated data as read data is performed, update the first read quantity, the first read quantity being used to identify the total quantity of to-be-integrated data read from the source system.

[0039] In a specific implementation, the first read data digest is generated based on a preset digest algorithm, and after the integrated recovery information is generated, the data integration system further comprises a receiving unit;

[0040] The receiving unit is configured to receive an integrated recovery instruction corresponding to the data integration instruction, the integrated recovery instruction having the corresponding integrated recovery information;

[0041] The reading unit is further configured to sequentially read a plurality of to-be-searched data from the source system based on the search condition;

[0042] The reading unit is further configured to, if any of the to-be-searched data is read, update a second read data digest based on the identification information of each read to-be-searched data and the preset digest algorithm, and update a second read quantity, the second read data digest being used to identify the uniqueness of each read to-be-searched data, and the second read quantity being used to identify the total quantity of to-be-searched data read from the source system;

[0043] The recovery unit is further configured to integrate each piece of to-be-searched data read thereafter into the target system if the second read quantity is consistent with the first read quantity in the integrated recovery information and the second read data digest is consistent with the first read data digest in the integrated recovery information.

[0044] The recovery unit is further configured to interrupt the integrated recovery operation if the second read quantity is consistent with the first read quantity in the integrated recovery information and the second read data digest is inconsistent with the first read data digest in the integrated recovery information.

[0045] In a specific implementation, the reading unit is specifically configured to input the identification information of the first piece of read data into a preset digest algorithm, and take the first data digest output by the preset digest algorithm as the first read data digest.

[0046] The identification information of the i-th piece of read data is input into a preset digest algorithm, an i-th data digest determined based on the (i-1)-th first read data digest and the identification information of the i-th piece of read data is obtained, and the first read data digest is updated to the i-th data digest.

[0047] In a specific implementation, the query condition includes a filtering condition and a sorting condition.

[0048] The third aspect of the embodiments of the present application provides a data integration device, which comprises:

[0049] a central processing unit, a memory, and an input / output interface;

[0050] The memory is a volatile memory or a persistent memory.

[0051] The central processing unit is configured to communicate with the memory and perform instruction operations in the memory to execute the method of the first aspect.

[0052] The fourth aspect of the embodiments of the present application provides a computer program product containing instructions, which, when executed on a computer, causes the computer to execute the method of the first aspect.

[0053] The fifth aspect of the embodiments of the present application provides a computer storage medium, which stores instructions, and the instructions, when executed on a computer, cause the computer to execute the method of the first aspect.

[0054] From the above technical solution can be seen, the embodiments of the present application have the following advantages: in the embodiments of the present application, only by including the query condition and the integrated recovery information of the first read data summary, the integrated data in the plurality of to-be-integrated data and whether these data have been modified can be accurately determined. And in the whole integration recovery process, the to-be-integrated data integrated into the target system before the integration interruption need not be integrated repeatedly, which helps to improve the overall data integration efficiency. BRIEF DESCRIPTION OF DRAWINGS

[0055] Figure 1 A system architecture diagram of the data integration method disclosed by the embodiments of the present application;

[0056] Figure 2 A flowchart of the data integration method disclosed by the embodiments of the present application;

[0057] Figure 3 A structural diagram of the data integration system disclosed by the embodiments of the present application;

[0058] Figure 4 A structural diagram of the computer device disclosed by the embodiments of the present application. DETAILED DESCRIPTION

[0059] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0060] The embodiments of the present application provide a data integration method and related devices, for improving the data integration efficiency.

[0061] In order to better implement the data integration method of the embodiments of the present application, the present application provides a data integration architecture as shown in Figure 1 The data integration architecture includes a source system, a target system and a data integration system. The source system, the target system and the data integration system can be respectively deployed on the same, completely different or not completely different computer devices. The purpose of data integration is to integrate the to-be-integrated data in the source system into the target system.

[0062] It should be noted that the source system and the target system in the data integration refer to two different data management systems, which usually have different or not completely same fields. For example, business data is migrated from an old data management system to a new data management system. The source system and the target system in the data synchronization and the data migration usually refer to two same business systems or data management systems, and the data synchronization and the migration are generally to migrate data from a source end of a deployment A system to a target end of the deployment A system. As can be seen, in the actual data integration process, first, a plurality of pieces of to-be-integrated data are read from the source system, and then data conversion operations such as field mapping or field calculation are performed on each piece of the read to-be-integrated data, and finally the converted to-be-integrated data is integrated into the target system, which will be described in the following embodiments.

[0063] wherein, Figure 1 The data D1, D2, Di, Di+1, Di+2, Dn and the like shown in the figure are to-be-integrated data read from the source system, and the data in the solid line box in the reading process shown in the figure is the read data, and the data in the dashed line box is the to-be-integrated data. Figure 1 The data in the solid line box in the reading process shown in the figure is the read data, and the data in the dashed line box is the to-be-integrated data.

[0064] On the basis of the foregoing data integration architecture, the embodiments of the present application provide a data integration method as follows, comprising:

[0065] 201, obtaining a data integration instruction between the source system and the target system, the data integration instruction containing a query condition.

[0066] When the integration system receives the data integration instruction between the source system and the target system, it means that the integration system needs to obtain to-be-integrated data from the source system and store the obtained to-be-integrated data after conversion in the target system.

[0067] It can be understood that in order to complete the indication of the data integration instruction, the to-be-integrated data should be obtained from the source system first, and there are usually a large amount of business data in the source system. In the data integration scenario, all the business data in the source system is generally not to-be-integrated data that needs to be integrated into the target system. Therefore, in order to accurately obtain the to-be-integrated data in the business system that needs to be integrated into the target system, the query condition will be contained in the data integration instruction. Generally, the query condition includes at least one of a filtering condition and a sorting condition. The filtering condition is used to filter the to-be-integrated data in the business system that needs to be integrated into the target system, and the sorting condition is used to arrange the to-be-integrated data filtered from the business system according to a certain rule, such as arranging the to-be-integrated data in descending order according to the field value of a certain field of the to-be-integrated data. The rule used for sorting can be set by default or on demand, which is not limited here.

[0068] 202, based on the query condition, sequentially reading a plurality of to-be-integrated data from the source system.

[0069] Based on the foregoing embodiments, it is known that the query condition is used to accurately obtain the to-be-integrated data in the business system that needs to be integrated into the target system. Therefore, each piece of business data found in the source system that meets the query condition should be to-be-integrated data. Moreover, if the query condition does not explicitly limit the sorting condition of the obtained to-be-integrated data, considering that the to-be-integrated data is obtained from the database of the source system, therefore, the plurality of to-be-integrated data has a relative physical order in the database table, which can be used as the default sorting of the plurality of to-be-integrated data in the embodiments of the present application.

[0070] In actual application, in order to release the memory space of the data integration system, all to-be-integrated data meeting the query condition can not be read and integrated into the integration system at one time, but a streaming reading mode can be used to obtain part of the to-be-integrated data each time.

[0071] 203、If any to-be-integrated data is read, the to-be-integrated data is integrated into the target system, and the read to-be-integrated data is determined as read data, and the first read data digest is updated based on the identification information of each read data, and the first read data digest is used to identify the uniqueness of each read data.

[0072] In order to ensure accuracy, the data integration system usually needs to process the plurality of to-be-integrated data one by one. Therefore, for each to-be-integrated data read, in addition to performing an integration operation on it, the latest first read data digest needs to be generated based on the identification information of itself and the identification information of all read data read before it, and the latest first read data digest is used to replace the previous one. Or, after reading one to-be-integrated data, the identification information of the current time and each read data read before it (the to-be-integrated data becomes read data after being read) is obtained to update the first read data digest.

[0073] In order to ensure that the first read data digest can be used to accurately confirm the read data in the plurality of to-be-integrated data in the data integration recovery process, the updated first read data digest at least needs to identify the uniqueness of each read data.

[0074] The identification information includes the field values of the to-be-integrated data itself or part of the preset fields in the to-be-integrated data, such as at least one of the timestamp field and the data row hash value field, which is not limited here.

[0075] 204、If the data integration between the source system and the target system is interrupted, integration recovery information is generated, and the integration recovery information includes the query condition and the first read data digest.

[0076] Based on the foregoing embodiments, it can be known that the query condition determines the to-be-integrated data that can be obtained from the source system, and the first read data digest accurately identifies the uniqueness of the plurality of read data. Therefore, generating the integration recovery information including the query condition and the first read data digest helps the data integration system to obtain the plurality of to-be-searched data based on the query condition after analyzing the factors causing the integration interruption, and to determine the read data based on the first read data digest, thereby avoiding repeated integration of the same data.

[0077] It should be noted that, as known from the foregoing embodiments, in the complete data integration process, the data reading operation needs to be performed first, and then the data integration operation is performed on the read data. Similarly, in the integration recovery process, the data reading operation needs to be performed first, and then the data integration operation is performed on the read data. The difference lies in that, in order to avoid repeated execution of the data integration operation, the integration recovery information focusing on each read data is proposed in the embodiments of the present application. Through the query condition and the first read data digest in the integration recovery information, the read data in the previously interrupted integration can be found, and the integration operation is not performed again on these data, thereby avoiding the decrease of integration efficiency caused by repeated integration.

[0078] In actual application, there is also a data recovery strategy of local full cache. Specifically, first, the data set generation logic (for example, a SELECT statement of a database, a call of a data query API, etc.) is executed to obtain the source data. Second, the data is temporarily stored on the storage device of the server from the first row of data. Finally, when the data integration is paused due to an exception, the position of the processed data row is recorded, and when the execution is restarted next time, the data is read from the local storage device, and the data is read from the position recorded last time.

[0079] The data format of the integration recovery information can be a standard data format, such as a JSON string, an XML string, etc., or a private format; and the storage facility of the integration recovery information can be a database, a local disk file, a distributed file system, etc., which is not limited here.

[0080] Based on the foregoing, the local full cache needs to temporarily store the data set to be integrated on the storage device of the server, increasing the hardware cost of the server and occupying a large amount of additional I / O traffic. In a distributed environment, in order to ensure that each node in the cluster can access the temporarily stored data, an independent network storage facility needs to be used, which puts higher requirements on the bandwidth of the network. In the embodiment of the application, only the integration recovery information including the first read data digest and the query condition can accurately determine the read data in the plurality of data to be integrated and whether these data have been modified. Moreover, in the entire integration recovery process, the data to be integrated integrated into the target system before the interruption of integration does not need to be integrated repeatedly, which helps to improve the overall data integration efficiency.

[0081] On the basis of the foregoing embodiment, after the foregoing step 204, the application can specifically realize the recovery of data integration by simulating the data reading operation in data integration, and the details are described in the following steps: receiving an integration recovery instruction corresponding to the data integration instruction, the integration recovery instruction having corresponding integration recovery information; sequentially reading a plurality of data to be searched from the source system based on the query condition; if any data to be searched is read, updating the second read data digest based on the identification information of each read data to be searched and the preset digest algorithm, the second read data digest being used to identify the uniqueness of each read data to be searched; and if the second read data digest is consistent with the first read data digest in the integration recovery information, integrating each data to be searched read thereafter into the target system.

[0082] First, after receiving the integration recovery instruction for a data integration, the integration recovery information can be obtained from the integration recovery instruction. The integration recovery instruction for a data integration includes an integration recovery instruction corresponding to a data integration instruction.

[0083] Secondly, after obtaining the integration recovery information, a plurality of data to be searched can be obtained from the source system according to the query condition in the integration recovery information. It can be understood that if the data in the source system has not changed from the interruption of the last data integration to the reception of the integration recovery instruction, the plurality of data to be searched obtained from the source system based on the foregoing query condition should be completely consistent with the plurality of data to be integrated obtained in step 202.

[0084] Finally, similar to the operation of data integration. In the process of data recovery, the similar steps of data integration are repeated, the difference is that not all the read data needs to perform integration operation. Specifically, if any read data is read, the second read data digest is updated based on the identification information of each read data and the preset digest algorithm, and the second read data digest is used to identify the uniqueness of each read data. Wherein, the preset digest algorithm used to update the second read data digest and the preset digest algorithm used to update the first read data digest are the same, therefore, if the second read data digest obtained after reading a certain data is consistent with the first read data digest in the integrated recovery information, it means that this read data is the last read data before the integration is interrupted, that is, in the embodiment of the application, for each read data read after this read data, the integration operation needs to be started.

[0085] It should be noted that when the second read data digest is consistent with the first read data digest in the integrated recovery information, the read data after processing needs to be treated as integrated data for processing in the same way as the first data integration, and the specific processing method is described in the foregoing related embodiments, which will not be repeated here.

[0086] In addition, if the second read data digest after each update is different from the first read data digest until all the read data is traversed, it means that the plurality of read data obtained from the source system according to the query condition is different from the plurality of integrated data obtained in step 202, and the data recovery without duplication cannot be performed. A recovery termination notification can be sent to indicate that the foregoing data recovery instruction cannot be executed. However, after that, a new data integration instruction can be initiated to perform data integration instead of data recovery.

[0087] Further, if the integrated recovery information further includes the first read quantity, the embodiment of the application further includes: if the step of determining the read integrated data as read data is performed, the first read quantity is updated, wherein the first read quantity is used to identify the total number of read data.

[0088] Specifically, as the updating of the first read data digest, if any of the to-be-integrated data is read, the to-be-integrated data is determined as read data, and therefore, in addition to adding 1 to the first read number after each to-be-integrated data is read, the updating of the first read number can also be triggered by the execution of the step of "determining the read to-be-integrated data as read data". The first read number updating operation in the embodiment of the application can be configured as needed, and only the atomicity between the three transactions of the first read number updating, the first read data digest updating, and the determination of the read to-be-integrated data as read data needs to be ensured, that is, the three transactions must occur simultaneously or not simultaneously.

[0089] Similarly, when the first read number is also included in the integration recovery information, the embodiment of the application can complete the data integration of the to-be-searched data with higher recovery efficiency, and the implementation can specifically refer to the following manner: receiving an integration recovery instruction corresponding to the data integration instruction, the integration recovery instruction having corresponding integration recovery information; sequentially reading a plurality of to-be-searched data from the source system based on the query condition; if any to-be-searched data is read, updating the second read data digest based on the identification information of each read to-be-searched data and a preset digest algorithm, and updating the second read number, the second read data digest being used to identify the uniqueness of each read to-be-searched data, and the second read number being used to identify the total number of to-be-searched data read from the source system; if the second read data digest is consistent with the first read data digest in the integration recovery information, and the second read number is consistent with the first read number in the integration recovery information, then each to-be-searched data read thereafter is integrated into the target system; if the second read number is greater than the first read number, then the integration recovery operation is interrupted.

[0090] Specifically, the data recovery manner in the embodiment of the application is similar to the data recovery manner when the first read number is not included in the integration recovery information, and the difference is that the second read number also needs to be updated if any to-be-searched data is read. Similar to the updating of the first read number, the second read number is added by 1 after each to-be-searched data is read, and therefore, if the second read number obtained after a to-be-searched data is read is consistent with the first read number in the integration recovery information, it indicates that the to-be-searched data is likely to be the last read data before the integration interruption, but whether the to-be-searched data is determined as the last read data before the integration interruption and whether the to-be-searched data and the to-be-searched data read before it have not changed between the last integration interruption and the current integration recovery also need to be further determined by the consistency between the second read data digest and the first read data digest. When the last read data before the integration interruption and the to-be-searched data and the to-be-searched data read before it have not changed between the last integration interruption and the current integration recovery are determined, it indicates that each to-be-searched data read thereafter needs to perform a data integration operation.

[0091] That is, the last read data can be determined to be the last read data before the integration interruption only when the second read data digest is consistent with the first read data digest and the second read quantity is consistent with the first read quantity. If the second read quantity is greater than the first read quantity, it indicates that the number of read data to be integrated before the integration interruption has been exceeded, and the first read data digest corresponding to the second read data digest has not been updated, which indicates that the data to be searched is not completely the same as the data to be integrated obtained in step 202, and there is no possibility of integration recovery, and the integration recovery operation can be directly interrupted. In addition to the step of interrupting the integration recovery operation triggered by the second read quantity being greater than the first read quantity, the second read quantity can be equal to the first read quantity, and the second integrated data digest can be not equal to the first integrated data digest, which is not limited in the embodiment of the application.

[0092] In the embodiment of the application, the trigger of interrupting the integration recovery operation is realized by the judgment. When facing a large number of data to be searched, it is not necessary to traverse all the data to be searched to determine that the data to be searched is not completely the same as the data to be integrated obtained in step 202, and the integration recovery operation can be interrupted when the second read quantity is greater than the first read quantity, which improves the integration recovery efficiency and avoids a large number of "invalid" readings.

[0093] In another specific implementation mode, for the technical effect that the first read data digest can identify the uniqueness of a plurality of read data, the first read data digest can be updated in the following manner: the identification information of the first read data is input into a preset digest algorithm, and the first data digest output by the preset digest algorithm is taken as the first read data digest; the identification information of the i th read data is input into the preset digest algorithm, an i th data digest determined based on the i-1 th first read data digest and the identification information of the i th read data is obtained, and the first read data digest is updated to the i th data digest.

[0094] Since the embodiments of the present application can adopt stream reading of the data to be searched and the data to be integrated, in order to maximize the data integration efficiency and the data recovery efficiency, the embodiments of the present application adopt an incremental digest algorithm which supports continuously accepting new inputs (i.e. the identification information of the read data) in the data stream and updating the hash value based on the new inputs and the original hash value. Specifically, the incremental digest algorithm can be realized by adjusting the process of the conventional digest algorithm to meet the needs of incremental digest, such as the incremental MD5, the incremental SHA-256 and other digest algorithms obtained by improving the conventional digest algorithms such as MD5 and SHA-256; or can be a digest algorithm such as XXH3 and TTH (Tiger Tree Hash) which is designed for incremental calculation.

[0095] Specifically, in the initial state without inputting any identification information, there will be a default digest value in the preset digest algorithm, and after inputting the identification information of any read data, the digest value in the preset digest algorithm will be updated, and the updated digest value will be used as the updated read data digest (such as the first read data digest or the second read data digest).

[0096] In addition, in the application scenario where the computing power is sufficient and the integration and recovery efficiency is not high, a digest algorithm other than the incremental digest algorithm can also be used, by recording the identification information of each integrated data and inputting the recorded identification information of each integrated data into the preset digest algorithm each time it needs to be updated, to obtain the updated read data digest, which can also achieve the technical effects required by the embodiments of the present application.

[0097] Please refer to Figure 3 The embodiments of the present application provide a data integration system, comprising:

[0098] The acquisition unit 301 is configured to acquire a data integration instruction between the source system and the target system, and the data integration instruction comprises a query condition;

[0099] The reading unit 302 is configured to read a plurality of data to be integrated from the source system in sequence based on the query condition;

[0100] The reading unit 302 is further configured to integrate the data to be integrated into the target system if any data to be integrated is read, and determine the read data to be integrated as read data, and update the first read data digest based on the identification information of each read data, and the first read data digest is used to identify the uniqueness of each read data;

[0101] The recovery unit 303 is configured to generate integration recovery information if the data integration between the source system and the target system is interrupted, and the integration recovery information comprises the query condition and the first read data digest.

[0102] In a specific implementation, the first read data digest is generated based on a preset digest algorithm, and after the integrated recovery information is generated, the data integration system further comprises a receiving unit;

[0103] The receiving unit is configured to receive an integrated recovery instruction corresponding to the data integration instruction, the integrated recovery instruction having corresponding integrated recovery information.

[0104] The reading unit 302 is further configured to sequentially read a plurality of to-be-searched data from the source system based on a search condition.

[0105] The recovery unit 303 is further configured to, if any to-be-searched data is read, update a second read data digest based on the identification information of each read to-be-searched data and a preset digest algorithm, the second read data digest being used to identify the uniqueness of each read to-be-searched data.

[0106] The recovery unit 303 is further configured to, if the second read data digest is consistent with the first read data digest in the integrated recovery information, integrate each to-be-searched data read thereafter into the target system.

[0107] In a specific implementation, the integrated recovery information further comprises a first read quantity, and the reading unit 302 is further configured to, if the step of determining the read to-be-integrated data as read data is performed, update the first read quantity, the first read quantity being used to identify the total quantity of to-be-integrated data read from the source system.

[0108] In a specific implementation, the first read data digest is generated based on a preset digest algorithm, and after the integrated recovery information is generated, the data integration system further comprises a receiving unit;

[0109] The receiving unit is configured to receive an integrated recovery instruction corresponding to the data integration instruction, the integrated recovery instruction having corresponding integrated recovery information.

[0110] The reading unit 302 is further configured to sequentially read a plurality of to-be-searched data from the source system based on a search condition.

[0111] The reading unit 302 is further configured to, if any to-be-searched data is read, update a second read data digest based on the identification information of each read to-be-searched data and a preset digest algorithm, and update a second read quantity, the second read data digest being used to identify the uniqueness of each read to-be-searched data, and the second read quantity being used to identify the total quantity of to-be-searched data read from the source system.

[0112] The recovery unit 303 is further configured to integrate each piece of data to be checked subsequently into the target system if the second read data digest is consistent with the first read data digest in the integrated recovery information and the second read quantity is consistent with the first read quantity in the integrated recovery information.

[0113] The recovery unit 303 is further configured to interrupt the integrated recovery operation if the second read count is consistent with the first read count in the integrated recovery information and the second read data digest is inconsistent with the first read data digest in the integrated recovery information.

[0114] In one specific implementation, the reading unit 302 is specifically used to input the identification information of the first read data into a preset digest algorithm, and to use the first data digest output by the preset digest algorithm as the first read data digest;

[0115] Input the identification information of the i-th read data into the preset digest algorithm to obtain the i-th data digest determined based on the (i-1)-th first read data digest and the identification information of the i-th read data, and update the first read data digest to the i-th data digest.

[0116] In one specific implementation, the query conditions include filtering conditions and sorting conditions.

[0117] Figure 4 This is a schematic diagram of a computer device structure provided in an embodiment of this application. The computer device 400 may include one or more central processing units (CPUs) 401 and a memory 405, in which one or more application programs or data are stored.

[0118] The memory 405 can be volatile or persistent storage. The program stored in the memory 405 can include one or more modules, each module including a series of instruction operations on the computer device. Furthermore, the central processing unit 401 can be configured to communicate with the memory 405 and execute the series of instruction operations stored in the memory 405 on the computer device 400.

[0119] Computer device 400 may also include one or more power supplies 402, one or more wired or wireless network interfaces 403, one or more input / output interfaces 404, and / or one or more operating systems, such as Windows Server. TM Mac OS X TM Unix TM Linux TM FreeBSD TM wait.

[0120] The central processor 401 can execute the operations performed by the data integration system in the embodiments described above, which will not be repeated here. Figures 1 to 3 The central processor 401 can execute the operations performed by the data integration system in the embodiments described above, which will not be repeated here.

[0121] It should be noted that although the steps in the flowcharts involved in the embodiments are drawn in sequence according to the arrows, unless otherwise stated herein, the execution of these steps is not strictly limited in sequence, and these steps can be executed in other sequences. Moreover, at least part of the steps in the flowcharts involved in the embodiments can include multiple steps or stages, which do not necessarily be executed at the same time, but can be executed at different times, and the execution sequence of these steps or stages is not necessarily sequential, but can be executed alternately or alternately with at least part of other steps or steps or stages in other steps.

[0122] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the system, device and unit described above can refer to the corresponding process in the foregoing method embodiments, which will not be repeated here.

[0123] In several embodiments provided in the present application, it should be understood that the disclosed system, device and method can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of the units is only a logical function division, and actual implementation can have another division manner, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units shown or discussed can be indirect coupling or communication connection through some interface, device or unit, which can be electrical, mechanical or other forms.

[0124] The units described as separate components can or can not be physically separated, and the components shown as units can or can not be physical units, that is, they can be located in one place, or can be distributed on a plurality of network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment scheme.

[0125] In addition, each functional unit in each embodiment of the present application can be integrated in one processing unit, or each unit can exist physically, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of software functional unit.

[0126] The integrated unit, if implemented in the form of a software function unit and sold or used as an independent product, can be stored in a computer readable storage medium. Based on such an understanding, the technical solutions of the present application, essentially or in other words, the part that contributes to the prior art or the whole or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, and includes a number of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the method described in various embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, read-only memory), a random access memory (RAM, random access memory), a magnetic disk or an optical disk, and various media that can store program codes.

[0127] The embodiments of the present application also provide a computer program product containing instructions, which, when the computer program product is run on a computer, enable the computer to execute the data integration method as described above.

Claims

1. A data integration method, characterized by, The method comprises the following steps: acquiring data integration instructions between a source system and a target system, wherein the data integration instructions comprise query conditions; sequentially reading a plurality of to-be-integrated data from the source system based on the query conditions; if any of the to-be-integrated data is read, integrating the to-be-integrated data into the target system, determining the read to-be-integrated data as read data, updating a first read quantity and a first read data digest based on identification information of each of the read data, wherein the first read quantity is used to identify a total quantity of read data, and the first read data digest is used to identify uniqueness of each of the read data; the first read data digest is generated based on a preset digest algorithm; if the data integration between the source system and the target system is interrupted, generating integration recovery information, wherein the integration recovery information comprises the query conditions, the first read quantity and the first read data digest; receiving an integration recovery instruction corresponding to the data integration instructions, wherein the integration recovery instruction has corresponding integration recovery information; sequentially reading a plurality of to-be-searched data from the source system based on the query conditions; if any of the to-be-searched data is read, updating a second read data digest based on identification information of each of the read to-be-searched data and the preset digest algorithm, and updating a second read quantity, wherein the second read data digest is used to identify uniqueness of each of the read to-be-searched data, and the second read quantity is used to identify a total quantity of to-be-searched data read from the source system; if the second read data digest is consistent with the first read data digest in the integration recovery information, and the second read quantity is consistent with the first read quantity in the integration recovery information, integrating each of the to-be-searched data read thereafter into the target system; if the second read quantity is greater than the first read quantity, interrupting the integration recovery operation.

2. The data integration method of claim 1, wherein, The method further comprises the following steps: inputting identification information of a first read data into the preset digest algorithm, and taking a first data digest output by the preset digest algorithm as the first read data digest; inputting identification information of an i-th read data into the preset digest algorithm, obtaining an i-th data digest determined based on an (i-1)-th first read data digest and the identification information of the i-th read data, and updating the first read data digest to the i-th data digest.

3. The method of claim 1, wherein, The query conditions comprise filtering conditions and sorting conditions.

4. A data integration system, characterized by The method comprises the following steps: an acquiring unit, configured to acquire data integration instructions between a source system and a target system, wherein the data integration instructions comprise query conditions; a reading unit, configured to sequentially read a plurality of to-be-integrated data from the source system based on the query conditions; The reading unit is further configured to integrate the to-be-integrated data into the target system if any to-be-integrated data is read, determine the read to-be-integrated data as read data, update a first read quantity and a first read data digest based on the identification information of each read data, wherein the first read quantity is used to identify the total number of read data, and the first read data digest is used to identify the uniqueness of each read data; and the first read data digest is generated based on a preset digest algorithm. The recovery unit is configured to generate integration recovery information if the data integration between the source system and the target system is interrupted, the integration recovery information including the query condition and the first read data digest; receive an integration recovery instruction corresponding to the data integration instruction, the integration recovery instruction having corresponding integration recovery information; sequentially read a plurality of to-be-searched data from the source system based on the query condition; update a second read data digest and a second read quantity based on the identification information of each read to-be-searched data and the preset digest algorithm if any to-be-searched data is read, the second read data digest being used to identify the uniqueness of each read to-be-searched data, and the second read quantity being used to identify the total number of to-be-searched data read from the source system; integrate each to-be-searched data read thereafter into the target system if the second read data digest is consistent with the first read data digest in the integration recovery information, and the second read quantity is consistent with the first read quantity in the integration recovery information; and interrupt the integration recovery operation if the second read quantity is greater than the first read quantity. The computer program product comprises:

5. A computer device, comprising: a central processing unit, a memory and an input / output interface; the memory is a volatile memory or a persistent storage memory; the central processing unit is configured to communicate with the memory and execute instruction operations in the memory to perform the method of any one of claims 1 to 3. The computer program product, when running on a computer, causes the computer to perform the method of any one of claims 1 to 3.

6. A computer program product comprising instructions, characterized in that, The computer storage medium stores instructions, and the instructions, when executed on a computer, cause the computer to perform the method of any one of claims 1 to 3.

7. A computer storage medium, characterized in that ​

Images