Data processing method, computer device, storage medium, and computer program product

Abstract

The present application relates to a data processing method, a computer device, a storage medium, and a computer program product. The method belongs to the field of big data and can be used for data synchronization and verification of financial business data. The method includes: obtaining a source data record set and a target data record set; the target data record set is obtained by performing data synchronization based on the source data record set; performing data consistency verification based on the source modification times corresponding to the source data records and the target modification times corresponding to the target data records to obtain a consistency verification result; obtaining record indication information corresponding to the differential data records with differences between the source data record set and the target data record set based on the consistency verification result; and performing data synchronization on the target data record set based on the record indication information to obtain a synchronized target data record set. Using this method can improve the efficiency of data verification.

Description

Technical Field

[0001] The present application relates to the field of big data technology, and in particular, to a data processing method, a computer device, a storage medium, and a computer program product. Background Art

[0002] With the development of Internet technology, there is more and more data on the Internet. Therefore, there is often a need to synchronize data stored in different storage spaces. For example, with the continuous development of enterprise digitization and intelligence, the amount of data stored by enterprises for different applications or product lines is increasing at a geometric rate. Due to the limitation of physical storage media, these data will lead to the emergence of data islands. To solve this problem, enterprises can build a data lake and synchronize enterprise data to the data lake to achieve data sharing.

[0003] In traditional technologies, after synchronizing the data at the source data end to the target storage end, the data at the source data end and the data at the target storage end are inconsistent. Therefore, it is necessary to perform data consistency verification to find the data with differences. However, currently, the verification is mainly performed by comparing each pair of data records one by one, and the verification efficiency is relatively low. Summary of the Invention

[0004] Based on this, it is necessary to provide a data processing method, apparatus, computer device, computer-readable storage medium, and computer program product for the above technical problems.

[0005] In a first aspect, the present application provides a data processing method. The method includes: obtaining a source data record set and a target data record set; the target data record set is obtained by synchronizing data based on the source data record set; the source data records in the source data record set correspond to source modification times, and the target data records in the target data record set correspond to target modification times; performing data consistency verification based on the source modification times corresponding to the source data records and the target modification times corresponding to the target data records to obtain a consistency verification result; obtaining record indication information corresponding to the different data records with differences between the source data record set and the target data record set based on the consistency verification result; and synchronizing the target data record set based on the record indication information to obtain a synchronized target data record set.

[0006] In a second aspect, the present application further provides a data processing device. The device includes: a record set acquisition module, configured to acquire a source data record set and a target data record set; the target data record set is obtained by performing data synchronization based on the source data record set; source data records in the source data record set correspond to source modification times, and target data records in the target data record set correspond to target modification times; a consistency verification module, configured to perform data consistency verification based on the source modification times corresponding to the source data records and the target modification times corresponding to the target data records, to obtain a consistency verification result; a record indication information obtaining module, configured to obtain record indication information corresponding to differential data records with differences between the source data record set and the target data record set based on the consistency verification result; a first synchronization module, configured to perform data synchronization on the target data record set based on the record indication information, to obtain a synchronized target data record set.

[0007] In one embodiment, source data records in the source data record set correspond to source data modification times, and target data records in the target data record set correspond to target data modification times; the consistency verification module is configured to: obtain a first modification time range; obtain a first source record quantity corresponding to the source data record set, where the first source record quantity is the quantity of source data records in the source data record set whose source data modification times are within the first modification time range and whose source modification times are a first value; the first value is the modification time corresponding to a newly added data record; obtain a first target record quantity corresponding to the target data record set, where the first target record quantity is the quantity of target data records in the target data record set whose target data modification times are within the first modification time range and whose target modification times are the first value; perform data consistency verification based on the first source record quantity and the first target record quantity, to obtain a consistency verification result.

[0008] In one embodiment, the device further includes: a division module, configured to divide the first modification time range to obtain a plurality of sub-time ranges when it is determined that the first source record quantity is inconsistent with the first target record quantity; a continued verification module, configured to use each of the sub-time ranges as an updated first modification time range, to continue performing data consistency verification based on the updated first modification time range until a verification stop condition is met; a time range obtaining module, configured to use the first modification time range in which the first source record quantity is inconsistent with the first target record quantity when the verification stops as the time range where the differential data records with differences are located; the verification stop condition includes at least one of the length of the updated first modification time range being less than a length threshold or the number of divisions exceeding a number threshold.

[0009] In one embodiment, the consistency verification module is configured to: when the source modification times corresponding to the same target record index are inconsistent with the target modification times, add the target record index to the data difference index set; and use the data records corresponding to the target data record indexes in the data difference index set as the differential data records with differences.

[0010] In one embodiment, the apparatus further includes: an operation log record sequence acquisition module, configured to acquire the operation log record sequence corresponding to the source data record set; an operation type identifier addition module, configured to add corresponding operation type identifiers to each operation log record in the operation log record sequence based on the modification operation type corresponding to the operation log record sequence; a second synchronization module, configured to perform data synchronization based on the operation log record sequence to obtain the target data record set; and the step of adding the target record index to the data difference index set when the source modification times corresponding to the same target record index are inconsistent with the target modification times includes: determining the data record index corresponding to the operation log record whose corresponding operation type identifier is the edit type identifier as the target record index; and when the source modification times corresponding to the target record index are inconsistent with the target modification times, adding the target record index to the data difference index set.

[0011] In one embodiment, the target data record set is obtained by synchronizing based on the operation log sequence corresponding to the source data record set, and the apparatus further includes: a target deleted record quantity acquisition module, configured to acquire the quantity of target deleted records corresponding to the target data record set, where the quantity of target deleted records is the quantity of data records deleted based on the delete operation log within the second modification time range for the target data record set; a source deleted record quantity acquisition module, configured to acquire the quantity of source deleted records corresponding to the source data record set, where the quantity of source deleted records is the quantity of data records that have been deleted within the second modification time range for the source data record set; and a differential data record determination module, configured to determine that the data consistency verification result corresponding to the second modification time range has differential data records when the quantity of target deleted records is inconsistent with the quantity of source deleted records.

[0012] In a third aspect, the present application further provides a computer device. The computer device includes a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, the following steps are implemented: obtaining a source data record set and a target data record set; the target data record set is obtained by performing data synchronization based on the source data record set; the source data records in the source data record set correspond to source modification times, and the target data records in the target data record set correspond to target modification times; performing data consistency verification based on the source modification times corresponding to the source data records and the target modification times corresponding to the target data records to obtain a consistency verification result; obtaining record indication information corresponding to the differential data records with differences between the source data record set and the target data record set based on the consistency verification result; and performing data synchronization on the target data record set based on the record indication information to obtain a synchronized target data record set.

[0013] In a fourth aspect, the present application further provides a computer-readable storage medium. On the computer-readable storage medium, a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented: obtaining a source data record set and a target data record set; the target data record set is obtained by performing data synchronization based on the source data record set; the source data records in the source data record set correspond to source modification times, and the target data records in the target data record set correspond to target modification times; performing data consistency verification based on the source modification times corresponding to the source data records and the target modification times corresponding to the target data records to obtain a consistency verification result; obtaining record indication information corresponding to the differential data records with differences between the source data record set and the target data record set based on the consistency verification result; and performing data synchronization on the target data record set based on the record indication information to obtain a synchronized target data record set.

[0014] In a fifth aspect, the present application also provides a computer program product. The computer program product includes a computer program which, when executed by a processor, implements the following steps: obtaining a source data record set and a target data record set; the target data record set is obtained by data synchronization based on the source data record set; the source data records in the source data record set correspond to source modification times, and the target data records in the target data record set correspond to target modification times; performing data consistency verification based on the source modification times corresponding to the source data records and the target modification times corresponding to the target data records to obtain a consistency verification result; obtaining record indication information corresponding to the differential data records with differences between the source data record set and the target data record set based on the consistency verification result; and performing data synchronization on the target data record set based on the record indication information to obtain a synchronized target data record set.

[0015] In the above data processing method, apparatus, computer device, storage medium, and computer program product, since modification times are corresponding in the data records, it can be reflected whether the modification times of the data records in the target data record set are consistent with the modification times of the corresponding data records in the source data record set. Therefore, during data verification, data consistency verification is performed based on the source modification times corresponding to the source data records and the target modification times corresponding to the target data records, and record indication information of the differential data records with differences between the two sets can be obtained, thereby improving the efficiency of data verification and enabling data synchronization to be performed on the target data record set based on the record indication information, ensuring data consistency between the two sets. Description of the Drawings

[0016] Figure 1 It is an application environment diagram of the data processing method in an embodiment;

[0017] Figure 2 It is a flowchart of the data processing method in an embodiment;

[0018] Figure 3 It is a flowchart of the step of performing data consistency verification based on the source modification times corresponding to the source data records and the target modification times corresponding to the target data records to obtain a consistency verification result in an embodiment;

[0019] Figure 4 It is a flowchart of the data processing method in an embodiment;

[0020] Figure 5 It is a structural block diagram of the data processing apparatus in an embodiment;

[0021] Figure 6 It is an internal structure diagram of the computer device in an embodiment. Detailed implementation manners

[0022] In order to make the objectives, technical solutions and advantages of the present application clearer and more understandable, the present application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0023] The solution disclosed in the present application can be used for data synchronization of financial service data in a data lake in the financial field, such as synchronizing indicator data in financial service data. It can also be used in any field other than the financial field. For example, it can be used to synchronize interactive service data. The application field of the solution disclosed in the present application is not limited.

[0024] The data processing method provided in the embodiments of the present application can be applied to, for example, Figure 1 the application environment shown in the figure. Among them, the server 104 synchronizes the source data records of the source data terminal 102 to the target data terminal 106 through the network. After the data synchronization, the server can perform data verification on the data of the source data terminal 102 and the target data terminal 106 to determine the consistency between the set of source data records stored in the source data terminal 102 and the set of target data records stored in the target data terminal 106. If it is determined that the set of source data records stored in the source data terminal 102 is inconsistent with the set of target data records stored in the target data terminal 106, the server 104 synchronizes the inconsistent data records. Among them, the source data terminal and the target data terminal can be, but are not limited to, servers, various personal computers, laptop computers, smart phones, tablet computers, Internet of Things devices, and portable wearable devices. The Internet of Things devices can be smart speakers, smart TVs, smart air conditioners, smart vehicle-mounted devices, etc. The portable wearable devices can be smart watches, smart bracelets, head-mounted devices, etc. The server 104 can be implemented by an independent server or a server cluster composed of multiple servers. For example, when performing synchronization, the server in the synchronization platform can be used to execute the synchronization steps, and when performing data verification, the server in the big data processing service platform can be used to execute the synchronization steps.

[0025] In one embodiment, as Figure 2 shown in the figure, a data processing method is provided. Taking the method applied to the Figure 1 server 104 in the figure as an example, the method includes the following steps:

[0026] Step S202, obtain a set of source data records and a set of target data records; the set of target data records is obtained by performing data synchronization based on the set of source data records; the source data records in the set of source data records correspond to source modification times, and the target data records in the set of target data records correspond to target modification times.

[0027] Among them, the data record set may include one or more data records. A data record is a set of complete information, such as a row of information in a table. For example, a data table may include information of multiple customers, and the information of each customer is a data record. For example, assuming that all information about a certain customer corresponds to a row of the data table, then one row represents a data record. The source data record set is the source of the synchronized data, and the target data record set is the set of data records after synchronization. For example, when synchronizing the data in Table A to Table B, all data records in Table A form the source data record set. All data records in Table B form the target data record set.

[0028] The source modification count refers to the number of times the source data record is modified, and the target modification count refers to the number of times the target data record is modified. Fields corresponding to the modification counts can be added to the data tables to be synchronized, and the values corresponding to the fields of the data records are used to represent the modification counts corresponding to the data records. For example, a field SEQ_NO can be added to each table at the source data end and each table at the target data end. The value corresponding to this field is used to identify the modification count of each record in the table, and the value is incremented by 1 each time it is modified. For example: Assuming a data record is a row, the modification count corresponding to the data record can be determined according to the value corresponding to the field SEQ_NO in the row where the data record is located. Among them, SEQ_NO = 1 can indicate that this record is a newly added record, and the modification count is 1; SEQ_NO > 1 indicates that this record is an edited record. It can be understood that it can also be that SEQ_NO = 0 indicates that this record is a newly added record, and SEQ_NO > 0 indicates that this record is an edited record. New addition means that a data record comes into being from nothing, editing means updating all or part of the data in the data record, and deletion means deleting the data record from the set.

[0029] In the embodiment of the present application, after synchronizing the data in the source data record set to the target data record set, the server may start to execute step S202 to determine the consistency of the synchronized data.

[0030] In one embodiment, the source data record set and the target data record set may be stored in the same device or in different devices. For example, the source data record set is stored in the server corresponding to each application, while the target data record set is stored in the data lake. A data lake is a centralized data storage repository. To solve the problem of data silos, data sharing can be achieved through the data lake. The data lake can not only store any type of data, but also control data access permissions and perform processing and analysis on the data lake. Each application can apply for access permissions to the data tables of other applications, and process the data to obtain data copies that meet specific internal schema formats.

[0031] Step S204: Based on the source modification times corresponding to the source data records and the target modification times corresponding to the target data records, perform data consistency verification to obtain a consistency verification result.

[0032] In the embodiments of the present application, data consistency verification is to confirm whether the data records in the source data record set are consistent with the data records in the target data record set. The consistency verification result can be consistent or inconsistent.

[0033] In one embodiment, when performing consistency verification, the source modification times and target modification times of the data records corresponding to the same data record index can be compared. If it is determined that the source modification time is inconsistent with the target modification time, it means that the modification times of the data records corresponding to the same data record index at the source data end and the target data end are inconsistent.

[0034] In one embodiment, when performing consistency verification, it can be determined whether the number of data records with the same modification times within the same last modification time period in the source data record set and the target data record set is consistent. If it is inconsistent, it means that the data records corresponding to this modification time period are inconsistent. For example, assume that the number of data records with the last modification time between 9:00 and 10:00 on a certain day and the modification times of 2 times in the source data record set is 20, and the number of data records with the last modification time between 9:00 and 10:00 on a certain day and the modification times of 2 times in the target data record set is 19. Then it means that among the data records with the modification time between 9:00 and 10:00 on a certain day and the modification times of 2 times in the source data record set, at least one data record is inconsistent with the data records in the target data record set.

[0035] In one embodiment, step S204 can be executed for the data records of a specific modification type, and for the data records of other modification types, other methods can be used for data consistency verification. For example, for the data records of the new type and the edit type, step S204 is executed. For the data records of the delete type, it can be determined whether the deleted data records within the same time period are consistent based on whether the number of deleted data records corresponding to the source data record set is consistent with the number of deleted data records corresponding to the target data record set within the same modification time period. The data records of the new type refer to the data records whose last operation is new, the data records of the edit type refer to the data records whose last operation is edit, and the data records of the delete type refer to the data records whose last operation is delete.

[0036] Step S206: Based on the consistency verification result, obtain the record indication information corresponding to the differential data records with differences between the source data record set and the target data record set.

[0037] The record indication information is used to indicate the range of data records with differences. It can be the index of the data record, or can include at least one of the time range and the number of modifications. For example, the record indication information can be: in the set of source data records, there are inconsistent problems with the data records whose number of modifications is 3 and the modification time range is from 9:00 to 10:00 on a certain day. It can be understood that all or some of the data records within the range pointed to by the record indication information may have inconsistent problems. For example, it may be that in the set of source data records, there is an inconsistent problem with a certain data record whose number of modifications is 3 and the modification time range is from 9:00 to 10:00 on a certain day.

[0038] Step S208: Based on the record indication information, perform data synchronization on the target data record set to obtain the synchronized target data record set.

[0039] In the embodiments of the present application, data synchronization can be performed on the data records corresponding to the record indication information, and all the data records pointed to by the record indication information in the source data record set are synchronized to ensure the consistency of the data records in the target data record set and the source data record set. For example, the record indication information can be: in the set of source data records, there are inconsistent problems with the data records whose number of modifications is 3 and the modification time range is from 9:00 to 10:00 on a certain day, and there are 20 data records corresponding to this record indication information. Then, even if only some of the data records pointed to by the record indication information may have inconsistent problems, for example, there is 1 inconsistent problem, these 20 data records can still be synchronized, so that it is not necessary to compare each data record between the two sets within this time range one by one. Of course, it is also possible to compare each data record corresponding to the record indication information to determine which specific data record has an inconsistent problem and only synchronize the inconsistent data records.

[0040] In the above data processing method, since there is a corresponding number of modifications in the data record, which can reflect whether the number of modifications of the data records in the target data record set is consistent with the number of modifications of the corresponding data records in the source data record set, when performing data verification, based on the source modification times corresponding to the source data records and the target modification times corresponding to the target data records, data consistency verification is performed, and the record indication information of the differential data records with differences between the two sets can be obtained, thereby improving the efficiency of data verification, and data synchronization can be performed on the target data record set based on the record indication information to ensure the consistency of the data between the two sets.

[0041] In one embodiment, the source data records in the source data record set correspond to source data modification times, and the target data records in the target data record set correspond to target data modification times. As Figure 3 shown, data consistency verification is performed based on the source modification times corresponding to the source data records and the target modification times corresponding to the target data records. The obtained consistency verification results include:

[0042] Step S302, obtain the first modification time range.

[0043] The modification time range refers to the modification time when the data record was last modified based on the modification operation. The modification operation can be an edit operation, an addition operation, or a deletion operation. The time length corresponding to the first modification time range can be set as needed. For example, when obtaining the first modification time range for the first time, it can be the modification time range corresponding to the operation record log obtained during the previous data synchronization as the first modification time range.

[0044] The source data modification time corresponding to the source data record refers to the time when the source data record was modified based on the modification operation. The target record modification time is the last modification time of the source data record synchronized when the target data record is synchronized. For example, for the data record with record index 00001, the last modification time of the source data record in the source data record set is 9:32:06 on a certain day, and the penultimate modification time is 8:30:02 on a certain day. However, during the last synchronization, the source data record corresponding to this record index was not synchronized to the corresponding target data record. Then the source data modification time corresponding to the source data record is 9:32:06 on a certain day, and the target data modification time corresponding to the target data record is 8:30:02 on a certain day.

[0045] The target data record can also be modified based on the modification log of the source data record. The modification time of the modification operation is stored in the modification log. When synchronizing the target data record based on the modification log, the time corresponding to the modification log can be used as the modification time corresponding to the target data record.

[0046] Step S304, obtain the first source record quantity corresponding to the source data record set. The first source record quantity is the quantity of source data records in the source data record set whose source data modification times are within the first modification time range and the source modification times are the first value; the first value is the modification time corresponding to the newly added data record.

[0047] For example, if SEQ_NO = 1 indicates that this record is a newly added record, then the first value is 1. Assume that the first modification time range is from 9:00 to 10:00 on a certain day. Then, if among the source data record set, 20 data records were modified between 9:00 and 10:00 on a certain day, and after the modification, the number of source data records with SEQ_NO = 1 is 10, then the first source record quantity is 10.

[0048] Step S306: Obtain the first target record quantity corresponding to the target data record set. The first target record quantity is the number of target data records in the target data record set whose target data modification time is within the first modification time range and whose target modification times are the first value.

[0049] Assume that the first modification time range is from 9:00 to 10:00 on a certain day. Then, if among the target data record set, 19 data records were modified between 9:00 and 10:00 on a certain day, and after the modification, the number of target data records with SEQ_NO = 1 is 9, then the first target record quantity is 9.

[0050] Step S308: Perform data consistency verification based on the first source record quantity and the first target record quantity to obtain a consistency verification result.

[0051] In the embodiments of the present application, it is possible to determine whether the first source record quantity and the first target record quantity are consistent. If they are inconsistent, it means that among the source data records in the source data record set whose source data modification time is within the first modification time range and whose source modification times are the first value, there are source data records that are inconsistent with the target data records, and it is necessary to synchronize the target data records based on the source data records within this range or the modification logs corresponding to the source data records within this range.

[0052] In the embodiments of the present application, for the data records whose last executed operation is an addition operation, by setting the first value as the modification times corresponding to the newly added data records, it is possible to find the data records whose last executed operation is an addition operation. And based on the comparison of the data record quantities corresponding to the modification time range, it is possible to determine whether the data in the source data record set and the target data record set within this modification time range are consistent, saving the time for comparing each data record within this modification time range one by one and improving the data verification efficiency.

[0053] In one embodiment, the data processing method further includes: when it is determined that the number of first source records is inconsistent with the number of first target records, dividing the first modification time range to obtain multiple sub-time ranges; using each sub-time range as the updated first modification time range respectively to continue data consistency verification based on the updated first modification time range until the verification stop condition is met; using the first modification time range in which the number of first source records is inconsistent with the number of first target records when the verification stops as the time range where the differential data records with differences are located; the verification stop condition includes at least one of the length of the updated first modification time range being less than the length threshold or the number of divisions exceeding the number threshold.

[0054] The initial first modification time range is a relatively large time range. If all the data records within the first modification time range are synchronized or compared one by one, the efficiency is relatively low. Therefore, the time range can be split into shorter-duration time ranges, and the data records in the shorter-duration time ranges are verified in the manner of steps S302 - S308 until the verification stop condition is met. For the consistent number of first source records and the number of first target records, it is confirmed that the data records within the modification time range are consistent, and the time range is not further divided or verified. If they are inconsistent, division and verification continue.

[0055] The division method can be the dichotomy method. For example, assume that the initial first modification time range is from 9:00 to 11:00 on a certain day. When dividing, it can be divided into two sub-ranges: from 9:00 to 10:00 on a certain day and from 10:00 to 11:00 on a certain day. Then determine the number of first source records and the number of first target records with a modification count of 1 within these two sub-ranges. If the number of first source records and the number of first target records with a modification count of 1 from 9:00 to 10:00 on a certain day are inconsistent, the time range from 9:00 to 10:00 on a certain day can be further divided. When it is divided to 9:00 to 9:15 on a certain day, if the time length from 9:00 to 9:15 is less than the length threshold or the number of divisions exceeds the number threshold, the data record set within the time range from 9:00 to 9:15 with a modification count of 1 can be used as the data record set where the differential data records are located.

[0056] In the embodiment of the present application, by dividing the first modification time range in which the number of first source records is inconsistent with the number of first target records and continuing to execute the verification process, the number of data records corresponding to the record indication information can be reduced, thereby reducing the time for synchronizing the target data record set based on the record indication information to obtain the synchronized target data record set.

[0057] In one embodiment, data consistency verification is performed based on the source modification times corresponding to the source data records and the target modification times corresponding to the target data records. The obtained consistency verification results include: when the source modification times corresponding to the same target record index are inconsistent with the target modification times, adding the target record index to the data difference index set; and using the data records corresponding to the respective target data record indices in the data difference index set as the differential data records with differences.

[0058] In the embodiments of the present application, the record index is the index of the data record. Each record index can be obtained, and the source modification times of the source data records corresponding to the record index and the target modification times of the target data records corresponding to the record index can be obtained. If the source modification times are inconsistent with the target modification times, the target record index is added to the data difference index set. The data records corresponding to the indices in the data difference index set are inconsistent in information between the source data record set and the target data record set, that is, there are differences. For example, the value of the index data in the source data record is changed from 100 to 50, while the value of the index data in the target data record remains 100.

[0059] In the embodiments of the present application, by comparing the source modification times corresponding to the same target record index with the target modification times, it is possible to determine the data records in the source data set that are inconsistent with the modifications in the target data set, so as to quickly locate which specific data records are inconsistent and improve the accuracy of verification.

[0060] In one embodiment, the target data record set is synchronized based on the operation log sequence corresponding to the source data record set. As Figure 4 shown, the data processing method further includes the following steps:

[0061] Step S402, obtaining the operation log record sequence corresponding to the source data record set.

[0062] The operation log record sequence corresponding to the source data record set is the operation log record for operating on the data records in the source data set. The operation log record sequence is sorted according to the operation time.

[0063] Step S404, adding corresponding operation type identifiers to each operation log record in the operation log record sequence based on the modification operation type corresponding to the operation log record sequence.

[0064] The operation type identifier can be used to identify the type of the operation. For example, an op_type field and an op_ts field can be added to each log record in each modification log record sequence. op_type represents the type of the operation. op_type = I indicates that the operation log record is an addition, op_type = U indicates that the operation log record is an edit, and op_type = D indicates that the operation log record is a deletion. The op_ts field annotates the modification timestamp of each record, and this modification timestamp can be accurate to the second level.

[0065] Step S406: Synchronize data based on the operation log record sequence to obtain a set of target data records.

[0066] After obtaining the operation log record sequence, the server can perform a synchronization operation on the data stored in the target data side based on the operation log record sequence to obtain a set of target data records.

[0067] In one embodiment, the operation log record sequence can be sent to a message queue, such as a Kafka message queue. The message queue generates an offset value corresponding to the operation log according to the chronological order of the operation times corresponding to each operation log record in the operation log record sequence. The earlier the operation time, the smaller the offset value. When obtaining operation logs according to the order of the message queue for data synchronization, the operation logs with smaller offset values can be obtained first for synchronization. Through the embodiments of the present application, it can be ensured that during data synchronization, the synchronization operation is performed in the chronological order of the operation logs, thereby improving the accuracy of synchronization.

[0068] In one embodiment, when the source modification times corresponding to the same target record index are inconsistent with the target modification times, adding the target record index to the data difference index set includes: determining the data record index corresponding to the operation log record whose corresponding operation type identifier is an edit type identifier as the target record index; when the source modification times corresponding to the target record index are inconsistent with the target modification times, adding the target record index to the data difference index set.

[0069] Taking the synchronization of the data from the source data side to the data lake as an example, when synchronizing data based on the modification log sequence, an incremental data file can be generated. The data index corresponding to the operation log record whose operation type identifier is an edit type identifier is written into the incremental data file. Compare whether the source modification times of the source data records corresponding to the data index in the incremental data file at the source data side are consistent with the target modification times of the target data records corresponding to the data index in the target data side. If they are inconsistent, write the target data index into the B data table, and this data table is used to record the data difference index set.

[0070] In the embodiments of the present application, when the last operation of the data record is an edit operation, the target record index is added to the data difference index set only when the source modification times corresponding to the target record index are inconsistent with the target modification times. Thereby, it is possible to specifically perform data consistency verification on the data records that have undergone edit operations, and use other methods to perform data consistency verification on the data records corresponding to delete operations and new addition operations, improving the flexibility of data verification.

[0071] In one embodiment, the target data record set is synchronized based on the operation log sequence corresponding to the source data record set. The method further includes: obtaining the target deletion record quantity corresponding to the target data record set, where the target deletion record quantity is the quantity of data records deleted based on the deletion operation log corresponding to the target data record set within the second modification time range; obtaining the source deletion record quantity corresponding to the source data record set, where the source deletion record quantity is the quantity of data records that have been deleted corresponding to the source data record set within the second modification time range; when the target deletion record quantity is inconsistent with the source deletion record quantity, it is determined that the data consistency verification result corresponding to the second modification time range is that there are differential data records.

[0072] For data records of the deletion type, taking the synchronization of data from the source data side to the data lake as an example, when synchronizing based on the operation log sequence corresponding to the source data record set, the data lake side deletes the records with the operation type of deletion, records the deleted records in data table C, and records the time of the deletion operation in the log record as the modification time. To verify the data consistency of the deleted records, count the number of records in table C within a certain time interval of the modification time range as the target deletion record quantity; count the reduction quantity of data records on the source data side within the same modification time period as the source deletion record quantity, and compare whether the target deletion record quantity is consistent with the source deletion record quantity to determine whether the data records deleted by the two data record sets within the second modification time range are consistent. If they are inconsistent, the data consistency verification result corresponding to the second modification time range is that there are differential data records. For the data records deleted within the second modification time range, they can be deleted again in the target data record set according to the corresponding data index, and the data index is deleted in data table C, indicating that the deletion has been performed again on the data lake side, thereby ensuring the consistency of the data between the source data side and the data lake side.

[0073] In one embodiment, the second modified time range can also be segmented. For example, the dichotomy method can be used to switch to a smaller time range, and then the above method for data consistency verification of deletion-type data records is used to continue the data verification until the verification stop condition is reached. The modified time period when the number of target deletion records is inconsistent with the number of source deletion records at the end of the verification is used as the time range where the differential data records with differences are located.

[0074] The method provided by the embodiments of the present application can be applied to synchronize the data of the source data end to the data lake. As an enterprise has more applications to be imported into the lake and more tables need to be imported, the complexity of the table structures of different applications is different. For example, some data tables to be imported into the lake have more fields and a huge amount of table data. To avoid the problem of data inconsistency caused by the loss of operation logs in the message queue during the process of synchronizing data from the server corresponding to the source application to the data lake due to network instability or other reasons, a solution is proposed that can quickly find the records where the source data is inconsistent with the target data lake during the process of synchronizing from the source application to the data lake, and quickly re-store the inconsistent records into the data lake, thereby quickly verifying the data imported into the lake and improving the efficiency of the enterprise in importing data into the lake.

[0075] Taking the synchronization of the data of the source data end to the data lake as an example, the data processing method provided by the embodiments of the present application is described as follows, including the following steps:

[0076] 1. Obtain the operation log record sequence corresponding to the source data record set;

[0077] The source data record set in the source data end can be stored in the mysql database. Each table in the source data end corresponds to a technical field SEQ_NO, which is used to mark the modification times of each record in each table. For example, SEQ_NO = 1 indicates that this record is a newly added record; SEQ_NO> 1 indicates that this record is a modified record. When the data records in the mysql database receive an add operation, an edit operation, or a delete operation, the data records in the mysql database change accordingly. The log of mysql records the operations on the tables in the database, and the log parsing platform can parse the logs recorded in the mysql log into individual records to form an operation log record sequence.

[0078] 2. Add corresponding operation type identifiers to each operation log record in the operation log record sequence based on the modification operation type corresponding to the operation log record sequence.

[0079] The data replication platform adds an op_type field and an op_ts field to each operation record. The record with op_type = I represents an addition operation, op_type = U represents an edit operation, and op_type = D represents a deletion operation. The op_ts field marks the modification timestamp of each operation log record.

[0080] 3. Perform data synchronization based on the operation log record sequence to obtain a target data record set.

[0081] The distributed message service platform receives the operation log record sequence sent by the data replication platform and converts each operation log record into a kafka message queue. The data lake side synchronizes the data to the data lake by consuming the kafka message queue.

[0082] Through the above steps, the data table of mysql can be synchronized to the data lake. However, during the data synchronization process, message loss may occur due to reasons such as unstable network, resulting in data inconsistency. To quickly find the records that are inconsistent between the source data side and the target data lake and quickly synchronize the inconsistent records to the data lake, step 4 can be continued.

[0083] 4. Obtain the source data record set and the target data record set;

[0084] The big data processing service platform can execute the steps of obtaining the source data record set and the target data record set. The big data processing service platform is a data processing and verification tool that can realize the function of processing and exporting the data records in the data lake table as files.

[0085] 5. Perform data consistency verification based on the source modification times corresponding to the source data records and the target modification times corresponding to the target data records to obtain a consistency verification result.

[0086] Different verification methods can be adopted for edit operation types, deletion operation types, and addition operation types.

[0087] For the addition operation type, by comparing the number of data records between the source data side and the data lake side within the same modification time period, if it is found that the number of data records between the two is inconsistent within the same time, the dichotomy method can be used to divide the time range, find the smaller time partition where the inconsistency occurs to narrow the search range, and finally accurately locate the inconsistent data within 15 minutes or a smaller time range interval. Take all the records in the source data side and the data lake side in this smaller time partition for one-by-one comparison to find the different records. Update the update time of the different records on the source data side and re-consume the operation records through the quasi-real-time data lake ingestion link to ingest the data records into the lake, that is, synchronize the data to the data lake.

[0088] For the edit operation type, load the incremental file RECORD_FILE generated by mysql, and load the incremental data records into the incremental file table A of the big data processing platform; create a new table B in the big data processing service platform to record the data inconsistencies between the data lake side and the source data side. Table B needs to register the unique index information of the data records inconsistent between the data lake side and the source data side, that is, register the index of the data records; then, the verification script can be obtained on the big data processing platform. By using the verification script, compare the value of the SEQ_NO field of each data record in Table A with the value of the SEQ_NO field of the corresponding data record in the data lake table. If the SEQ_NO field of a data record in Table A is inconsistent with the SEQ of the same data record in the data lake, insert the unique index value of this record in the data lake into Table B. The number of records in Table B can be checked regularly every day. If the number of records is greater than 0, then according to the unique index, update the update time of the corresponding table in the source data side for the updated difference records again, and re-consume the operation records through the quasi-real-time lake entry link to lake the data records, that is, synchronize the data to the data lake.

[0089] For the delete operation type, the data lake side deletes the records to be deleted synchronized from the source data side table, and separately registers the deleted records as Table C. To verify the data consistency of the deleted records, the data lake side counts the number of records in Table C within a certain time interval with a relatively large modification timestamp op_ts; the source data side counts the reduction in the number of records in the table within the same time period as the data lake side; compare whether the number of deleted records on the data lake side and the source data side is consistent within the same time period. If they are inconsistent, use the binary search method to find out the specific time period when the records are inconsistent, accurate to the data within 15 minutes or a smaller time range interval, and take out all the records in the smaller time partition of the source data side and the data lake side for comparison to find the difference records, delete the records that have not been deleted in the data table of the data lake, and register a deleted record in Table C.

[0090] It should be understood that although the steps in the flowcharts involved in the above-described embodiments are shown in sequence according to the arrows, these steps do not necessarily have to be executed in the order indicated by the arrows. Unless there is a clear description in this article, the execution of these steps has no strict order limit, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-described embodiments may include multiple steps or multiple stages. These steps or stages do not necessarily have to be executed at the same moment, but can be executed at different moments. The execution order of these steps or stages does not necessarily have to be sequential, but can be executed alternately or alternately with at least a part of other steps or steps or stages in other steps.

[0091] Based on the same inventive concept, an embodiment of this application further provides a data processing device for implementing the data processing method involved above. The solution provided by this device for solving problems is similar to the solution described in the above method. Therefore, the specific limitations in one or more embodiments of the data processing device provided below can refer to the limitations on the data processing method in the above text, and will not be elaborated here.

[0092] In one embodiment, as Figure 5 shown, a data processing device is provided, including:

[0093] A record set acquisition module 502, configured to acquire a source data record set and a target data record set; the target data record set is obtained by performing data synchronization based on the source data record set; the source data records in the source data record set correspond to source modification times, and the target data records in the target data record set correspond to target modification times;

[0094] A consistency verification module 504, configured to perform data consistency verification based on the source modification times corresponding to the source data records and the target modification times corresponding to the target data records, and obtain a consistency verification result;

[0095] A record indication information obtaining module 506, configured to obtain record indication information corresponding to the differential data records with differences between the source data record set and the target data record set based on the consistency verification result;

[0096] A first synchronization module 508, configured to perform data synchronization on the target data record set based on the record indication information to obtain a synchronized target data record set.

[0097] In one of the embodiments, the source data records in the source data record set correspond to source data modification times, and the target data records in the target data record set correspond to target data modification times; the consistency verification module is configured to: obtain a first modification time range; obtain the first source record quantity corresponding to the source data record set, where the first source record quantity is the quantity of source data records in the source data record set whose source data modification times are within the first modification time range and the source modification times are the first value; the first value is the modification time corresponding to a new data record; obtain the first target record quantity corresponding to the target data record set, where the first target record quantity is the quantity of target data records in the target data record set whose target data modification times are within the first modification time range and the target modification times are the first value; perform data consistency verification based on the first source record quantity and the first target record quantity to obtain a consistency verification result.

[0098] In one embodiment, the device further includes: a division module, configured to divide the first modification time range to obtain a plurality of sub-time ranges when it is determined that the number of first source records is inconsistent with the number of first target records; a continuous verification module, configured to use each of the sub-time ranges as the updated first modification time range to continue data consistency verification based on the updated first modification time range until a verification stop condition is met; a time range obtaining module, configured to use the first modification time range in which the number of first source records is inconsistent with the number of first target records when the verification stops as the time range where the differential data records with differences are located; the verification stop condition includes at least one of the length of the updated first modification time range being less than a length threshold or the number of divisions exceeding a number threshold.

[0099] In one embodiment, the consistency verification module is configured to: when the number of source modification times corresponding to the same target record index is inconsistent with the number of target modification times, add the target record index to the data difference index set; the data records corresponding to each target data record index in the data difference index set are used as the differential data records with differences.

[0100] In one embodiment, the device further includes: an operation log record sequence obtaining module, configured to obtain the operation log record sequence corresponding to the source data record set; an operation type identifier adding module, configured to add corresponding operation type identifiers to each operation log record in the operation log record sequence based on the modification operation type corresponding to the operation log record sequence; a second synchronization module, configured to perform data synchronization based on the operation log record sequence to obtain the target data record set; the step of adding the target record index to the data difference index set when the number of source modification times corresponding to the same target record index is inconsistent with the number of target modification times includes: determining the data record index corresponding to the operation log record whose corresponding operation type identifier is the edit type identifier as the target record index; when the number of source modification times corresponding to the target record index is inconsistent with the number of target modification times, adding the target record index to the data difference index set.

[0101] In one embodiment, the target data record set is synchronized based on the operation log sequence corresponding to the source data record set. The apparatus further includes: a target deletion record quantity acquisition module, configured to acquire the target deletion record quantity corresponding to the target data record set, where the target deletion record quantity is the quantity of data records deleted based on the deletion operation log corresponding to the target data record set within a second modification time range; a source deletion record quantity acquisition module, configured to acquire the source deletion record quantity corresponding to the source data record set, where the source deletion record quantity is the quantity of data records that have been deleted corresponding to the source data record set within the second modification time range; and a differential data record determination module, configured to, when the target deletion record quantity is inconsistent with the source deletion record quantity, determine that the data consistency check result corresponding to the second modification time range has differential data records.

[0102] Each module in the above data processing apparatus can be implemented in whole or in part by software, hardware, and their combination. The above modules can be embedded in or independent of the processor in the computer device in the form of hardware, or stored in the memory of the computer device in the form of software, so as to facilitate the processor to call and execute the operations corresponding to the above respective modules.

[0103] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as Figure 6 shown. The computer device includes a processor, a memory, and a network interface connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store XX data. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, it implements a data processing method.

[0104] Those skilled in the art can understand that Figure 6 the structure shown in

[0105] In one embodiment, a computer device is provided, including a memory and a processor. A computer program is stored in the memory. When the processor executes the computer program, the following steps are implemented: obtaining a source data record set and a target data record set; the target data record set is obtained by performing data synchronization based on the source data record set; the source data records in the source data record set correspond to source modification times, and the target data records in the target data record set correspond to target modification times; performing data consistency verification based on the source modification times corresponding to the source data records and the target modification times corresponding to the target data records to obtain a consistency verification result; obtaining record indication information corresponding to the differential data records with differences between the source data record set and the target data record set based on the consistency verification result; and performing data synchronization on the target data record set based on the record indication information to obtain a synchronized target data record set.

[0106] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the following steps are implemented: obtaining a source data record set and a target data record set; the target data record set is obtained by performing data synchronization based on the source data record set; the source data records in the source data record set correspond to source modification times, and the target data records in the target data record set correspond to target modification times; performing data consistency verification based on the source modification times corresponding to the source data records and the target modification times corresponding to the target data records to obtain a consistency verification result; obtaining record indication information corresponding to the differential data records with differences between the source data record set and the target data record set based on the consistency verification result; and performing data synchronization on the target data record set based on the record indication information to obtain a synchronized target data record set.

[0107] In one embodiment, a computer program product is provided, including a computer program. When the computer program is executed by a processor, the following steps are implemented: obtaining a source data record set and a target data record set; the target data record set is obtained by performing data synchronization based on the source data record set; the source data records in the source data record set correspond to source modification times, and the target data records in the target data record set correspond to target modification times; performing data consistency verification based on the source modification times corresponding to the source data records and the target modification times corresponding to the target data records to obtain a consistency verification result; obtaining record indication information corresponding to the differential data records with differences between the source data record set and the target data record set based on the consistency verification result; and performing data synchronization on the target data record set based on the record indication information to obtain a synchronized target data record set.

[0108] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0109] Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments can be completed by instructing relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above methods. Among them, any reference to a memory, database, or other medium used in the embodiments provided in this application can include at least one of non-volatile and volatile memories. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc. The databases involved in the embodiments provided in this application can include at least one of relational databases and non-relational databases. Non-relational databases can include distributed databases based on blockchain, etc., without limitation. The processors involved in the embodiments provided in this application can be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, data processing logics based on quantum computing, etc., without limitation.

[0110] The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity of description, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, it should be considered to be within the scope described in this specification.

[0111] The above-described embodiments merely represent several implementation manners of the present application. The description thereof is relatively specific and detailed, but it should not be construed as a limitation on the patent scope of the present application. It should be noted that for those of ordinary skill in the art, without departing from the concept of the present application, several modifications and improvements can still be made, and these all fall within the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the appended claims.

Claims

1. A data processing method, characterized in that, The method includes: Obtaining a source data record set and a target data record set; the target data record set is obtained by performing data synchronization based on the source data record set; the source data records in the source data record set correspond to source modification counts, and the target data records in the target data record set correspond to target modification counts; Performing data consistency verification based on the source modification count corresponding to the source data record and the target modification count corresponding to the target data record to obtain a consistency verification result; Based on the consistency verification result, obtaining record indication information corresponding to the differential data records with differences between the source data record set and the target data record set; Performing data synchronization on the target data record set based on the record indication information to obtain a synchronized target data record set; Wherein, the source data records in the source data record set correspond to source data modification times, and the target data records in the target data record set correspond to target data modification times; the performing data consistency verification based on the source modification count corresponding to the source data record and the target modification count corresponding to the target data record to obtain a consistency verification result includes: Obtaining a first modification time range; Obtaining a first source record count corresponding to the source data record set, where the first source record count is the number of source data records in the source data record set whose source data modification times are within the first modification time range and whose source modification counts are a first value; the first value is the modification count corresponding to a newly added data record; Obtaining a first target record count corresponding to the target data record set, where the first target record count is the number of target data records in the target data record set whose target data modification times are within the first modification time range and whose target modification counts are the first value; Performing data consistency verification based on the first source record count and the first target record count to obtain a consistency verification result.

2. The method according to claim 1, characterized in that, The method further includes: When it is determined that the first source record count is inconsistent with the first target record count, dividing the first modification time range to obtain multiple sub-time ranges; Using each of the sub-time ranges as the updated first modification time range to continue performing data consistency verification based on the updated first modification time range until a verification stop condition is met; Using the first modification time range in which the first source record count is inconsistent with the first target record count when the verification stops as the time range where the differential data records with differences are located; The verification stop condition includes at least one of the length of the updated first modification time range being less than a length threshold or the number of divisions exceeding a number threshold.

3. The method according to claim 1, wherein The performing data consistency verification based on the source modification count corresponding to the source data record and the target modification count corresponding to the target data record to obtain a consistency verification result includes: When the source modification count corresponding to the same target record index is inconsistent with the target modification count, adding the target record index to the data difference index set; The data records corresponding to each target data record index in the data difference index set are used as the difference data records with differences.

4. The method according to claim 3, characterized in that, The method further includes: Obtaining an operation log record sequence corresponding to the source data record set; Adding corresponding operation type identifiers to each operation log record in the operation log record sequence based on the modification operation type corresponding to the operation log record sequence; Performing data synchronization based on the operation log record sequence to obtain the target data record set; When the source modification times corresponding to the same target record index are inconsistent with the target modification times, adding the target record index to the data difference index set includes: Determining the data record index corresponding to the operation log record with the operation type identifier determined to be the edit type identifier as the target record index; When the source modification times corresponding to the target record index are inconsistent with the target modification times, adding the target record index to the data difference index set.

5. The method according to claim 1, wherein The target data record set is obtained by synchronizing based on the operation log sequence corresponding to the source data record set, and the method further includes: Obtaining the target deletion record quantity corresponding to the target data record set, where the target deletion record quantity is the quantity of data records deleted based on the deletion operation log within the second modification time range for the target data record set; Obtaining the source deletion record quantity corresponding to the source data record set, where the source deletion record quantity is the quantity of data records that have been deleted within the second modification time range for the source data record set; When the target deletion record quantity is inconsistent with the source deletion record quantity, determining that the data consistency check result corresponding to the second modification time range has difference data records.

6. A data processing device, characterized in that, The device includes: A record set acquisition module, configured to acquire a source data record set and a target data record set; the target data record set is obtained by performing data synchronization based on the source data record set; the source data records in the source data record set correspond to source modification times, and the target data records in the target data record set correspond to target modification times; A consistency check module, configured to perform data consistency check based on the source modification times corresponding to the source data records and the target modification times corresponding to the target data records to obtain a consistency check result; A record indication information obtaining module, configured to obtain record indication information corresponding to the difference data records with differences between the source data record set and the target data record set based on the consistency check result; A first synchronization module, configured to perform data synchronization on the target data record set based on the record indication information to obtain a synchronized target data record set; Wherein, the source data records in the source data record set correspond to source data modification times, and the target data records in the target data record set correspond to target data modification times; the consistency check module is specifically configured to: Obtain a first modification time range; Obtain a first source record quantity corresponding to the set of source data records, where the first source record quantity is the quantity of source data records in the set of source data records whose source data modification time is within a first modification time range and whose source modification times are a first value; the first value is the modification times corresponding to newly added data records; Obtain a first target record quantity corresponding to the set of target data records, where the first target record quantity is the quantity of target data records in the set of target data records whose target data modification time is within a first modification time range and whose target modification times are the first value; Perform data consistency verification based on the first source record quantity and the first target record quantity to obtain a consistency verification result.

7. A computer device, comprising a memory and a processor, the memory storing a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method described in any one of claims 1 to 5.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method described in any one of claims 1 to 5.

9. A computer program product, comprising a computer program, characterized in that, When this computer program is executed by a processor, it implements the steps of the method described in any one of claims 1 to 5.

Images