Structured data comparison method, device, electronic device and storage medium

Abstract

The present application relates to the field of data processing technology, and provides a structured data comparison method, device, electronic device and storage medium, the method comprising: obtaining a structured data set and an application scenario requirement corresponding to the structured data set, the structured data set comprising a first structured data and a second structured data, the first structured data being the reference source data, and the second structured data being the data after the change; obtaining a field comparison rule based on the application scenario requirement, and parsing the structured data set based on the field comparison rule to obtain a comparison item number and a sorting field; grouping the structured data set based on the comparison item number to obtain at least one group of comparison data; for each group of comparison data, sorting the comparison data based on the sorting field, and comparing them in sequence to obtain a comparison result. In this way, field comparison rules that meet the requirements of various application scenarios can be supported, and at the same time, batch data comparison can be supported to improve comparison efficiency.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a structured data comparison method, device, electronic device and storage medium. Background Art

[0002] Structured data, also known as row data, has specific fields and can be represented and stored using a relational database. The data can be logically expressed using a two-dimensional table structure. When migrating structured data, it is necessary to compare the data before and after the migration to ensure the correctness of the data migration work.

[0003] In the prior art, when comparing structured data, the structured data file can be loaded into a relational database such as Oracle / MySQL, and two tables can be associated by a full join (SQL Full Join) to compare the strings in the two tables.

[0004] However, the above data comparison method relies on the performance of a single-machine database. When there is batch data to be compared, it takes a lot of time and the comparison efficiency is poor. Summary of the invention

[0005] The present application provides a structured data comparison method, device, electronic device and storage medium, which can support field comparison rules that meet the needs of various application scenarios, and simultaneously support batch data comparison to improve comparison efficiency.

[0006] In a first aspect, the present application provides a structured data comparison method, the method comprising:

[0007] Acquire a structured data set and application scenario requirements corresponding to the structured data set, wherein the structured data set includes first structured data and second structured data, wherein the first structured data is reference source data, and the second structured data is changed data;

[0008] Acquire a field comparison rule based on the application scenario requirements, and parse the structured data set based on the field comparison rule to obtain a comparison item number and a sorting field;

[0009] Grouping the structured data set based on the comparison item numbers to obtain at least one set of comparison data;

[0010] For each set of comparison data, the comparison data are sorted based on the sorting field, and are compared in sequence to obtain a comparison result.

[0011] Optionally, parsing the structured data set based on the field comparison rule to obtain comparison item numbers and sorting fields includes:

[0012] Obtaining the corresponding comparison fields in the field comparison rule, wherein the comparison fields include a type field, a function field, and a number field;

[0013] Based on the comparison field, corresponding data in the structured data set is acquired, and based on the field comparison rule, the data is preprocessed to obtain the comparison item number and sorting field corresponding to the data.

[0014] Optionally, preprocessing the data based on the field comparison rule to obtain a comparison item number and a sorting field corresponding to the data includes:

[0015] Identify the character strings to be compared in the data based on the type field, and concatenate the character strings to be compared; the character strings to be compared include a primary key character string and a business character string;

[0016] The concatenated strings to be compared are processed based on the function field to obtain a comparison string;

[0017] The comparison string is assembled using the number field, and the comparison item number and sort field corresponding to the assembled comparison string are identified.

[0018] Optionally, grouping the structured data set based on the comparison item numbers includes:

[0019] A primary key string corresponding to each data in the structured data set is obtained, and the structured data set is grouped based on the primary key string and the comparison item number.

[0020] Optionally, the comparison data is sorted based on the sorting field, and the comparison data is sequentially compared to obtain a comparison result, including:

[0021] Obtain the number of data items in each set of comparison data, and determine whether the number of data items is greater than 1;

[0022] If so, the comparison data are sorted based on the sorting field, and the comparison data are compared in sequence to obtain a comparison result;

[0023] If not, a comparison result is obtained based on the source of the comparison data.

[0024] Optionally, obtaining a comparison result based on a source of the comparison data includes:

[0025] If the comparison data comes from the first structured data, determining that the comparison data is deleted data;

[0026] If the comparison data comes from the second structured data, it is determined that the comparison data is newly added data.

[0027] Optionally, obtaining a structured data set and an application scenario requirement corresponding to the structured data set includes:

[0028] Using standardized interfaces to obtain real-time data from different databases, and converting the format of the real-time data to obtain a structured data set;

[0029] For structured data sets corresponding to different databases, obtain the application scenario requirements corresponding to each structured data set.

[0030] Optionally, the method further includes:

[0031] After obtaining the comparison results, determining whether the comparison results are consistent;

[0032] If yes, then the structured data set is compared again at a preset interval, and the comparison result is obtained and overwritten with the previous comparison result to verify the accuracy of the comparison;

[0033] If not, an alarm is generated to remind the user to verify the comparison result.

[0034] In a second aspect, the present application provides a structured data comparison device, the device comprising:

[0035] An acquisition module, used to acquire a structured data set and application scenario requirements corresponding to the structured data set, wherein the structured data set includes first structured data and second structured data, wherein the first structured data is reference source data and the second structured data is changed data;

[0036] A parsing module, used to obtain a field comparison rule based on the application scenario requirements, and parse the structured data set based on the field comparison rule to obtain a comparison item number and a sorting field;

[0037] A grouping module, used for grouping the structured data set based on the comparison item number to obtain at least one group of comparison data;

[0038] The comparison module is used to sort the comparison data based on the sorting field for each group of comparison data, and compare them in sequence to obtain a comparison result.

[0039] In a third aspect, the present application provides an electronic device, comprising: a processor, and a memory communicatively connected to the processor;

[0040] The memory stores computer-executable instructions;

[0041] The processor executes the computer-executable instructions stored in the memory to implement the method as described in any one of the first aspects.

[0042] In a fourth aspect, the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are executed by a processor, they are used to implement the method as described in any one of the first aspects.

[0043] In a fifth aspect, the present application provides a computer program product, comprising a program code. When a computer runs the computer program, the program code executes the method as described in any one of the first aspects.

[0044] In summary, the present application provides a structured data comparison method, device, electronic device and storage medium, which can obtain a structured data set and the application scenario requirements corresponding to the structured data set, and obtain field comparison rules based on the application scenario requirements. Further, the structured data set is parsed based on the field comparison rules to obtain the comparison item number and the sorting field; then the structured data set is grouped based on the comparison item number to obtain at least one group of comparison data; further, for each group of comparison data, the comparison data is sorted based on the sorting field, and the comparisons are performed in sequence to obtain a comparison result, wherein the structured data set includes a first structured data and a second structured data, the first structured data is the reference source data, and the second structured data is the changed data; in this way, field comparison rules that meet the requirements of various application scenarios can be supported, and batch data comparison can be supported at the same time, while ensuring the correctness of data comparison, the comparison efficiency is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0046] Figure 1 A schematic diagram of an application scenario provided for an embodiment of the present application;

[0047] Figure 2 A schematic diagram of a flow chart of a structured data comparison method provided in an embodiment of the present application;

[0048] Figure 3 A schematic diagram of a framework of a standardized interface provided in an embodiment of the present application;

[0049] Figure 4 A flow chart of a batch data comparison method provided in an embodiment of the present application;

[0050] Figure 5A flowchart of a real-time data comparison method provided in an embodiment of the present application;

[0051] Figure 6 A schematic diagram of the structure of a structured data comparison device provided in an embodiment of the present application;

[0052] Figure 7 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application.

[0053] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0054] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0055] In order to facilitate the clear description of the technical solutions of the embodiments of the present application, in the embodiments of the present application, the words "first", "second" and the like are used to distinguish the same items or similar items with substantially the same functions and effects. For example, the first device and the second device are only used to distinguish different devices, and the order of precedence is not limited. Those skilled in the art can understand that the words "first", "second" and the like do not limit the quantity and execution order, and the words "first", "second" and the like do not necessarily limit the difference.

[0056] It should be noted that, in this application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in this application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

[0057] In the present application, "at least one" means one or more, and "plurality" means two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can mean: a, b, c, ab, ac, bc, or abc, where a, b, c can be single or multiple.

[0058] The present application is introduced below with reference to the accompanying drawings. Figure 1 A schematic diagram of an application scenario provided in an embodiment of the present application. A structured data comparison method provided in the present application can be applied to the following: Figure 1 In the application scenario shown, the application scenario includes: a first database 1011, a second database 1012, a data processing platform 102 and a display terminal 103; when the data in the first database 1011 is migrated to the second database 1012, it is necessary to compare the data before and after the migration.

[0059] Specifically, the data processing platform 102 can obtain the benchmark source data from the first database 1011 and the migrated data from the second database 1012, and accordingly, obtain the corresponding configuration file under the application scenario, which is a field comparison rule that is manually configured in advance and is applicable to the application scenario. The configuration file includes the comparison fields and comparison rules that require data comparison. Furthermore, the data processing platform 102 parses the configuration file to obtain the field comparison rules, and uses the field comparison rules to compare the benchmark source data obtained from the first database 1011 and the migrated data obtained from the second database 1012 to obtain the comparison results, and records the inconsistent data, generates an alarm prompt, and sends it to the display terminal 103, so that the user operates the display terminal 103 to find the inconsistent data for verification.

[0060] It should be noted that the display terminal 103 can be a display device corresponding to the data processing platform 102, or it can be a terminal device of the user. The embodiment of the present application does not make any specific limitation on this. In the present application, the data processing platform 102 can compare data from multiple databases, and can also compare changed data in a database. It is not limited to the data comparison in the data migration scenario between the two databases mentioned above. The embodiment of the present application does not make any specific limitation on the number of databases and application scenarios.

[0061] The above-mentioned terminal equipment can be a wireless terminal or a wired terminal. The wireless terminal can be a device that provides voice and / or other business data connectivity to users, a handheld device with wireless connection function, or other processing equipment connected to a wireless modem. The wireless terminal can communicate with one or more core network devices via a radio access network (Radio Access Network, referred to as RAN). The wireless terminal can be a mobile terminal, such as a mobile phone (or "cellular" phone) and a computer with a mobile terminal. For example, it can be a portable, pocket-sized, handheld, computer-built-in or vehicle-mounted mobile device that exchanges language and / or data with the radio access network. For another example, the wireless terminal can also be a personal communication service (Personal Communication Service, referred to as PCS) phone, a cordless phone, a Session Initiation Protocol (Session Initiation Protocol, referred to as SIP) phone, a wireless local loop (Wireless Local Loop, referred to as WLL) station, a personal digital assistant (Personal Digital Assistant, referred to as PDA) and other devices. The wireless terminal may also be referred to as a system, a subscriber unit, a subscriber station, a mobile station, a mobile station, a remote station, a remote terminal, an access terminal, a user terminal, a user agent, a user device or user equipment, without limitation. Optionally, the terminal device may also be a smart phone, a tablet computer or other device.

[0062] Therefore, the comparison of structured data can be applied to scenarios such as data migration or online maintenance transactions. Specifically, when migrating structured data, it is necessary to compare the data before and after the migration to ensure the correctness of the data migration work; when modifying data in online maintenance transactions, it is necessary to see the difference before and after the data modification in real time on the front end, that is, it is necessary to record the difference before and after the data modification during the online maintenance transaction.

[0063] In one possible implementation, when comparing structured data, the structured data file can be loaded into a relational database such as Oracle / MySQL, and two tables can be associated through a full connection, and then the strings in the two tables can be compared.

[0064] However, the above data comparison method requires loading files into the database and relies on the performance of a single-machine database. When there is batch data to be compared, it takes a lot of time and the comparison efficiency is poor.

[0065] It should be noted that when there is batch data to be compared, multiple comparison rules are also needed to adapt to different comparison requirements. However, the implementation of complex comparison rules has a certain threshold for developers. In addition, for the need for real-time comparison, if it is implemented based on online transactions, it needs to be bound to the transaction business logic, which will make online transactions very complicated.

[0066] In response to the above problems, the present application provides a structured data comparison method, which is developed using Spark program, that is, distributed technology is used to replace a stand-alone database, and the field comparison rules in different application scenarios are loaded and parsed through the Spark program, and then the structured data set is parsed based on the field comparison rules to obtain the comparison item number and the sorting field. The structured data set includes the reference source data and the changed data. Furthermore, the structured data set is grouped using the comparison item number to obtain at least one group of comparison data. Furthermore, the comparison data in the same group is sorted based on the sorting field, and then the data is compared, that is, the comparison data of the same record and the same comparison item number are put together and sorted for comparison. This can support field comparison rules that meet the needs of various application scenarios, and then support flexible comparison of batch data, thereby improving comparison efficiency.

[0067] The technical solution of the present application is described in detail with specific embodiments below. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0068] Figure 2 A flow chart of a structured data comparison method provided in an embodiment of the present application is shown in FIG. Figure 2 As shown, the structured data comparison method includes the following steps:

[0069] S201. Obtain a structured data set and application scenario requirements corresponding to the structured data set, wherein the structured data set includes first structured data and second structured data, wherein the first structured data is reference source data, and the second structured data is changed data.

[0070] In an embodiment of the present application, the first structured data and the second structured data may come from the same database or different databases, the structured data may be a database table or a data file, the first structured data is the reference source data, that is, the original data stored in the database, and the second structured data is the data after changes, such as newly added, deleted, updated and migrated data, and the embodiment of the present application does not make any specific limitations on this.

[0071] Optionally, the second structured data may be change data captured in real time by a real-time data change collection component from a redolog log, and the change data may include all fields of records after addition, deletion, and update. Then the first structured data may include all fields of records before addition, deletion, and update. In this way, the present application may collect real-time data and perform a comparison of the real-time data.

[0072] S202: Acquire a field comparison rule based on the application scenario requirements, and parse the structured data set based on the field comparison rule to obtain a comparison item number and a sorting field.

[0073] In an embodiment of the present application, the field comparison rule is a pre-defined configuration file, and different application scenarios correspond to different configuration files. The configuration file includes comparison fields and comparison rules. The comparison field is the field type corresponding to the data in the structured data set, such as field name, type field, function field, and number field, etc. The comparison rule can be to use the obtained function field to process the data and then compare it in sequence, and / or use the number field to combine the data and then compare it in sequence, and / or use the data corresponding to a certain type of field to compare in sequence, etc. The embodiment of the present application does not limit the specific content corresponding to the comparison field and the comparison rule, which depends on the business scenario and supports the setting of complex and diverse field comparison rules.

[0074] In this step, the structured data set is parsed based on the field comparison rule, that is, the comparison field corresponding to the field comparison rule is parsed, and the comparison field is used to obtain the comparison item number and sorting field corresponding to the structured data set. The comparison item number is used to mark the fields that need to be compared in the comparison data, such as 200010 indicates that the 9th to 10th fields in a certain data need to be compared; the sorting field may refer to a field in the data that can be used for sorting, such as an operator number, an organization number, an update timestamp, etc., which can be manually customized and the embodiment of the present application does not make specific limitations on this.

[0075] S203: Group the structured data set based on the comparison item numbers to obtain at least one group of comparison data.

[0076] In this step, the structured data set is grouped based on the comparison item number, that is, the character strings to be compared are assembled based on the comparison item number, and further, the comparison data of the same record and the same comparison item number are grouped together to obtain at least one group of comparison data.

[0077] S204 , for each group of comparison data, sort the comparison data based on the sorting field, and compare them in sequence to obtain a comparison result.

[0078] In this step, the comparison data is sorted based on the sorting field, such as the data update timestamp corresponding to each group of structured data sets, and the data in each group is sorted in chronological order based on the data update timestamp. Furthermore, the sorted data are compared in sequence to obtain comparison results; the comparison results include results with consistent data and results with inconsistent data. For results with inconsistent data, the present application can also store inconsistent data for subsequent queries.

[0079] Therefore, the present application provides a structured data comparison method, which can obtain a structured data set and an application scenario requirement corresponding to the structured data set, and obtain a field comparison rule based on the application scenario requirement, and further, parse the structured data set based on the field comparison rule to obtain a comparison item number and a sorting field; then group the structured data set based on the comparison item number to obtain at least one group of comparison data; further, for each group of comparison data, sort the comparison data based on the sorting field, and compare them in sequence to obtain a comparison result, wherein the structured data set includes a reference source data and a changed data; in this way, the structured data comparison method provided by the present application can support data comparison according to field comparison rules that meet various application scenario requirements, and at the same time support batch data comparison, thereby improving comparison efficiency while ensuring the correctness of data comparison.

[0080] Optionally, parsing the structured data set based on the field comparison rule to obtain comparison item numbers and sorting fields includes:

[0081] Obtaining the corresponding comparison fields in the field comparison rule, wherein the comparison fields include a type field, a function field, and a number field;

[0082] Based on the comparison field, corresponding data in the structured data set is acquired, and based on the field comparison rule, the data is preprocessed to obtain the comparison item number and sorting field corresponding to the data.

[0083] In an embodiment of the present application, the comparison field may include a type field, a function field and a number field, wherein the type field is used to mark the type of field comparison, such as Y, N, K, S, C, HB, D, etc., Y indicates that the field is used for comparison, N indicates that the field is not used for comparison, K indicates that the field is used as a primary key, C indicates that the field is a fixed field and can be manually customized, S indicates that the field corresponds to a comparison item number, HB indicates that the field participates in the comparison as part of a joint field, and D indicates that the field is a sorting field; the function field may refer to a function expression for processing the field, such as trim, to_char, lower, etc., wherein to_char indicates that the data type of the field is converted to a character type, trim indicates that spaces in the data of the field are deleted, and trim indicates that the data of the field is converted into a lowercase string. If there is no function field, it means that there is no need to process the field; the number field is used to identify the comparison result of each business field, and the comparison result can be filtered according to this number, such as the comparison item number of the 9th to 10th bytes of a certain data is 200010.

[0084] For example, the corresponding comparison fields in the field comparison rules in the embodiment of the present application may be as shown in Table 1, and field comparison rules that meet the requirements of different application scenarios may be formed based on the comparison fields:

[0085] Table 1

[0086]

[0087] Among them, the first column represents the English names of the fields corresponding to different fields in the data. The above is just an example. The English names of the fields corresponding to the fields in the data can also be expressed in other forms. The embodiments of the present application do not make specific limitations on this. The second column represents the type field corresponding to the data, the third column represents the function field corresponding to the data, and the fourth column represents the number field corresponding to the data. The type field, the function field and the number field can be custom designed. The embodiments of the present application do not make specific limitations on this. The above is just an example.

[0088] It can be understood that if the data in the structured data set corresponds to a function field, the function corresponding to the function field can be used to perform function processing on the field, and then the data after function processing can be used for data comparison. Optionally, commonly used string processing functions are preset in the field comparison rules. At the same time, the field comparison rules can also support user-defined functions to expand the function type corresponding to the function field. The embodiments of the present application do not make specific limitations on this.

[0089] Therefore, the embodiment of the present application supports complex and diverse field comparison rules, and can pre-process data by obtaining the comparison fields in the field comparison rules, thereby improving processing flexibility.

[0090] Optionally, preprocessing the data based on the field comparison rule to obtain a comparison item number and a sorting field corresponding to the data includes:

[0091] Identify the character strings to be compared in the data based on the type field, and concatenate the character strings to be compared; the character strings to be compared include a primary key character string and a business character string;

[0092] The concatenated strings to be compared are processed based on the function field to obtain a comparison string;

[0093] The comparison string is assembled using the number field, and the comparison item number and sort field corresponding to the assembled comparison string are identified.

[0094] In an embodiment of the present application, the primary key string may refer to the string data corresponding to the primary key, such as cst_id, and the business string refers to the string data corresponding to multiple business fields that constitute a certain piece of data. For example, the contact address includes multiple business fields such as country, region, province, city, county, and detailed address, that is, Country A, Province B, City C, County D, and Village E.

[0095] Optionally, if the data is composed of multiple business fields, each field is separated by |@|, such as the data corresponding to the contact address: Country A |@| Province B |@| City C |@| County D |@| Village E. Therefore, the present application supports merging multiple business fields as one field for comparison, but if any of the merged fields changes, the merged field as a whole needs to be used as the output result for data comparison. The embodiment of the present application does not specifically limit the separator used to combine each business field together. The above is only an example.

[0096] It is understandable that a piece of data can be split into multiple business fields by bytes for comparison, so a certain business field in a piece of data can represent a business meaning, and other adjacent business fields can represent other business meanings.

[0097] In this step, the character string to be compared in the data is identified based on the type field, and unnecessary comparison information can be removed, such as identifying the primary key field in the data and splicing the primary key character string based on the identified primary key field; further, the comparison character string is assembled using the numbering field, such as numbering each comparison item, assembling the character string to be compared, and then identifying the comparison item number and sorting field corresponding to the assembled comparison character string; wherein, the fixed output field corresponding to the comparison character string can also be identified for data comparison to improve the accuracy of the comparison.

[0098] Therefore, the embodiments of the present application can support complex comparison logic, such as data can be split into multiple business fields for comparison, and support functions to process data, and can also remove unnecessary comparison information, and is applicable to a wide range of scenarios.

[0099] Optionally, grouping the structured data set based on the comparison item numbers includes:

[0100] A primary key string corresponding to each data in the structured data set is obtained, and the structured data set is grouped based on the primary key string and the comparison item number.

[0101] It can be understood that the present application can group the structured data set based on the primary key string.

[0102] Therefore, the embodiments of the present application can group structured data sets by combining primary key strings and comparison item numbers, or can group structured data sets based only on primary key strings or comparison item numbers to improve the accuracy of grouping.

[0103] Optionally, the comparison data is sorted based on the sorting field, and the comparison data is sequentially compared to obtain a comparison result, including:

[0104] Obtain the number of data items in each set of comparison data, and determine whether the number of data items is greater than 1;

[0105] If so, the comparison data are sorted based on the sorting field, and the comparison data are compared in sequence to obtain a comparison result;

[0106] If not, a comparison result is obtained based on the source of the comparison data.

[0107] For example, in Figure 1 In the application scenario, after the data processing platform 102 groups the reference source data obtained from the first database 1011 and the migrated data obtained from the second database 1012 based on the comparison item number to obtain at least one set of comparison data, the data processing platform 102 can obtain the number of data items N in each set of comparison data, and determine whether the number of data items N is greater than 1; if so, sort the comparison data based on the sorting field, and compare them in sequence to obtain the comparison result; if not, obtain the comparison result based on whether the comparison data comes from the first database 1011 or the second database 1012, where N is a positive integer greater than or equal to 1.

[0108] It is understandable that the embodiments of the present application may also set preset requirements, determine whether the preset requirements are met based on the number of data items, and determine whether the comparison data needs to be sorted and then compared, so as to obtain the comparison results. The embodiments of the present application do not specifically limit the preset requirements.

[0109] Therefore, the embodiment of the present application can use the sorting field to sort and compare the comparison data based on whether the number of data items in each group of comparison data meets the preset requirements to obtain the comparison result. When the number of data items does not meet the preset requirements, the comparison result can be directly determined according to the source of the comparison data to improve the comparison efficiency.

[0110] Optionally, obtaining a comparison result based on a source of the comparison data includes:

[0111] If the comparison data comes from the first structured data, determining that the comparison data is deleted data;

[0112] If the comparison data comes from the second structured data, it is determined that the comparison data is newly added data.

[0113] In this step, if it is determined that the comparison data comes from the first structured data and does not exist in the second structured data, it means that the comparison data is deleted data; if it is determined that the comparison data comes from the second structured data and does not exist in the first structured data, it means that the comparison data is newly added data.

[0114] Therefore, the embodiment of the present application can directly obtain the comparison result based on the source of the comparison data, thereby improving the timeliness of the comparison.

[0115] Optionally, obtaining a structured data set and an application scenario requirement corresponding to the structured data set includes:

[0116] Using standardized interfaces to obtain real-time data from different databases, and converting the format of the real-time data to obtain a structured data set;

[0117] For structured data sets corresponding to different databases, obtain the application scenario requirements corresponding to each structured data set.

[0118] In an embodiment of the present application, a standardized interface may refer to a unified interface designed for calling data in different databases. The standardized interface is a pre-defined code framework. When calling data in different databases, it is only necessary to modify the corresponding parameter values ​​in the code framework.

[0119] In this step, Kafka message middleware can be used to read the real-time change data of the database. However, the results of parsing the change data by different database products vary greatly. Therefore, it is necessary to standardize the change data, that is, convert the format of the collected real-time data to obtain a structured data set that can be used for data comparison. The data format after standardization is unified, which facilitates the unification of subsequent processing processes.

[0120] Among them, the data during batch data comparison is in text format, and the standardized output of the real-time data comparison process is also in text format, and the text file formats of the two are consistent.

[0121] For example, Figure 3 A schematic diagram of a standardized interface framework provided in an embodiment of the present application; Figure 3 As shown, the embodiment of the present application provides a unified standardized interface for calling the changed data of each database, each changed data corresponds to a corresponding implementation class, and the call of each implementation class needs to be implemented based on the standardized interface; for example, the implementation classes corresponding to the changed data of databases such as Mysql, Oracle, DB2, OceanBase, etc. include: databaseType (data type), processinsertRec (addition), processDeleteRec (deletion), processUpdateNotKeyRec (update non-primary key ordinary fields), processUpdateKeyRec (update primary key fields), which are implemented based on the standardized interface. Therefore, for various types of databases, the user only needs to implement the standardized interface.

[0122] It should be noted that different implementation classes in the calling database correspond to different comparison results. Taking Oracle database as an example, the comparison result of each DML operation includes the transaction start and end markers, and there is only one result statement for adding / deleting; the comparison result corresponding to updating non-primary key ordinary fields is updating the previous record or the next record; the comparison result corresponding to updating the primary key field is three statements V, D, and I. For the updated primary key field of the OceanBase database, the corresponding comparison result is a delete operation and an insert operation. For other DML operations, the corresponding comparison result is a record containing information before and after the change. Different implementation classes in different databases correspond to different comparison results, and the embodiments of the present application will not be described one by one here.

[0123] It is understandable that real-time comparison usually requires online maintenance transactions or a special comparison program on the back end to perform customized comparisons on each data change. The development and maintenance costs are very high and the workload is repetitive. However, the embodiments of the present application can implement real-time data comparison based on a standardized interface and also support parsing data from different database products. The real-time comparison logic is consistent with the batch comparison logic, shares the core code, and has the same processing caliber, which can reduce the subsequent operation and maintenance costs.

[0124] Optionally, the method further includes:

[0125] After obtaining the comparison results, determining whether the comparison results are consistent;

[0126] If yes, then the structured data set is compared again at a preset interval, and the comparison result is obtained and overwritten with the previous comparison result to verify the accuracy of the comparison;

[0127] If not, an alarm is generated to remind the user to verify the comparison result.

[0128] In an embodiment of the present application, the preset time may refer to the time set for re-comparison. The embodiment of the present application does not limit the specific numerical value corresponding to the preset time, which may be one day. The embodiment of the present application does not specifically limit the content of the generated alarm prompt and the sending form. It may generate an alarm prompt based on inconsistent data, such as an alarm prompt of "Data 1 in Table A and Data 2 in Table B are inconsistent", and the sending form may be a text message and a display box, etc.

[0129] It should be noted that although real-time data comparison makes up for the shortcoming of poor timeliness of batch processing, there is also the possibility of loss of real-time data. Therefore, this application supports re-execution of data comparison after a preset time interval, and the comparison result will overwrite the previous comparison result. That is, through the database idempotence method, the data reading application is unaware of repeated comparisons, thereby ensuring data timeliness while also ensuring data integrity.

[0130] Therefore, the embodiment of the present application reduces the possibility of real-time data loss by re-execution, thereby improving data timeliness and data integrity. It can also generate alarm prompts to remind users of abnormal comparison results, so as to facilitate subsequent data calls for verification and improve convenience.

[0131] In combination with the above embodiments, the structured data comparison method provided by the present application supports two data comparison modes: real-time and batch. The real-time comparison is implemented based on the flink datastream technology, and the batch comparison is implemented based on the flink dataset technology. The real-time comparison logic is consistent with the batch comparison logic, and the comparison caliber is consistent. The two share the core code. Specifically, Figure 4 A flow chart of a batch data comparison method provided in an embodiment of the present application; Figure 4 As shown, the batch data comparison method includes the following steps:

[0132] Step 1: Start the Spark program, load and parse the configuration file (application scenario requirements), read the data in file A and file B from the data file, and\or read the data in table a and table b from the database table, parse the data according to the configuration file, and execute step 2.

[0133] It should be noted that the data in file A and table a are reference source data, and the data in file B and table b are changed data. The embodiment of the present application can perform comparison processing on the data in file A and file B separately, or on the data in table a and table b separately, or on the data in file A and file B separately, and on the data in table a and table b separately. The embodiment of the present application does not make specific limitations on this.

[0134] Step 2: Merge the two parts of data in file A and file B, and / or merge the two parts of data in table a and table b, group the input data (structured data set) according to the primary key and the comparison item number, and obtain at least one set of comparison data. For the comparison data in the same group, they can be sorted according to the "data update timestamp", where the number of data items in the same group is recorded as N. Further, the comparison result is determined according to the number of data items N, where N is a positive integer greater than or equal to 1, and then execute step 3.

[0135] Step 3: If the number of data entries is 1, and it is determined that the data comes from file A and / or table a, and there is no such data in file B and / or table b, it means that the record corresponding to the data is deleted. If it is determined that the data comes from file B and / or table b, and there is no such data in file A and / or table a, it means that the record corresponding to the data is newly added. If the number of data entries is greater than 1, the sorted results can be compared in sequence to obtain the comparison results, wherein the data corresponding to the inconsistent items can be recorded in the comparison results for users to view.

[0136] Figure 5 A flow chart of a real-time data comparison method provided in an embodiment of the present application; Figure 5 As shown, the real-time data comparison method includes the following steps:

[0137] Step 1: Start the Spark Streaming program, load and parse the configuration file (application scenario requirements), and use the Kafka message middleware to call a unified standardized interface to collect the changed data and original data in the Oracle database in real time. In order to adapt to various database products, the change records are standardized into a json string in a unified format. Correspondingly, it is also necessary to unify the value of the data update type and adapt the records of the updated primary key, that is, convert the changed data and the original data into a unified format. Among them, the modified data in the Oracle database is called changed data, and then execute the subsequent steps.

[0138] It should be noted that the subsequent execution steps are Figure 4 Step 2 and step 3 in the embodiment are similar, and details can be found in the above Figure 4 The description of the embodiment is not repeated here.

[0139] This application supports both real-time and batch methods for structured data comparison, and supports complex and diverse field comparison rules in the form of configuration files. The learning and usage thresholds are low, so that users do not need to repeat development. This application also presets commonly used string processing functions, supports custom functions to expand function processing functions, and performs data comparison in a combined stream and batch manner, which reduces the subsequent operation and maintenance costs while improving timeliness.

[0140] In the foregoing embodiments, the structured data comparison method provided in the embodiments of the present application is introduced, and in order to implement the various functions in the method provided in the embodiments of the present application, the electronic device as the execution subject may include a hardware structure and / or a software module, and implement the above functions in the form of a hardware structure, a software module, or a hardware structure plus a software module. Whether one of the above functions is executed in the form of a hardware structure, a software module, or a hardware structure plus a software module depends on the specific application and design constraints of the technical solution.

[0141] For example, Figure 6 A schematic diagram of a structured data comparison device provided in an embodiment of the present application is shown in FIG. Figure 6 As shown, the device includes: an acquisition module 610, a parsing module 620, a grouping module 630 and a comparison module 640; wherein the acquisition module 610 is used to acquire a structured data set and an application scenario requirement corresponding to the structured data set, the structured data set includes a first structured data and a second structured data, the first structured data is a reference source data, and the second structured data is a changed data;

[0142] The parsing module 620 is used to obtain a field comparison rule based on the application scenario requirements, and parse the structured data set based on the field comparison rule to obtain a comparison item number and a sorting field;

[0143] The grouping module 630 is used to group the structured data set based on the comparison item number to obtain at least one set of comparison data;

[0144] The comparison module 640 is used to sort the comparison data based on the sorting field for each group of comparison data, and compare them in sequence to obtain a comparison result.

[0145] Optionally, the parsing module 620 includes an acquisition unit and a preprocessing unit;

[0146] Specifically, the acquisition unit is used to acquire the corresponding comparison fields in the field comparison rule, and the comparison fields include a type field, a function field and a number field;

[0147] The preprocessing unit is used to obtain corresponding data in the structured data set based on the comparison field, and preprocess the data based on the field comparison rule to obtain the comparison item number and sorting field corresponding to the data.

[0148] Optionally, the preprocessing unit is specifically used to:

[0149] Identify the character strings to be compared in the data based on the type field, and concatenate the character strings to be compared; the character strings to be compared include a primary key character string and a business character string;

[0150] The concatenated strings to be compared are processed based on the function field to obtain a comparison string;

[0151] The comparison string is assembled using the number field, and the comparison item number and sort field corresponding to the assembled comparison string are identified.

[0152] Optionally, the grouping module 630 is specifically configured to:

[0153] A primary key string corresponding to each data in the structured data set is obtained, and the structured data set is grouped based on the primary key string and the comparison item number.

[0154] Optionally, the comparison module 640 includes a judgment unit, a comparison unit and a determination unit;

[0155] Specifically, the judging unit is used to obtain the number of data items in each set of comparison data, and judge whether the number of data items is greater than 1;

[0156] The comparison unit is used to sort the comparison data based on the sorting field when the number of data items is greater than 1, and compare them in sequence to obtain a comparison result;

[0157] The determining unit is used to obtain a comparison result based on a source of the comparison data when the number of data items is equal to 1.

[0158] Optionally, the determining unit is specifically configured to:

[0159] If the comparison data comes from the first structured data, determining that the comparison data is deleted data;

[0160] If the comparison data comes from the second structured data, it is determined that the comparison data is newly added data.

[0161] Optionally, the acquisition module 610 is specifically configured to:

[0162] Using standardized interfaces to obtain real-time data from different databases, and converting the format of the real-time data to obtain a structured data set;

[0163] For structured data sets corresponding to different databases, obtain the application scenario requirements corresponding to each structured data set.

[0164] Optionally, the device further includes a judgment module, wherein the judgment module is configured to:

[0165] After obtaining the comparison results, determining whether the comparison results are consistent;

[0166] If yes, then the structured data set is compared again at a preset interval, and the comparison result is obtained and overwritten with the previous comparison result to verify the accuracy of the comparison;

[0167] If not, an alarm is generated to remind the user to verify the comparison result.

[0168] The specific implementation principle and effect of a structured data comparison device provided in an embodiment of the present application can be found in the relevant description and effects corresponding to the above embodiment, and will not be elaborated here.

[0169] The present application also provides a schematic diagram of the structure of an electronic device. Figure 7 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application is shown in FIG. Figure 7 As shown, the electronic device may include: a processor 701 and a memory 702 communicatively connected to the processor; the memory 702 stores a computer program; the processor 701 executes the computer program stored in the memory 702, so that the processor 701 executes the method described in any of the above embodiments.

[0170] The memory 702 and the processor 701 may be connected via a bus 703 .

[0171] An embodiment of the present application further provides a computer-readable storage medium, which stores computer program execution instructions. When the computer program execution instructions are executed by a processor, they are used to implement the method described in any of the aforementioned embodiments of the present application.

[0172] An embodiment of the present application further provides a chip for executing instructions, wherein the chip is used to execute the method described in any of the aforementioned embodiments as executed by an electronic device in any of the aforementioned embodiments of the present application.

[0173] An embodiment of the present application further provides a computer program product, which includes a program code. When a computer runs the computer program, the program code executes the method described in any of the aforementioned embodiments as executed by an electronic device in any of the aforementioned embodiments of the present application.

[0174] In the technical solution of this application, the collection, storage, use, processing, transmission, provision and disclosure of information such as financial data or user data involved shall comply with the provisions of relevant laws and regulations and shall not violate public order and good morals.

[0175] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of modules is only a logical function division. There may be other division methods in actual implementation, such as multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or modules, which can be electrical, mechanical or other forms.

[0176] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to implement the solution of this embodiment.

[0177] In addition, each functional module in each embodiment of the present application can be integrated into one processing unit, or each module can exist physically separately, or two or more modules can be integrated into one unit. The above-mentioned module-composed unit can be implemented in the form of hardware or in the form of hardware plus software functional units.

[0178] The above-mentioned integrated module implemented in the form of a software function module can be stored in a computer-readable storage medium. The above-mentioned software function module is stored in a storage medium, including a number of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor to perform some steps of the method described in each embodiment of the present application.

[0179] It should be understood that the processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), etc. A general-purpose processor may be a microprocessor or any conventional processor. The steps of the method disclosed in the application may be directly implemented as being executed by a hardware processor, or may be implemented by a combination of hardware and software modules in the processor.

[0180] The memory may include high-speed random access memory (RAM), and may also include non-volatile memory (NVM), such as at least one disk storage, and may also be a USB flash drive, a mobile hard disk, a read-only memory, a disk or an optical disk, etc.

[0181] The bus may be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, the bus in the drawings of the present application is not limited to only one bus or one type of bus.

[0182] The above storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk. The storage medium can be any available medium that can be accessed by a general or special computer.

[0183] An exemplary storage medium is coupled to a processor so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can be located in an application specific integrated circuit (ASIC). Of course, the processor and the storage medium can also exist as discrete components in an electronic device or a main control device.

[0184] The above is only a specific implementation of the embodiment of the present application, but the protection scope of the embodiment of the present application is not limited thereto, and any changes or replacements within the technical scope disclosed in the embodiment of the present application should be included in the protection scope of the embodiment of the present application. Therefore, the protection scope of the embodiment of the present application should be based on the protection scope of the claims.

Claims

1. A structured data comparison method, characterized in that: The method comprises: Acquire a structured data set and application scenario requirements corresponding to the structured data set, wherein the structured data set includes first structured data and second structured data, wherein the first structured data is reference source data, and the second structured data is changed data; Acquire a field comparison rule based on the application scenario requirements, and parse the structured data set based on the field comparison rule to obtain a comparison item number and a sorting field; Grouping the structured data set based on the comparison item numbers to obtain at least one set of comparison data; For each group of comparison data, the comparison data are sorted based on the sorting field, and the comparison data are sequentially compared to obtain comparison results, wherein the comparison results include results of consistent data and results of inconsistent data; The step of parsing the structured data set based on the field comparison rule to obtain comparison item numbers and sorting fields includes: Obtaining a comparison field corresponding to the field comparison rule, wherein the comparison field includes a type field, a function field, and a number field, wherein the type field is used to mark the type of field comparison, and the type includes a type used to characterize whether the field is compared; Acquire corresponding data in the structured data set based on the comparison field, and Identify the character strings to be compared in the data based on the type field, and concatenate the character strings to be compared; the character strings to be compared include a primary key character string and a business character string; The concatenated strings to be compared are processed based on the function field to obtain a comparison string; Assembling the comparison string using the number field, and identifying the comparison item number and sort field corresponding to the assembled comparison string; The sorting of the comparison data based on the sorting field and sequentially comparing the comparison data to obtain the comparison result includes: Obtain the number of data items in each set of comparison data, and determine whether the number of data items is greater than 1; When the number of data items is greater than 1, the comparison data is sorted based on the sorting field, and the comparisons are performed in sequence to obtain a comparison result; When the number of data items is equal to 1, if the comparison data comes from the first structured data, it is determined that the comparison data is deleted data; if the comparison data comes from the second structured data, it is determined that the comparison data is newly added data.

2. The method according to claim 1, characterized in that The structured data set is grouped based on the comparison item number, comprising: A primary key string corresponding to each data in the structured data set is obtained, and the structured data set is grouped based on the primary key string and the comparison item number.

3. The method according to claim 1, characterized in that Obtaining a structured data set and application scenario requirements corresponding to the structured data set includes: Using standardized interfaces to obtain real-time data from different databases, and converting the format of the real-time data to obtain a structured data set; For structured data sets corresponding to different databases, obtain the application scenario requirements corresponding to each structured data set.

4. The method according to any one of claims 1 to 3, characterized in that: The method further comprises: After obtaining the comparison results, determining whether the comparison results are consistent; If yes, then the structured data set is compared again at a preset interval, and the comparison result is obtained and overwritten with the previous comparison result to verify the accuracy of the comparison; If not, an alarm is generated to remind the user to verify the comparison result.

5. A structured data comparison device, characterized in that: The device comprises: An acquisition module, used to acquire a structured data set and application scenario requirements corresponding to the structured data set, wherein the structured data set includes first structured data and second structured data, wherein the first structured data is reference source data and the second structured data is changed data; A parsing module, used to obtain a field comparison rule based on the application scenario requirements, and parse the structured data set based on the field comparison rule to obtain a comparison item number and a sorting field; A grouping module, used for grouping the structured data set based on the comparison item number to obtain at least one group of comparison data; A comparison module, for sorting the comparison data based on the sorting field for each group of comparison data, and comparing them in sequence to obtain comparison results, wherein the comparison results include results of consistent data and results of inconsistent data; The parsing module includes an acquisition unit and a preprocessing unit; The acquisition unit is used to acquire the corresponding comparison field in the field comparison rule, wherein the comparison field includes a type field, a function field and a number field, wherein the type field is used to mark the type of field comparison, and the type includes a type used to indicate whether the field is to be compared; The preprocessing unit is used to obtain corresponding data in the structured data set based on the comparison field, Identify the character strings to be compared in the data based on the type field, and concatenate the character strings to be compared; the character strings to be compared include a primary key character string and a business character string; The concatenated strings to be compared are processed based on the function field to obtain a comparison string; Assembling the comparison string using the number field, and identifying the comparison item number and sort field corresponding to the assembled comparison string; The comparison module includes a judgment unit, a comparison unit and a determination unit; The judging unit is used to obtain the number of data items in each set of comparison data, and judge whether the number of data items is greater than 1; The comparison unit is used to sort the comparison data based on the sorting field when the number of the data is greater than 1, and compare them in sequence to obtain a comparison result; The determination unit is used to, when the number of data items is equal to 1, determine that the comparison data is deleted data if the comparison data comes from the first structured data; and determine that the comparison data is newly added data if the comparison data comes from the second structured data.

6. An electronic device, characterized in that: include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory to implement the method according to any one of claims 1 to 4.

7. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are executed by a processor, they are used to implement the method according to any one of claims 1 to 4.

8. A computer program product, characterized in that The computer program comprises program codes, and when a computer runs the computer program, the program codes execute the method according to any one of claims 1 to 4.

Images