Data reconciliation method and device

Abstract

The invention discloses a data reconciliation method and device, and the method comprises the steps: calling a preset SDK to carry out the storage processing of business data according to a preset data protocol when the business data is generated, and obtaining business protocol data; reporting the service protocol data to a reconciliation big table of a data platform based on a preset container data processing interface; according to the reconciliation configuration information, reading each service protocol data in the reconciliation big table, and performing reconciliation processing according to the reconciliation field mapping relation to obtain a reconciliation result; and if account checking failure exists in the account checking result, recall alarm processing is carried out on the business protocol data with account checking failure. The business data is obtained through the preset SDK, the limitation of the existing account checking mode on the data source can be avoided, and the multi-form data source is supported. And storing the service protocol data according to the preset protocol and storing the service protocol data in the reconciliation big table, configuring reconciliation configuration information based on the preset data protocol to perform reconciliation, and when reconciliation fails, performing recall alarm, thereby facilitating positioning of the service protocol data which fails in reconciliation.

Description

Technical Field

[0001] The present application relates to the technical field of account reconciliation, and in particular to a data reconciliation method and device. Background Art

[0002] Reconciliation is to check the accounts, and to examine and check the relevant data to find any mismatched data. Most existing reconciliations are based on a certain API interface query to obtain data, and then match them one by one. This processing method can only process a limited amount of data, and can only match a few hundred or a few thousand at a time, and cannot handle the global massive trillion-level reconciliation. And when the amount of data increases, the real-time performance of the existing reconciliation method is poor, and the delay will increase to the minute level. Furthermore, the data sources supported by the existing reconciliation method generally only include database databases, which cannot support data sources of different forms, and also leads to limited reconciliation data. Summary of the invention

[0003] In view of the above problems, embodiments of the present application are proposed to provide a data reconciliation method and device that overcome the above problems or at least partially solve the above problems.

[0004] According to a first aspect of an embodiment of the present application, a data reconciliation method is provided, which includes:

[0005] When business data is generated, the preset SDK is called to store and process the business data according to the preset data protocol to obtain business protocol data;

[0006] Report business agreement data to the reconciliation table of the data platform based on the preset container data processing interface;

[0007] According to the reconciliation configuration information, read the business agreement data in the reconciliation table, perform reconciliation processing according to the reconciliation field mapping relationship, and obtain the reconciliation result; the reconciliation configuration information is configured according to the preset data protocol;

[0008] If there is a reconciliation failure in the reconciliation result, the business agreement data that failed the reconciliation will be recalled and an alarm will be issued.

[0009] Optionally, when the business data is generated, calling a preset SDK to store and process the business data according to a preset data protocol, and obtaining the business protocol data further includes:

[0010] When business data is generated in the business system, the storage component of the preset SDK is called to locally store the business data according to the preset data protocol to obtain business protocol data; wherein the business protocol data includes business identification, scenario identification, resource identification, external resource identification, creation time, processing links and / or business data; business data includes data fields and data values.

[0011] Optionally, reporting the business agreement data to the reconciliation table of the data platform based on the preset container data processing interface further includes:

[0012] Based on the preset container data processing interface, the locally stored business protocol data is read, and the read business protocol data of at least one domain is reported to the preset message queue. The reported business protocol data is aggregated by the big data real-time processing platform and then written to the reconciliation table of the data platform.

[0013] Optionally, reading each business agreement data in the reconciliation table according to the reconciliation configuration information, performing reconciliation processing according to the reconciliation field mapping relationship, and obtaining the reconciliation result further includes:

[0014] Obtaining reconciliation configuration information based on the configuration instantiation interface; the reconciliation configuration information includes reconciliation frequency information, reconciliation cycle information, reconciliation business identifier and / or reconciliation scenario identifier;

[0015] According to the reconciliation configuration information, read the corresponding business agreement data in the reconciliation table;

[0016] The read business agreement data is reconciled according to the reconciliation field mapping relationship; the reconciliation field mapping relationship represents the mapping relationship between each data field in the business agreement data of the reconciliation parties.

[0017] Optionally, according to the reconciliation configuration information, reading the corresponding business agreement data in the reconciliation table further includes:

[0018] Determine the reconciliation data reading mode according to the reconciliation cycle information of the reconciliation configuration information; the reconciliation data reading mode includes reading the reconciliation table or asynchronously reading other storage media;

[0019] According to the reconciliation data reading mode, the business agreement data is read for reconciliation.

[0020] Optionally, the recall alarm processing of the business agreement data that failed to be reconciled further includes:

[0021] Recalling and storing the business agreement data that failed to be reconciled, so as to locate the business agreement data that failed to be reconciled;

[0022] Report business agreement data that failed reconciliation and issue an alarm notification.

[0023] Optionally, the method further comprises:

[0024] Obtain the logical log of the database based on the preset data subscription component, and obtain the business data stored in the database according to the logical log;

[0025] The acquired business data of the database is reported to the preset message queue, and the reported business agreement data is aggregated by the big data real-time processing platform and then written into the reconciliation table of the data platform.

[0026] According to a second aspect of an embodiment of the present application, a data reconciliation device is provided, comprising:

[0027] The data storage module is adapted to call the preset SDK to store and process the business data according to the preset data protocol when the business data is generated, so as to obtain the business protocol data;

[0028] The data reporting module is suitable for reporting the business agreement data to the reconciliation table of the data platform based on the preset container data processing interface;

[0029] The reconciliation module is suitable for reading the business agreement data in the reconciliation table according to the reconciliation configuration information, performing reconciliation processing according to the reconciliation field mapping relationship, and obtaining the reconciliation result; the reconciliation configuration information is configured according to the preset data protocol;

[0030] The recall alarm module is suitable for recalling the business agreement data of the failed reconciliation if there is a reconciliation failure in the reconciliation result.

[0031] According to a third aspect of an embodiment of the present application, there is provided a computing device, comprising: a processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface communicate with each other via the communication bus;

[0032] The memory is used to store at least one executable instruction, and the executable instruction enables the processor to execute operations corresponding to the above-mentioned data reconciliation method.

[0033] According to a fourth aspect of an embodiment of the present application, a computer storage medium is provided, wherein at least one executable instruction is stored in the storage medium, and the executable instruction enables a processor to perform operations corresponding to the above-mentioned data reconciliation method.

[0034] According to a fifth aspect of an embodiment of the present application, a computer program product is provided, comprising at least one executable instruction, wherein the executable instruction enables a processor to perform operations corresponding to the above-mentioned data reconciliation method.

[0035] According to the data reconciliation method and device provided by the present application, business data is obtained by preset SDK, which is not restricted by the existing reconciliation method on the data source and supports multiple forms of data sources. Business protocol data is obtained according to the preset protocol and stored in the reconciliation table, which is convenient for using the reconciliation table and performing flexible reconciliation based on the preset data protocol configuration reconciliation configuration information. When reconciliation fails, a recall alarm can be issued to facilitate the location and verification of the business protocol data that failed the reconciliation.

[0036] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present application. Also, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings:

[0038] Figure 1 A flow chart of a data reconciliation method according to an embodiment of the present application is shown;

[0039] Figure 2 A flow chart of a data reconciliation method according to another embodiment of the present application is shown;

[0040] Figure 3 It shows a schematic diagram of the process of obtaining business data using the preset SDK and reporting business agreement data to the reconciliation table;

[0041] Figure 4 A schematic diagram of the structure of a data reconciliation device according to an embodiment of the present application is shown;

[0042] Figure 5 A schematic diagram of the structure of a computing device according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0043] The exemplary embodiments of the present application will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present application are shown in the accompanying drawings, it should be understood that the present application can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided in order to enable a more thorough understanding of the present application and to fully convey the scope of the present application to those skilled in the art.

[0044] First, the terms involved in one or more embodiments of the present application are explained.

[0045] Reconciliation: Data entities are matched through identification.

[0046] Full-link reconciliation: Physically dispersed data is collected globally and reconciled based on business, scenarios, etc.

[0047] Figure 1 A flow chart of a data reconciliation method according to an embodiment of the present application is shown. Figure 1As shown, the method comprises the following steps:

[0048] Step S101, when business data is generated, the preset SDK is called to store and process the business data according to the preset data protocol to obtain business protocol data.

[0049] During the execution of the business system call, business data will be generated. In this embodiment, when the business system generates business data, the business system can store the business data generated by the business system by calling the preset SDK (Software Development Kit). The preset SDK can be configured in the same container as the business system. The preset SDK is transparent and imperceptible to the business system and will not affect the business system. The business system executes normally. When generating business data, the preset SDK can be called. After calling, the preset SDK can store and process the business data according to the preset data protocol to obtain business protocol data. The preset data protocol defines the various data included in the business protocol data. In addition to the business data, the business protocol data also includes business identification, scene identification, resource identification (domain identification of the business data), external resource identification (bound to the data in the business data), creation time, processing links (different data processing links, such as creation, transcoding, review and other different processing links for manuscripts), etc. Business data can be directly obtained from the business system, and the business data includes data fields and data values. Other identifiers other than business data are constructed based on business data, the creation time can be the current time, and the processing link is determined according to the processing link that the business data is currently in. The above is an example, and the specific setting depends on the implementation situation, and is not limited here.

[0050] When business data is generated, the preset SDK can be used to obtain business data in real time. There is no need to limit the source of reconciliation data to data in the database. Different data sources can be connected to provide various data, making data acquisition more convenient.

[0051] Step S102: reporting the business agreement data to the reconciliation table of the data platform based on the preset container data processing interface.

[0052] After obtaining the business protocol data, the business protocol data can be reported to the reconciliation table of the data platform based on the preset container processing interface, such as the recordio data processing interface of the container, so that the business protocol data can be obtained from the reconciliation table in real time for reconciliation. Based on the preset container data processing interface, the business protocol data of different domains can be reported to the data platform. In order to improve the real-time performance of data transmission, the data platform adopts a real-time processing platform such as the berserker-bsql big data, whose delay is at the millisecond level to improve processing efficiency. The business protocol data can be aggregated and processed on the berserker-bsql data processing platform, and the aggregated business protocol data can be written to the reconciliation table, such as the reconciliation table of the clickhouse platform. The business protocol data stored in the reconciliation table are all in the format of the preset data protocol, which is convenient for subsequent data search and reconciliation. The clickhouse platform is a high-performance nosql platform. Its column storage and efficient data compression algorithm can improve the efficiency of processing large-scale data, and the query response time can be completed at the millisecond level.

[0053] Step S103, reading each business agreement data in the reconciliation table according to the reconciliation configuration information, performing reconciliation processing according to the reconciliation field mapping relationship, and obtaining a reconciliation result.

[0054] The reconciliation configuration information can be configured according to the preset data protocol. For example, the storage format of the business protocol data is defined in the preset data protocol, including the business identifier, the scenario identifier, etc. The reconciliation configuration information can be configured according to the preset data protocol. The reconciliation configuration information can specify reconciliation based on the business identifier, reconciliation based on the scenario identifier, etc. According to the reconciliation configuration information, the corresponding business protocol data can be read from the reconciliation table for reconciliation. During reconciliation, the business data included in the business protocol data can be business data A generated when creating a manuscript, or business data B generated when reviewing a manuscript. Business data A and business data B may have different data fields. Therefore, when storing the business protocol data, it is also necessary to store the reconciliation field mapping relationship and store the mapping relationship of the data fields of different business data, such as storing the mapping relationship between the data field A1 in the business data A and the data field B2 in the business data B. When reconciling, the data value corresponding to the data field A1 can be reconciled with the data value corresponding to the data field B2 to determine whether the reconciliation of business data A and business data B is successful. The reconciliation field mapping relationship can be set in advance according to the business data, and stored together when the business data is stored, or, when reporting to the data platform, the corresponding storage reconciliation field mapping relationship can be stored according to the stored business data. The reconciliation field mapping relationship is set according to the specific business, and when stored, it is stored in the business agreement data together with the corresponding business data. The above is an example, which is set according to the implementation situation and is not limited here.

[0055] The business agreement data of the parties to the reconciliation is obtained from the reconciliation table. According to the reconciliation field mapping relationship, the data fields of the business data included in each business agreement data are parsed, such as by parsing through the Fields_mapping field, to obtain each data field and the corresponding data value. The data fields of the parties to be compared are determined according to the reconciliation field mapping relationship, and the data values ​​of the corresponding data fields are compared. For example, sql join can be used to compare to determine whether the data values ​​of each data field are consistent, and the reconciliation result is obtained.

[0056] Step S104: If there is a reconciliation failure in the reconciliation result, the business agreement data of the reconciliation failure is recalled and an alarm is issued.

[0057] The reconciliation result includes, for example, a successful reconciliation, that is, the data values ​​of the data fields that need to be compared of the business data contained in the business agreement data of the parties to the reconciliation are consistent, or the reconciliation result is a reconciliation failure, that is, there is an inconsistency in the data values ​​of the data fields that need to be compared. At this time, an alarm can be processed for the reconciliation, such as sending an alarm notification to the monitoring platform to inform that the business data has failed to reconcile, etc. Further, only the alarm can only obtain the existence of a reconciliation failure, and it is impossible to accurately know which business data has failed to reconcile. Therefore, this embodiment, while issuing an alarm, also recalls the business protocol data that failed to reconcile, such as counting the amount of data that failed to reconcile (the count number of data that failed to reconcile), the data fields and data values ​​of the business data that failed to reconcile, etc., and stores and reports the above data that has been recalled, so as to facilitate problem location based on the recalled business protocol data that failed to reconcile, and timely discover problems with the business, etc.

[0058] According to the data reconciliation method provided by this application, business data is obtained through the preset SDK, which is not restricted by the existing reconciliation method on the data source and supports multiple forms of data sources. Business protocol data is obtained according to the preset protocol and stored in the reconciliation table, which is convenient for using the reconciliation table and performing flexible reconciliation based on the preset data protocol configuration reconciliation configuration information. When reconciliation fails, a recall alarm can be issued to facilitate the location and verification of the business protocol data that failed to reconcile.

[0059] Figure 2 A flow chart of a data reconciliation method according to an embodiment of the present application is shown. Figure 2 As shown, the method comprises the following steps:

[0060] Step S201, when business data is generated in the business system, the storage component of the preset SDK is called to locally store the business data according to the preset data protocol to obtain business protocol data.

[0061] In this embodiment, when the business system generates business data, a preset SDK is used to store the business data. Specifically, Figure 3 As shown, the application of the business system is deployed in the application container. When the business system is executed, such as executing the main function, and generating business data, by calling the storage component of the preset SDK, it can be processed according to the preset data protocol, and the business data is first stored locally to obtain the business protocol data. When obtaining the business data, what is obtained is the data field of the business data and the corresponding data value. The data field and the data value can be composed into a json string in a one-to-one correspondence, and stored as the overall business data. When reconciling accounts, after obtaining the business data, the json string is parsed to obtain each data field and the corresponding data value for reconciliation. The above is an example, which is set according to the implementation situation and is not limited here.

[0062] Here, the business protocol data is first stored locally on the physical machine, which has a large storage space and does not generate reporting traffic, and will not affect the normal execution of the business system.

[0063] The preset data protocol can be as follows:

[0064] Business_id--Business ID

[0065] Scene_id--scene id

[0066] Resource_id--resource id (ID within the current business domain)

[0067] External_Resource_id--External resource ID (used to bind with the data id in the current business)

[0068] Ctime--creation time

[0069] Operation_step--processing steps (such as creating manuscripts, transcoding, and reviewing)

[0070] Original_data – Business data (Business data here stores various business data of different business systems. Taking manuscripts as an example, it includes manuscript creation business data, manuscript review business data, etc.)

[0071] Fields_mapping--Reconciliation field mapping relationship (such as the field correspondence between creating business data and reviewing business data, such as: the cover field of creating business data corresponds to the cover2 field of reviewing business data, etc.)

[0072] The above is an example, and the specific preset data protocol can be set according to the implementation situation and is not limited here.

[0073] Based on the preset SDK, the business data storage is obtained to obtain the business agreement data guarantee, which can obtain business data in real time and facilitate real-time reconciliation.

[0074] Step S202, based on the preset container data processing interface, read the locally stored business protocol data, report the read business protocol data of at least one domain to the preset message queue, aggregate the reported business protocol data through the big data real-time processing platform, and write it to the reconciliation table of the data platform.

[0075] The deployment of business systems involves many physical machines, and the reporting of multiple locally stored business protocol data involves cross-domain reporting. To facilitate reporting, this embodiment reports based on a preset container data processing interface, such as Figure 3 As shown, a preset container data processing interface is set in the container machine, such as a reading interface, which reads the business protocol data stored locally and reports it to the data platform based on the log agent. To ensure the real-time performance of data processing, after the preset container data processing interface reads the business protocol data stored locally, it can report the business protocol data of at least one domain read to the preset message queue. The preset message queue can adopt a high-performance message queue such as kafka-proxy, which has high throughput and can process a large amount of data per second. It can also communicate with Figure 3 The big data real-time processing platform in the data platform can be integrated to reduce latency. The big data real-time processing platform can use berserker-bsql to read the business agreement data in kafka-proxy, aggregate it, and write it to the reconciliation table of the data platform. The aggregation process can check the business agreement data for compliance and other aspects to ensure that the business agreement data stored in the reconciliation table can be accurately reconciled. Figure 3 , aggregate the business protocol data in the preset message queue and send it to the big data real-time processing platform, which is not limited here. The reconciliation table can use a large reconciliation table such as ClickHouse, which is based on column storage and efficient data compression algorithms to improve the efficiency of reconciliation processing.

[0076] The business agreement data stored in the reconciliation table is highly real-time, which facilitates real-time reconciliation and timely detection of problems in business data.

[0077] Step S203, based on the preset data subscription component, obtain the logical log of the database, and obtain the business data stored in the database according to the logical log; report the obtained business data of the database to the preset message queue, aggregate the reported business protocol data through the big data real-time processing platform, and write it to the reconciliation table of the data platform.

[0078] In addition to obtaining business protocol data based on the preset SDK, this embodiment can also obtain logical log binlogs in different databases through preset data subscription components, such as subscribing to binlogs in different business databases by preset data subscription components, and obtaining business data in each business database based on subscription tasks. Through the distributed log transmission system, the business data in each business database can be reported to the preset message queue, and the reported business protocol data is aggregated by the big data real-time processing platform and written to the reconciliation table of the data platform. The above link can refer to the description of step S202, which will not be repeated here.

[0079] Furthermore, here, after the preset data subscription component obtains the business data of each business database, it is also necessary to convert and process each business data according to the preset data protocol to ensure that when it is subsequently written into the reconciliation table, it is consistent with the business protocol data format obtained by the preset SDK, which is convenient for reconciliation. During the conversion process, it can be processed based on the preset data protocol, and the reconciliation field mapping relationship can also be written during the conversion to determine the mapping relationship between the reconciliation fields during reconciliation, which is convenient for reconciliation comparison. This is set here according to the implementation situation.

[0080] Step S204, obtaining reconciliation configuration information based on the configuration instantiation interface.

[0081] When reconciling accounts, you can set up automatic reconciliation processing through reconciliation configuration information. Specifically, if you provide a configuration instantiation interface, you can flexibly configure the reconciliation configuration information. The reconciliation configuration information includes reconciliation frequency information, reconciliation cycle information, reconciliation business identifier, reconciliation scenario identifier, etc. The reconciliation frequency information can set the reconciliation execution frequency, such as multiple reconciliations per day, etc., and you can set timing information, such as reconciliation at 8 o'clock every day, reconciliation at 12 o'clock and 24 o'clock every day, etc., and automatically perform reconciliation processing according to the timing information; the reconciliation cycle information can set the time span of the acquired data, such as acquiring business agreement data within a time period of 1 day, 3 days, 1 week, etc. for reconciliation, etc., which can realize data reconciliation of different period spans to meet different reconciliation requirements. The reconciliation business identifier and the reconciliation scenario identifier can specify specific identifiers based on the business or scenario for reconciliation, that is, specify the reconciliation object. When reconciling, obtain the business agreement data corresponding to the reconciliation business identifier or the reconciliation scenario identifier for reconciliation.

[0082] The reconciliation configuration information can be flexibly set through the configuration instantiation interface, or it can be pre-set and imported through the configuration instantiation interface to obtain the reconciliation configuration information, which is not limited here.

[0083] Step S205 , reading the corresponding business agreement data in the reconciliation table according to the reconciliation configuration information, and performing reconciliation processing on the read business agreement data according to the reconciliation field mapping relationship.

[0084] Reconciliation can be performed regularly according to the reconciliation frequency information in the reconciliation configuration information. The corresponding business agreement data in the reconciliation table can be read according to the reconciliation business identifier and the reconciliation scenario identifier in the reconciliation configuration information. The read business agreement data is reconciled according to the corresponding data fields according to the reconciliation field mapping relationship to obtain the reconciliation result. If the reconciliation frequency information is to be executed at 12 o'clock every day, and the reconciliation business identifiers are business identifier 1 and business identifier 2, then the scheduled task can be started at 12 o'clock every day to read the business agreement data of business identifier 1 and business identifier 2 in the reconciliation table. First, the business data in each business agreement data is parsed to obtain each data field and the corresponding data value. According to the mapping relationship of the reconciliation fields in the business agreement data of business identifier 1 and business identifier 2, the reconciliation is performed separately. For example, if the mapping relationship of the reconciliation fields is that the cover field of the creation business data corresponds to the cover2 field of the review business data, then the data values ​​of the cover field and the cover2 field are reconciled. When reconciling, a join statement such as SQL can be used to determine whether the data values ​​of the two fields are consistent. If not, the query cannot be found, and it is determined that the reconciliation has failed. The above is an example, which is set according to the implementation situation and is not limited here.

[0085] Further, according to the reconciliation cycle information of the reconciliation configuration information, the scope of data reading can be determined. For example, when the interval between the previous processing link and the next processing link in the business data processing link is long, such as for some delayed manuscripts, the time interval between the two processing links of manuscript review and manuscript development is long, the business agreement data stored in the reconciliation table will be archived after the reconciliation is completed or after a period of time, and part of the business agreement data will be stored in other storage media, such as low-cost storage media, and the part of the business agreement data will be deleted from the reconciliation table. According to the reconciliation cycle information, a cycle threshold can be set. If it is greater than the cycle threshold, the reconciliation data reading mode can adopt asynchronous reading of other storage media, that is, part of the data is read from the reconciliation table, and the other part of the data needs to start an asynchronous job due to archiving, and read from other storage media, and then reconcile the read business agreement data. If it is not greater than the cycle threshold, the reconciliation data reading mode is to read the reconciliation table, and the business agreement data of all parties to the reconciliation are read from the reconciliation table for reconciliation. The cycle threshold is set according to the archiving time limit and is not limited here.

[0086] Determine different reconciliation data reading modes. You can use different reconciliation reading modes to obtain business agreement data and then perform reconciliation. This is convenient for coping with reconciliation in different business situations. It is suitable for both real-time business data reconciliation and business data reconciliation with a longer rolling time period.

[0087] Step S206, recalling and storing the business protocol data that failed to be reconciled, so as to locate the business protocol data that failed to be reconciled, reporting the business protocol data that failed to be reconciled, and issuing an alarm notification.

[0088] In the reconciliation process, according to the reconciliation result, if the reconciliation result is successful, that is, the data values ​​of the various business data contained in the business agreement data of the reconciliation parties are consistent during the reconciliation, no subsequent processing is required. If the reconciliation result is a reconciliation failure, in order to facilitate the timely positioning of the business data that failed to reconcile, the business agreement data that failed to reconcile can be recalled. During the recall process, the amount of data that failed to reconcile (the total amount of business agreement data that failed to reconcile), the data fields and data values ​​of the business data that failed to reconcile, etc. can be counted to obtain the reconciliation failure record, and the recalled reconciliation failure record can be stored and reported, and an alarm is issued to notify the reconciliation failure, so as to timely locate the problem based on the reconciliation failure record and determine the problems in the business processing.

[0089] According to the data reconciliation method provided by the present application, the business data generated by the business system is obtained in real time through the storage component of the preset SDK, stored locally on the physical machine, and the business protocol data stored locally is uniformly read by the preset container data processing interface, and the business protocol data of different domains are reported to the preset message queue. The reported business protocol data can be aggregated in real time with low latency through the big data real-time processing platform and written to the reconciliation table of the data platform, and reconciliation is performed directly based on the reconciliation table. The acquisition of business protocol data in the present application is not subject to the restrictions of existing reconciliation methods on data sources, and supports multi-form data sources. Reconciliation is more convenient by using the reconciliation table to query and reconcile, such as SQL statements. For business protocol data that fail to be reconciled, a recall alarm is issued to facilitate problem location of the business protocol data that failed to be reconciled, and timely discovery of business problems, etc.

[0090] Figure 4 FIG. 1 shows a schematic diagram of the structure of a data reconciliation device provided in an embodiment of the present application. Figure 4 As shown, the device comprises:

[0091] The data storage module 410 is adapted to call the preset SDK to store and process the business data according to the preset data protocol when the business data is generated, so as to obtain the business protocol data;

[0092] The data reporting module 420 is adapted to report the business agreement data to the reconciliation table of the data platform based on the preset container data processing interface;

[0093] The reconciliation module 430 is adapted to read the business protocol data in the reconciliation table according to the reconciliation configuration information, perform reconciliation processing according to the reconciliation field mapping relationship, and obtain the reconciliation result; the reconciliation configuration information is configured according to the preset data protocol;

[0094] The recall alarm module 440 is adapted to perform a recall alarm process on the business agreement data of the failed reconciliation if there is a reconciliation failure in the reconciliation result.

[0095] Optionally, the data storage module 410 is further adapted to:

[0096] When business data is generated in the business system, the storage component of the preset SDK is called to locally store the business data according to the preset data protocol to obtain business protocol data; wherein the business protocol data includes business identification, scenario identification, resource identification, external resource identification, creation time, processing links and / or business data; business data includes data fields and data values.

[0097] Optionally, the data reporting module 420 is further adapted to:

[0098] Based on the preset container data processing interface, the locally stored business protocol data is read, and the read business protocol data of at least one domain is reported to the preset message queue. The reported business protocol data is aggregated by the big data real-time processing platform and then written to the reconciliation table of the data platform.

[0099] Optionally, the reconciliation module 430 is further adapted to:

[0100] Obtaining reconciliation configuration information based on the configuration instantiation interface; the reconciliation configuration information includes reconciliation frequency information, reconciliation cycle information, reconciliation business identifier and / or reconciliation scenario identifier;

[0101] According to the reconciliation configuration information, read the corresponding business agreement data in the reconciliation table;

[0102] The read business agreement data is reconciled according to the reconciliation field mapping relationship; the reconciliation field mapping relationship represents the mapping relationship between each data field in the business agreement data of the reconciliation parties.

[0103] Optionally, the reconciliation module 430 is further adapted to:

[0104] Determine the reconciliation data reading mode according to the reconciliation cycle information of the reconciliation configuration information; the reconciliation data reading mode includes reading the reconciliation table or asynchronously reading other storage media;

[0105] According to the reconciliation data reading mode, the business agreement data is read for reconciliation.

[0106] Optionally, the recall alert module 440 is further adapted to:

[0107] Recalling and storing the business agreement data that failed to be reconciled, so as to locate the business agreement data that failed to be reconciled;

[0108] Report business agreement data that failed reconciliation and issue an alarm notification.

[0109] Optionally, the device also includes: a database reporting module 450, which is suitable for obtaining the logical log of the database based on a preset data subscription component, and obtaining the business data stored in the database according to the logical log; reporting the acquired business data of the database to a preset message queue, aggregating the reported business protocol data via the big data real-time processing platform, and writing it to the reconciliation table of the data platform.

[0110] The description of each module above refers to the corresponding description in the method embodiment and will not be repeated here.

[0111] According to the data reconciliation device provided by the present application, business data is obtained through the preset SDK, which is not restricted by the existing reconciliation method on the data source and supports multiple forms of data sources. Business protocol data is obtained according to the preset protocol and stored in the reconciliation table, which is convenient for using the reconciliation table and performing flexible reconciliation based on the preset data protocol configuration reconciliation configuration information. When the reconciliation fails, a recall alarm can be issued to facilitate the location and verification of the business protocol data that failed the reconciliation.

[0112] The present application also provides a non-volatile computer storage medium, which stores at least one executable instruction, and the executable instruction can execute the operation corresponding to the data reconciliation method in any of the above method embodiments.

[0113] The present application also provides a computer program product, which includes at least one executable instruction or computer program, and the executable instruction or computer program can enable a processor to perform operations corresponding to the data reconciliation method in any of the above method embodiments.

[0114] Figure 5 A schematic diagram of the structure of a computing device according to an embodiment of the present application is shown. The specific embodiment of the present application does not limit the specific implementation of the computing device.

[0115] like Figure 5 As shown, the computing device may include: a processor (processor) 502 , a communications interface (Communications Interface) 504 , a memory (memory) 506 , and a communication bus 508 .

[0116] in:

[0117] The processor 502 , the communication interface 504 , and the memory 506 communicate with each other via a communication bus 508 .

[0118] The communication interface 504 is used to communicate with other devices such as clients or other servers.

[0119] The processor 502 is used to execute the program 510, and specifically can execute the relevant steps in the above-mentioned data reconciliation method embodiment.

[0120] Specifically, the program 510 may include program codes, which include computer operation instructions.

[0121] The processor 502 may be a central processing unit (CPU), or an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the present application. The one or more processors included in the computing device may be processors of the same type, such as one or more CPUs; or processors of different types, such as one or more CPUs and one or more ASICs.

[0122] The memory 506 is used to store the program 510. The memory 506 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0123] The program 510 can be specifically used to enable the processor 502 to execute the data reconciliation method in any of the above-mentioned method embodiments. The specific implementation of each step in the program 510 can refer to the corresponding descriptions in the corresponding steps and units in the above-mentioned data reconciliation embodiments, which will not be repeated here. Those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working process of the above-described devices and modules can refer to the corresponding process description in the above-mentioned method embodiments, which will not be repeated here.

[0124] The algorithm or display provided here is not inherently related to any specific computer, virtual system or other device. Various general systems can also be used together with the teaching based on this. According to the above description, it is obvious to construct the structure required for this type of system. In addition, the application is not directed to any specific programming language either. It should be understood that various programming languages ​​can be utilized to realize the content of the application described here, and the above description of specific languages ​​is to disclose the preferred embodiment of the application.

[0125] In the description provided herein, a large number of specific details are described. However, it is understood that the embodiments of the present application can be practiced without these specific details. In some instances, well-known methods, structures and techniques are not shown in detail so as not to obscure the understanding of this description.

[0126] Similarly, it should be understood that in order to streamline the present application and help understand one or more of the various inventive aspects, in the above description of the exemplary embodiments of the present application, the various features of the present application are sometimes grouped together into a single embodiment, figure, or description thereof. However, the disclosed method should not be interpreted as reflecting the following intention: the claimed application requires more features than the features clearly stated in each claim. More specifically, as reflected in the claims below, the inventive aspects are less than all the features of the single embodiment disclosed above. Therefore, the claims following the specific embodiment are hereby expressly incorporated into the specific embodiment, wherein each claim itself serves as a separate embodiment of the present application.

[0127] Those skilled in the art will appreciate that the modules in the devices in the embodiments may be adaptively changed and arranged in one or more devices different from the embodiments. The modules or units or components in the embodiments may be combined into one module or unit or component, and in addition they may be divided into a plurality of submodules or subunits or subcomponents. Except that at least some of such features and / or processes or units are mutually exclusive, all features disclosed in this specification (including the accompanying claims, abstracts and drawings) and all processes or units of any method or device disclosed in this manner may be combined in any combination. Unless otherwise expressly stated, each feature disclosed in this specification (including the accompanying claims, abstracts and drawings) may be replaced by an alternative feature providing the same, equivalent or similar purpose.

[0128] In addition, those skilled in the art will appreciate that, although some embodiments herein include certain features included in other embodiments but not other features, the combination of features of different embodiments is meant to be within the scope of the present application and form different embodiments. For example, in the claims below, any one of the claimed embodiments may be used in any combination.

[0129] The various component embodiments of the present application can be implemented in hardware, or in software modules running on one or more processors, or in a combination thereof. It should be understood by those skilled in the art that a microprocessor or digital signal processor (DSP) can be used in practice to implement some or all functions of some or all components of the present application. The present application can also be implemented as a device or apparatus program (e.g., computer program and computer program product) for executing a part or all of the methods described herein. Such a program implementing the present application can be stored on a computer-readable medium, or can have the form of one or more signals. Such a signal can be downloaded from an Internet website, or provided on a carrier signal, or provided in any other form.

[0130] It should be noted that the above embodiments illustrate the present application rather than limit the present application, and that those skilled in the art may design alternative embodiments without departing from the scope of the appended claims. In the claims, any reference symbol between brackets shall not be constructed as a limitation on the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "one" or "an" preceding an element does not exclude the presence of multiple such elements. The present application may be implemented by means of hardware including several different elements and by means of a suitably programmed computer. In a unit claim that lists several devices, several of these devices may be embodied by the same hardware item. The use of the words first, second, and third, etc. does not indicate any order. These words may be interpreted as names. The steps in the above embodiments, unless otherwise specified, should not be understood as limitations on the order of execution.

Claims

1. A data reconciliation method, comprising: When business data is generated, the preset SDK is called to store and process the business data according to the preset data protocol to obtain business protocol data; Reporting the business agreement data to the reconciliation table of the data platform based on a preset container data processing interface; According to the reconciliation configuration information, the business agreement data in the reconciliation table is read, and the reconciliation processing is performed according to the reconciliation field mapping relationship to obtain the reconciliation result; the reconciliation configuration information is configured according to the preset data protocol; If there is a reconciliation failure in the reconciliation result, the business agreement data that failed the reconciliation will be recalled and an alarm will be issued.

2. The method according to claim 1, wherein: When the business data is generated, calling the preset SDK to store and process the business data according to the preset data protocol, and obtaining the business protocol data further includes: When business data is generated in the business system, the storage component of the preset SDK is called to locally store the business data according to the preset data protocol to obtain business protocol data; wherein the business protocol data includes business identification, scenario identification, resource identification, external resource identification, creation time, processing links and / or business data; the business data includes data fields and data values.

3. The method according to claim 1 or 2, wherein: The step of reporting the business agreement data to the reconciliation table of the data platform based on the preset container data processing interface further includes: Based on the preset container data processing interface, the locally stored business protocol data is read, and the read business protocol data of at least one domain is reported to the preset message queue. The reported business protocol data is aggregated by the big data real-time processing platform and then written to the reconciliation table of the data platform.

4. The method according to claim 1, wherein: The step of reading each business agreement data in the reconciliation table according to the reconciliation configuration information, performing reconciliation processing according to the reconciliation field mapping relationship, and obtaining the reconciliation result further includes: Acquiring reconciliation configuration information based on the configuration instantiation interface; the reconciliation configuration information includes reconciliation frequency information, reconciliation cycle information, reconciliation business identifier and / or reconciliation scenario identifier; According to the reconciliation configuration information, read the corresponding business agreement data in the reconciliation table; The read business agreement data is reconciled according to the reconciliation field mapping relationship; the reconciliation field mapping relationship represents the mapping relationship between the data fields in the business agreement data of the reconciliation parties.

5. The method according to claim 4, wherein: The step of reading the corresponding business agreement data in the reconciliation table according to the reconciliation configuration information further includes: Determine a reconciliation data reading mode according to the reconciliation cycle information of the reconciliation configuration information; the reconciliation data reading mode includes reading a large reconciliation table or asynchronously reading other storage media; According to the reconciliation data reading mode, the business agreement data is read and obtained for reconciliation.

6. The method according to any one of claims 1 to 5, wherein: The recall alarm processing of the business agreement data that failed to be reconciled further includes: Recalling and storing the business agreement data that failed to be reconciled, so as to locate the business agreement data that failed to be reconciled; Report the business agreement data of the reconciliation failure and issue an alarm notification.

7. The method according to any one of claims 1 to 6, wherein: The method further comprises: Obtaining a logical log of a database based on a preset data subscription component, and obtaining business data stored in the database according to the logical log; The acquired business data of the database is reported to the preset message queue, and the reported business agreement data is aggregated by the big data real-time processing platform and then written into the reconciliation table of the data platform.

8. A data reconciliation device, comprising: The data storage module is adapted to call the preset SDK to store and process the business data according to the preset data protocol when the business data is generated, so as to obtain the business protocol data; A data reporting module, adapted to report the business agreement data to a reconciliation table of a data platform based on a preset container data processing interface; A reconciliation module, adapted to read each business protocol data in the reconciliation table according to the reconciliation configuration information, perform reconciliation processing according to the reconciliation field mapping relationship, and obtain a reconciliation result; the reconciliation configuration information is configured according to a preset data protocol; The recall alarm module is suitable for recalling the business agreement data of the failed reconciliation if there is a reconciliation failure in the reconciliation result.

9. A computing device comprising: A processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface communicate with each other via the communication bus; The memory is used to store at least one executable instruction, and the executable instruction enables the processor to perform an operation corresponding to the data reconciliation method according to any one of claims 1-7.

10. A computer storage medium, wherein at least one executable instruction is stored in the storage medium, and the executable instruction enables a processor to perform operations corresponding to the data reconciliation method according to any one of claims 1 to 7.

11. A computer program product, comprising at least one executable instruction, wherein the executable instruction enables a processor to perform operations corresponding to the data reconciliation method according to any one of claims 1 to 7.

Images