Data aggregation method and device, electronic equipment and storage medium

Abstract

The invention relates to a data aggregation method and device, electronic equipment and a storage medium. The data aggregation method comprises the steps that metadata and business data corresponding to at least one business system are obtained, and the business data and the metadata are stored in a front database; creating an aggregation table based on the metadata, determining to-be-reconciliated data in the business data based on the identification information corresponding to the business data, and generating a data reconciliation task and a data aggregation task based on the aggregation table and the to-be-reconciliated data; executing the data reconciliation task, and after it is determined that reconciliation of the to-be-reconciliated data corresponding to the data reconciliation task is successful, storing the to-be-reconciliated data to an intermediate database; and executing the data aggregation task, and aggregating the to-be-reconciliated data from the intermediate database to the business database, so that reconciliation processing can be performed on the to-be-reconciliated data in the business data after the business data is acquired, and aggregation processing is executed after reconciliation succeeds, thereby improving the accuracy of data aggregation.

Description

Technical Field

[0001] The present disclosure relates to the field of data processing technology, and in particular to a data aggregation method, device, electronic device, and storage medium. Background Art

[0002] Data aggregation is a crucial step in building a data platform. Its primary purpose is to break down the physical silos of enterprise data, forming a unified data center that provides the raw material for subsequent data sharing, usage, and value mining. Each business end (or business system) within an enterprise is a data source, generating a vast amount of data. The production and collection of this data must comply with relevant requirements for data security and privacy protection.

[0003] The existing data aggregation method mainly collects data from the data source and then directly transfers the collected data to the data aggregation platform. During the data aggregation process, the accuracy of the aggregated data is not considered. Summary of the Invention

[0004] In order to solve the above technical problems, the present disclosure provides a data aggregation method, device, electronic device and storage medium.

[0005] A first aspect of an embodiment of the present disclosure provides a data aggregation method, including:

[0006] Obtain metadata and business data corresponding to at least one business system, and store the business data and metadata in a front-end database;

[0007] Create an aggregation table based on metadata, determine the data to be reconciled in the business data based on the identification information corresponding to the business data, and generate scheduling tasks based on the aggregation table and the data to be reconciled. The scheduling tasks include data reconciliation tasks and data aggregation tasks.

[0008] Execute the data reconciliation task, and after confirming that the corresponding pending reconciliation data is successfully reconciled, store the pending reconciliation data in the intermediate database;

[0009] Execute data aggregation tasks to aggregate the data to be reconciled from the intermediate database to the business database.

[0010] A second aspect of an embodiment of the present disclosure provides a data aggregation device, including:

[0011] A data acquisition module is used to acquire metadata and business data corresponding to at least one business system, and store the business data and metadata in a front-end database;

[0012] A task generation module is used to create an aggregation table based on metadata, determine the data to be reconciled in the business data based on the identification information corresponding to the business data, and generate scheduling tasks based on the aggregation table and the data to be reconciled. The scheduling tasks include data reconciliation tasks and data aggregation tasks.

[0013] A first task execution module is configured to execute a data reconciliation task and, after determining that the to-be-reconciled data corresponding to the data reconciliation task has been reconciled successfully, store the to-be-reconciled data in an intermediate database;

[0014] The second task execution module is used to execute the data aggregation task and aggregate the data to be reconciled from the intermediate database to the business database.

[0015] A third aspect of the present disclosure provides an electronic device, including:

[0016] processor;

[0017] a memory for storing executable instructions;

[0018] The processor is used to read executable instructions from the memory and execute the executable instructions to implement the data aggregation method provided by the first aspect above.

[0019] A fourth aspect of an embodiment of the present disclosure provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the processor implements the data aggregation method provided by the first aspect above.

[0020] The technical solution provided by the embodiments of the present disclosure has the following advantages over the prior art:

[0021] The data aggregation method, apparatus, electronic device, and storage medium provided by the embodiments of the present disclosure can obtain metadata and business data corresponding to at least one business system, store the business data and metadata in a front-end database, create an aggregation table based on the metadata, and determine the data to be reconciled in the business data based on identification information corresponding to the business data. A scheduling task is generated based on the aggregation table and the data to be reconciled, wherein the scheduling task includes a data reconciliation task and a data aggregation task. The data reconciliation task is then executed. After determining that the data to be reconciled corresponding to the data reconciliation task has been successfully reconciled, the data to be reconciled is stored in an intermediate database. The data aggregation task is executed to aggregate the data to be reconciled from the intermediate database to the business database. In this way, a front-end database, an intermediate database, and a business database can be set up on a data aggregation platform, and different business data can be stored in different databases. After obtaining the business data, the data to be reconciled in the business data is reconciled. After the reconciliation is successful, the data to be reconciled is stored in the intermediate database. The data corresponding to the aggregation task is then obtained from the intermediate database and aggregated. This avoids directly transferring the acquired business data to the data aggregation platform, thereby improving the accuracy of the aggregated data corresponding to the aggregation processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0023] In order to more clearly illustrate the embodiments of the present disclosure or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0024] Figure 1 is a flow chart of a data aggregation method provided by an embodiment of the present disclosure;

[0025] Figure 2 This is a flowchart of a method for executing a data reconciliation task provided by an embodiment of the present disclosure;

[0026] Figure 3 This is a flow chart of a method for executing a data aggregation task provided by an embodiment of the present disclosure;

[0027] Figure 4 is a structural diagram of a data aggregation device provided by an embodiment of the present disclosure;

[0028] Figure 5 It is a structural diagram of an electronic device provided by an embodiment of the present disclosure. DETAILED DESCRIPTION

[0029] In order to more clearly understand the above-mentioned objectives, features and advantages of the present disclosure, the scheme of the present disclosure will be further described below. It should be noted that the embodiments of the present disclosure and the features therein can be combined with each other in the absence of conflict.

[0030] In the following description, many specific details are set forth to facilitate a full understanding of the present disclosure, but the present disclosure may also be implemented in other ways different from those described herein; it is obvious that the embodiments in the specification are only part of the embodiments of the present disclosure, rather than all of the embodiments.

[0031] It should be understood that the various steps described in the method embodiments of the present disclosure may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this respect.

[0032] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or device comprising the element.

[0033] It should be noted that the modifications of "one" and "multiple" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, they should be understood as "one or more".

[0034] Typically, existing data aggregation methods primarily collect data from data sources and then directly transfer the collected data to a data aggregation platform. During the data aggregation process, the accuracy of the aggregated data is not considered. To address this issue, the present disclosure provides a data aggregation method, which is described below with reference to specific embodiments.

[0035] Figure 1This is a flowchart of a data aggregation method provided by an embodiment of the present disclosure. The method can be performed by a data aggregation device. The data aggregation device can be implemented in software and / or hardware. The data aggregation device can be configured in an electronic device, such as a server or a terminal, wherein the terminal specifically includes a computer or a tablet computer.

[0036] like Figure 1 As shown, the data aggregation method provided in this embodiment includes the following steps.

[0037] S110 : Obtain metadata and business data corresponding to at least one business system, and store the business data and metadata in a front-end database.

[0038] In an embodiment of the present disclosure, the electronic device can receive business data pushed by at least one business system in real time, and then obtain business data corresponding to at least one business system, and at the same time obtain metadata corresponding to each business system in at least one business system from the data ledger system.

[0039] In the embodiment of the present disclosure, the electronic device can be understood as a data aggregation platform. The electronic device can communicate with each business system, and the business system pushes the real-time business data to the electronic device through the interface between the electronic device, so that the electronic device can obtain the business data corresponding to each business system.

[0040] The business system can be understood as the source of business data, that is, the system corresponding to the user terminal. Business data is the data generated by the user terminal when performing business processing.

[0041] Metadata may include metadata of business systems and metadata of business data.

[0042] The data ledger system is used to extract metadata corresponding to the business system, and to sort and inventory the business attributes, business data, metadata and other information corresponding to the business system.

[0043] The data aggregation platform can perform data reconciliation tasks, data aggregation tasks and real-time tasks. Among them, real-time tasks refer to data synchronization by collecting log information of business data.

[0044] In the embodiment of the present disclosure, the front-end database is used to store business data pushed by the business system and metadata corresponding to the business system. The front-end database is a table without a primary key.

[0045] Optionally, the front-end database is set to a standard front-end database and a non-standard front-end database according to different user permissions.

[0046] S120: Create an aggregation table based on the metadata, determine the data to be reconciled in the business data based on the identification information corresponding to the business data, and generate a scheduling task based on the aggregation table and the data to be reconciled. The scheduling task includes a data reconciliation task and a data aggregation task.

[0047] In an embodiment of the present disclosure, after obtaining metadata and business data, the electronic device creates an aggregation table based on the metadata, determines the data to be reconciled in the business data based on identification information corresponding to the business data, and generates a scheduling task based on the aggregation table and the data to be reconciled.

[0048] The identification information corresponding to the business data is the identification of the business data by the source end that generates the business data, which is used to indicate whether the business data is reconciliation data or non-reconciliation data.

[0049] A data reconciliation task is used to perform reconciliation processing on the data to be reconciled. A data aggregation task corresponds one-to-one to a data reconciliation task and is used to perform data aggregation processing on the data to be reconciled after the data to be reconciled corresponding to the data reconciliation task is successfully reconciled.

[0050] In the embodiment of the present disclosure, the aggregation table can be understood as a table structure created based on the metadata corresponding to each business system, and can specifically include table fields, type indexes, key information, etc.

[0051] Specifically, after obtaining the metadata, the electronic device creates an aggregation table based on the metadata, and determines the business data that needs to be reconciled based on the identification information of the business data, and determines the business data that needs to be reconciled as the data to be reconciled, and then generates a scheduling task based on the aggregation table and the data to be reconciled.

[0052] S130 , executing the data reconciliation task, and after determining that the data to be reconciled corresponding to the data reconciliation task is reconciled successfully, storing the data to be reconciled in the intermediate database.

[0053] In the embodiment of the present disclosure, after creating a scheduling task, the electronic device first executes a data reconciliation task to determine whether the data to be reconciled is reconciled successfully, and then determines subsequent execution steps based on the reconciliation result of the data to be reconciled.

[0054] Executing the data reconciliation task can be understood as verifying the data volume and data attributes of the data to be reconciled, determining whether the data volume of the data to be reconciled is consistent with the data volume pushed by the business system, and whether the data attributes of the data to be reconciled are consistent with the data attributes of the business system.

[0055] Specifically, the electronic device may construct a reconciliation task instance based on a preset reconciliation program, and execute the reconciliation task instance corresponding to the data reconciliation task to implement reconciliation processing of the reconciliation data.

[0056] The intermediate database is used to store successfully reconciled data in the front-end database. It is located after the front-end database. The intermediate database is a table without a primary key.

[0057] In some embodiments of the present disclosure, after all the to-be-reconciled data corresponding to the data reconciliation task are reconciled successfully, the to-be-reconciled data are migrated from the front-end database to the intermediate database for storage.

[0058] In other embodiments of the present disclosure, when part of the data to be reconciled corresponding to the data reconciliation task is reconciled successfully, the data that has been reconciled successfully in the data to be reconciled is migrated from the front-end database to the intermediate database for storage, and at the same time, the data that has not been reconciled successfully is determined, and a reconciliation record is generated based on the data that has not been reconciled successfully, and the data is fed back to the user or displayed on the display interface of the electronic device.

[0059] In some other embodiments of the present disclosure, when all the data to be reconciled corresponding to the data reconciliation task fails to be reconciled, all the data to be reconciled will not be stored in the intermediate database, and a reconciliation record will be generated based on all the data to be reconciled that failed to be reconciled, and fed back to the user or displayed on the display interface of the electronic device.

[0060] S140: Execute the data aggregation task to aggregate the data to be reconciled from the intermediate database to the business database.

[0061] In an embodiment of the present disclosure, after the electronic device determines that all the data to be reconciled corresponding to the data reconciliation task have been reconciled successfully, it executes the data aggregation task, performs data aggregation processing on all the data to be reconciled that have been reconciled successfully, aggregates the data to be reconciled from the intermediate database to the business database, and completes the data aggregation processing of the data to be reconciled.

[0062] In the embodiment of the present disclosure, the business database is used to store the aggregated data and is located after the intermediate database. The business database is a table with a primary key.

[0063] Specifically, after the electronic device determines that all the data to be reconciled corresponding to the data reconciliation task have been reconciled successfully, it can compare the data to be reconciled corresponding to the data aggregation task with the data in the business database, determine the target operation for the data to be reconciled corresponding to the data aggregation task, execute the target operation, and thereby realize the aggregation of the data to be reconciled from the intermediate database to the business database.

[0064] In an embodiment of the present disclosure, metadata and business data corresponding to at least one business system can be obtained, the business data and metadata can be stored in a front-end database, an aggregation table can be created based on the metadata, and the data to be reconciled in the business data can be determined based on identification information corresponding to the business data. A scheduling task can be generated based on the aggregation table and the data to be reconciled, and the scheduling task can include a data reconciliation task and a data aggregation task. The data reconciliation task is then executed, and after determining that the data to be reconciled corresponding to the data reconciliation task is successfully reconciled, the data to be reconciled is stored in an intermediate database, and a data aggregation task is executed to aggregate the data to be reconciled from the intermediate database to the business database. As a result, a front-end database, an intermediate database, and a business database can be set up on a data aggregation platform, and different business data can be stored in different databases. At the same time, after obtaining the business data, the data to be reconciled in the business data can be reconciled, and after the reconciliation is successful, the data to be reconciled is stored in the intermediate database. Then, the data corresponding to the aggregation task is obtained from the intermediate database for aggregation processing, avoiding the direct transfer of the obtained business data to the data aggregation platform, thereby improving the accuracy of the corresponding aggregated data after aggregation processing.

[0065] Based on the above embodiments of the present disclosure, when the electronic device determines that part of the to-be-reconciled data corresponding to the data reconciliation task has been reconciled successfully or all of it has failed, the electronic device does not execute the data aggregation task.

[0066] At the same time, non-reconciliation data in the business data that does not require reconciliation is directly aggregated from the front-end database to the business database. In this way, when reconciling the data to be reconciled that needs to be reconciled, it can ensure that the data to be reconciled aggregated to the business database is complete and accurate, while ensuring the integrity and aggregation of non-reconciliation data in the business data.

[0067] In some embodiments of the present disclosure, after obtaining metadata and business data corresponding to at least one business system, the electronic device records each flow process information of the business data and metadata, stores the metadata and business data in a front-end database, stores all successfully reconciled data to be reconciled corresponding to the data reconciliation task in an intermediate database, and aggregates the data corresponding to the data aggregation task from the intermediate database to the business database, and generates flow record information for each metadata and business data, and stores the flow record information to ensure the convenience of subsequent tracing and querying of the metadata and business data, and at the same time further facilitate the management of metadata and business data during the data aggregation process.

[0068] Based on the above embodiments of the present disclosure, before creating an aggregation table based on metadata, the data aggregation method may also include: for each business system, determining the data structure type corresponding to the business system, performing data conversion processing on the metadata based on the data structure type, converting the first data structure type corresponding to the metadata into a second data structure type, and obtaining the converted target metadata.

[0069] In the embodiment of the present disclosure, the data structure type can be understood as the structure type of the data table corresponding to the business system.

[0070] The second data structure type can be understood as a data table structure type that can be applied to electronic devices (data aggregation platforms), such as a MySQL type.

[0071] Specifically, the specific implementation of converting the first data structure type corresponding to the metadata into the second data structure type is similar to the existing specific method of converting two different data structure types, and will not be described in detail here.

[0072] In the embodiments of the present disclosure, metadata can be converted into data structure types for various databases in the data aggregation platform by converting the data structure types, thereby facilitating the processing and management of subsequent business data and metadata during the data aggregation process.

[0073] Furthermore, creating the aggregate table based on the metadata may specifically include: constructing a table creation statement based on the target metadata; and executing the table creation statement to generate the aggregate table.

[0074] Specifically, after obtaining the target metadata corresponding to the business system, the electronic device can perform data extraction, data screening and other processing on the target metadata based on a preset statement construction program, and construct a table construction statement. After constructing the table construction statement, it executes the table construction statement to generate an aggregate table.

[0075] The aggregation table information may include metadata information, the identifier and attribute information of the aggregation table, etc.

[0076] In an embodiment of the present disclosure, generating a scheduling task based on the aggregation table and the data to be reconciled can specifically include: extracting the aggregation table information corresponding to the aggregation table, and determining the batch number and the first data volume of the data to be reconciled; generating a data reconciliation application form based on the aggregation table information, the batch number, the first data volume and the data to be reconciled; and generating a scheduling task based on the data reconciliation application form.

[0077] In the embodiment of the present disclosure, the batch number of the data to be reconciled is generated based on a preset batch number generation rule and a generation date of the data to be reconciled.

[0078] The first data volume is the data volume corresponding to the data to be reconciled.

[0079] Specifically, after obtaining the aggregation table, the electronic device extracts the aggregation table information corresponding to the aggregation table, and at the same time generates a batch number corresponding to the data to be reconciled based on the preset batch number generation rules and the generation date of the data to be reconciled, and determines the first data volume of the data to be reconciled. Based on the aggregation table information, batch number, first data volume and data to be reconciled, a data reconciliation application form is automatically generated, and a scheduling task is generated according to the data reconciliation application form. Therefore, in response to the generation of the data reconciliation application form, a scheduling task can be generated according to the data reconciliation application form, thereby ensuring the accuracy and timeliness of the generation of the scheduling task.

[0080] In an embodiment of the present disclosure, the reconciliation task includes the batch number of the data to be reconciled corresponding to the reconciliation task. Executing the data reconciliation task may specifically include: performing a validity check on the batch number; and when it is determined that the validity check of the batch number passes, executing the reconciliation processing corresponding to the reconciliation task.

[0081] Figure 2 This is a flow chart of a method for executing a data reconciliation task provided by an embodiment of the present disclosure. Figure 2 As shown, the legitimacy check of the batch number can be specifically performed at step S210. When it is determined that the legitimacy check of the batch number passes, steps S220, S240-S250 can be specifically performed. When it is determined that the legitimacy check of the batch number fails, step S230 can be specifically performed.

[0082] S210. Based on preset verification rules, perform a validity check on the field length corresponding to the batch number, the creation date corresponding to the batch number, and whether the batch number meets preset requirements.

[0083] In the embodiment of the present disclosure, the preset verification rule may be a pre-set rule for verifying the legitimacy of the batch number.

[0084] Specifically, the electronic device compares the field length corresponding to the batch number with the preset field length to determine whether the field length corresponding to the batch number meets the requirements. When it is determined that the field length corresponding to the batch number meets the requirements, it determines whether the creation date corresponding to the batch number meets the preset time rules. When it is determined that the creation date corresponding to the batch number meets the preset time rules, it determines whether the batch number meets the preset requirements. If the preset requirements are met, it is determined that the batch number legitimacy check has passed; otherwise, it is determined that the batch number legitimacy check has failed.

[0085] Optionally, the preset time rules and preset requirements can be arbitrarily set according to user needs. For example, the preset time rule can be whether the creation date corresponding to the batch number is today's date, and the preset requirement can be to meet the batch number increment rule, that is, to compare the batch number with the batch number corresponding to the previous target reconciliation data to determine whether the batch number shows an increasing trend compared with the batch number corresponding to the previous target reconciliation data.

[0086] S220: When it is determined that the batch number validity check has passed, the first metadata corresponding to the reconciliation task is matched with the second metadata in the front-end database to determine whether the metadata match is successful.

[0087] Specifically, when the electronic device determines that the batch number legitimacy check has passed, it performs reconciliation processing, obtains the second metadata currently corresponding to the business system in the front-end database, and matches the first metadata with the second metadata to determine whether the first metadata and the second metadata are consistent. If they are consistent, it is determined that the metadata match is successful, otherwise, the metadata match fails.

[0088] Due to the different permissions corresponding to the front-end library, the metadata corresponding to the business system may be modified by someone, causing the first metadata and the second metadata to be different. Therefore, matching is required to ensure the consistency of the metadata.

[0089] S230: When it is determined that the batch number validity check fails, a first reconciliation record is generated, and a data reconciliation application form corresponding to the reconciliation task is updated.

[0090] In the embodiment of the present disclosure, the first reconciliation record may include log information during the reconciliation process.

[0091] Updating the data reconciliation application form corresponding to the reconciliation task can be understood as recording the relevant information in the reconciliation application form that fails the batch number validity check in the data reconciliation application form.

[0092] S240: When it is determined that the metadata match is successful, the first data volume of the to-be-reconciled data corresponding to the reconciliation task is matched with the second data volume of the to-be-reconciled data corresponding to the front-end database.

[0093] Specifically, when the electronic device determines that the metadata match is successful, it compares the first data volume of the data to be reconciled with the current second data volume of the corresponding data to be reconciled in the front database to determine whether the first data volume and the second data volume are consistent. If they are consistent, it is determined that the reconciliation of the data to be reconciled is successful; otherwise, it is determined that the reconciliation of the data to be reconciled has failed.

[0094] Due to different permissions corresponding to the front-end database, the data to be reconciled corresponding to the data reconciliation task in the business system may be modified by people, such as deletion, update, etc., resulting in different data volumes of the data to be reconciled. Therefore, matching is required to ensure the consistency of the data volume of the data to be reconciled.

[0095] S250: When it is determined that the metadata matching fails, a second reconciliation record is generated, and a data reconciliation application form corresponding to the reconciliation task is updated.

[0096] In the embodiment of the present disclosure, the second reconciliation record may include log information during the reconciliation process.

[0097] In the disclosed embodiment, the batch number can be checked for legitimacy before executing the reconciliation processing corresponding to the reconciliation task, and the reconciliation processing is performed only after the batch number legitimacy verification is passed, thereby ensuring the accuracy of the reconciliation processing of the data to be reconciled, avoiding the waste of resources caused by repeated verification of the data to be reconciled with the same batch number or repeated verification of the data to be reconciled that has already been reconciled, and improving the accuracy of the reconciliation processing.

[0098] In an embodiment of the present disclosure, the data aggregation method may further include: real-time monitoring of whether a new reconciliation task application form is written; when it is monitored that a new reconciliation task application form is written, generating a new target reconciliation task and a new target data aggregation task based on the new reconciliation task application form, and placing the new target reconciliation task into a preset task queue; determining the execution order of the target reconciliation task based on the target batch number corresponding to the target reconciliation task; when the target reconciliation task processing is completed and it is determined that all the reconciliation data corresponding to the target reconciliation task have been reconciled successfully, executing the target data aggregation task.

[0099] Specifically, the specific implementation methods of executing the new target reconciliation task and executing the target data aggregation task are similar to the implementation methods of executing the data reconciliation task and executing the data aggregation task described above, and are not repeated here.

[0100] In the disclosed embodiment, it is possible to obtain the push of new business data in real time, generate a reconciliation task application form based on the required reconciliation data corresponding to the new business data, and determine the execution order of the target reconciliation tasks based on the corresponding target batch number in the reconciliation task application form, thereby ensuring the priority of the execution of the reconciliation tasks and the accuracy of the execution order of the reconciliation tasks, and further improving the accuracy of the aggregated data.

[0101] Figure 3 This is a flow chart of a method for executing a data aggregation task provided by an embodiment of the present disclosure. Figure 3 As shown, executing the data aggregation task may specifically include the following steps:

[0102] S310: Determine target data corresponding to the data aggregation task in the intermediate database.

[0103] In the disclosed embodiment, after storing all the to-be-reconciled data corresponding to the data reconciliation task from the front-end database to the intermediate database, the electronic device searches the intermediate database for target data corresponding to the data aggregation task.

[0104] S320: Perform reconciliation processing on the target data. After determining that the target data reconciliation is successful, determine a target operation to be performed on the target data based on the data in the business database.

[0105] In the embodiment of the present disclosure, the specific implementation method of performing reconciliation processing on the target data is similar to the implementation method of performing reconciliation processing on the reconciliation data in the above embodiment of the present disclosure, and will not be repeated here.

[0106] Since the business database has a primary key table, the target operation to be performed on the target data can be determined directly based on the data in the business database and the target data. The target operation can include updating the existing data in the business database or adding the target data to the business database.

[0107] S330: Execute a target operation to aggregate target data from the intermediate database into the business database. The target operation includes at least one of an update operation and a new operation.

[0108] In the embodiment of the present disclosure, the target data can be reconciled when executing the data aggregation task, further ensuring the accuracy of the aggregation from the intermediate database to the business database. At the same time, by setting the business database as a primary key table, the efficiency of data aggregation is improved.

[0109] In an embodiment of the present disclosure, after the target data is aggregated from the intermediate database to the business database, after receiving a data acquisition request sent by a user, shared data corresponding to the data acquisition request is acquired from the business database and shared with the user.

[0110] In an embodiment of the present disclosure, when there are multiple data aggregation tasks, executing the data aggregation tasks may specifically include: based on a preset scheduling strategy, respectively allocating the multiple data aggregation tasks to target executor nodes; and executing the data aggregation tasks based on the target executor nodes.

[0111] Optionally, the preset scheduling strategy may include a load balancing strategy, a polling strategy, determining the target executor node based on the attributes of the executor node corresponding to the aggregation table, determining the target executor node based on the data aggregation amount corresponding to the data aggregation task, etc.

[0112] In an embodiment of the present disclosure, when the number of data aggregation tasks exceeds a preset number threshold, the executor nodes may be expanded to meet the requirements for rapid execution of the data aggregation tasks.

[0113] In the embodiment of the present disclosure, when there are multiple data aggregation tasks, the target executor node corresponding to each data aggregation task can be determined based on a preset scheduling strategy, thereby further improving the efficiency of the data aggregation tasks.

[0114] Figure 4 It is a structural diagram of a data aggregation device provided by an embodiment of the present disclosure.

[0115] In the embodiments of the present disclosure, the data aggregation device may be provided within an electronic device and may be understood as a functional module within the electronic device. Specifically, the electronic device may be a server or a terminal, where the terminal specifically includes a computer or tablet computer, or any device capable of processing the data aggregation method, without limitation.

[0116] like Figure 4 As shown, the data aggregation device 400 may include a data acquisition module 410 , a task generation module 420 , a first task execution module 430 and a second task execution module 440 .

[0117] The data acquisition module 410 may be used to acquire metadata and business data corresponding to at least one business system, and store the business data and metadata in a front-end database.

[0118] The task generation module 420 can be used to create an aggregation table based on metadata, and determine the data to be reconciled in the business data based on the identification information corresponding to the business data, and generate scheduling tasks based on the aggregation table and the data to be reconciled. The scheduling tasks include data reconciliation tasks and data aggregation tasks.

[0119] The first task execution module 430 may be configured to execute a data reconciliation task, and after determining that the to-be-reconciled data corresponding to the data reconciliation task is reconciled successfully, store the to-be-reconciled data in an intermediate database.

[0120] The second task execution module 440 may be configured to execute a data aggregation task to aggregate the data to be reconciled from the intermediate database to the business database.

[0121] In an embodiment of the present disclosure, metadata and business data corresponding to at least one business system can be obtained, the business data and metadata can be stored in a front-end database, an aggregation table can be created based on the metadata, and the data to be reconciled in the business data can be determined based on identification information corresponding to the business data. A scheduling task can be generated based on the aggregation table and the data to be reconciled, and the scheduling task can include a data reconciliation task and a data aggregation task. The data reconciliation task is then executed, and after determining that the data to be reconciled corresponding to the data reconciliation task is successfully reconciled, the data to be reconciled is stored in an intermediate database, and a data aggregation task is executed to aggregate the data to be reconciled from the intermediate database to the business database. As a result, a front-end database, an intermediate database, and a business database can be set up on a data aggregation platform, and different business data can be stored in different databases. At the same time, after obtaining the business data, the data to be reconciled in the business data can be reconciled, and after the reconciliation is successful, the data to be reconciled is stored in the intermediate database. Then, the data corresponding to the aggregation task is obtained from the intermediate database for aggregation processing, avoiding the direct transfer of the obtained business data to the data aggregation platform, thereby improving the accuracy of the corresponding aggregated data after aggregation processing.

[0122] In some embodiments of the present disclosure, the data aggregation device 400 may further include a data conversion module.

[0123] The data conversion module can be used to determine the data structure type corresponding to each business system before creating an aggregation table based on metadata, perform data conversion on the metadata based on the data structure type, convert the first data structure type corresponding to the metadata into the second data structure type, and obtain the converted target metadata.

[0124] In some embodiments of the present disclosure, the task generation module 420 may be specifically configured to construct a table creation statement based on target metadata; and execute the table creation statement to generate a converged table.

[0125] In some embodiments of the present disclosure, the task generation module 420 can also be used to extract the aggregation table information corresponding to the aggregation table, and determine the batch number and first data volume of the data to be reconciled; generate a data reconciliation application form based on the aggregation table information, batch number, first data volume and data to be reconciled; and generate a scheduling task based on the data reconciliation application form.

[0126] In some embodiments of the present disclosure, the reconciliation task includes the batch number of the to-be-reconciled data corresponding to the reconciliation task.

[0127] The first task execution module 430 may include a legality verification unit and a reconciliation processing unit.

[0128] The validity check unit can be used to check the validity of the batch number.

[0129] The reconciliation processing unit may be configured to execute reconciliation processing corresponding to the reconciliation task when it is determined that the validity check of the batch number has passed.

[0130] In some embodiments of the present disclosure, the legitimacy verification unit can be specifically used to perform legitimacy verification on the field length corresponding to the batch number, the creation date corresponding to the batch number, and whether the batch number meets preset requirements based on preset verification rules.

[0131] The reconciliation processing unit can be specifically used to match the first metadata corresponding to the reconciliation task with the second metadata in the front-end database to determine whether the metadata match is successful; when it is determined that the metadata match is successful, the first data volume of the to-be-reconciled data corresponding to the reconciliation task is matched with the second data volume of the to-be-reconciled data corresponding to the front-end database.

[0132] In some embodiments of the present disclosure, the data aggregation device 400 may further include a monitoring module.

[0133] The monitoring module can be used to monitor in real time whether there is a new reconciliation task application form written; when a new reconciliation task application form is monitored, a new target reconciliation task is generated based on the new reconciliation task application form, and the new target reconciliation task is placed in the preset task queue; the execution order of the target reconciliation task is determined based on the target batch number corresponding to the target reconciliation task.

[0134] In some embodiments of the present disclosure, the second task execution module 440 can be specifically used to determine the target data corresponding to the data aggregation task in the intermediate database; perform reconciliation processing on the target data, and after determining that the target data reconciliation is successful, determine the target operation to be performed on the target data based on the data in the business database; execute the target operation to aggregate the target data from the intermediate database to the business database, and the target operation includes at least one of an update operation and a new operation.

[0135] It should be noted that Figure 4 The data aggregation device 400 shown can execute each step in the above method embodiment and realize each process and effect in the above method embodiment, which will not be described in detail here.

[0136] Figure 5 It is a structural diagram of an electronic device provided by an embodiment of the present disclosure.

[0137] In the embodiments of the present disclosure, Figure 5 The electronic device shown may be a server or a terminal, wherein the terminal specifically includes a mobile phone, a computer or a tablet computer, etc., and may also be any device capable of processing the data aggregation method, which is not limited here.

[0138] like Figure 5As shown, the electronic device may include a processor 510 and a memory 520 storing computer program instructions.

[0139] Specifically, the processor 510 may include a central processing unit (CPU), or an application specific integrated circuit (ASIC), or may be configured to implement one or more integrated circuits of the embodiments of the present disclosure.

[0140] Memory 520 may include a large-capacity memory for information or instructions. By way of example, and not limitation, memory 520 may include a hard disk drive (HDD), a floppy disk drive, flash memory, an optical disk, a magneto-optical disk, magnetic tape, or a Universal Serial Bus (USB) drive, or a combination of two or more of these. Where appropriate, memory 520 may include removable or non-removable (or fixed) media. Where appropriate, memory 520 may be internal or external to the integrated gateway device. In certain embodiments, memory 520 is non-volatile solid-state memory. In certain embodiments, memory 520 includes read-only memory (ROM). Where appropriate, the ROM may be mask-programmed ROM, programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable PROM (EEPROM), electrically alterable ROM (EAROM), or flash memory, or a combination of two or more of these.

[0141] The processor 510 reads and executes computer program instructions stored in the memory 520 to perform the steps of the data aggregation method provided in the embodiment of the present disclosure.

[0142] In one example, the electronic device may further include a transceiver 530 and a bus 540. Figure 5 As shown, the processor 510 , the memory 520 and the transceiver 530 are connected via a bus 540 and communicate with each other.

[0143] The bus 540 includes hardware, software, or both. By way of example and not limitation, the bus may include an Accelerated Graphics Port (AGP) or other graphics bus, an Extended Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), a Hyper Transport (HT) interconnect, an Industrial Standard Architecture (ISA) bus, an InfiniBand interconnect, a Low Pin Count (LPC) bus, a memory bus, a MicroChannel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local Bus (VLB) bus, or other suitable buses, or a combination of two or more of these. Where appropriate, the bus 540 may include one or more buses.

[0144] The embodiments of the present disclosure further provide a computer-readable storage medium, which may store a computer program. When the computer program is executed by a processor, the processor implements the data aggregation method provided by the embodiments of the present disclosure.

[0145] The above-mentioned storage medium may, for example, include a memory 520 of computer program instructions, and the above-mentioned instructions may be executed by the processor 510 of the electronic device to complete the data aggregation method provided in the embodiment of the present disclosure. Optionally, the storage medium may be a non-transitory computer-readable storage medium, for example, a non-transitory computer-readable storage medium may be a ROM, a random access memory (RAM), a compact disc read-only memory (CD-ROM), a magnetic tape, a floppy disk, an optical data storage device, etc.

[0146] The foregoing description is intended only to provide specific embodiments of the present disclosure, intended to enable those skilled in the art to understand and implement the present disclosure. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present disclosure. Therefore, the present disclosure is not intended to be limited to the embodiments described herein, but rather to be construed in the broadest manner consistent with the principles and novel features disclosed herein.

Claims

1. A data aggregation method, characterized in that: include: Obtain metadata and business data corresponding to at least one business system, and store the business data and metadata in a front-end database; Creating an aggregation table based on the metadata, determining the data to be reconciled in the business data based on identification information corresponding to the business data, and generating a scheduling task based on the aggregation table and the data to be reconciled, the scheduling task including a data reconciliation task and a data aggregation task; Executing the data reconciliation task, and after determining that the to-be-reconciled data corresponding to the data reconciliation task is reconciled successfully, storing the to-be-reconciled data in an intermediate database; Execute the data aggregation task to aggregate the to-be-reconciled data from the intermediate database to the business database.

2. The method according to claim 1, characterized in that Before creating the aggregation table based on the metadata, the method further includes: For each business system, determining a data structure type corresponding to the business system, performing data conversion processing on the metadata based on the data structure type, converting a first data structure type corresponding to the metadata into a second data structure type, and obtaining converted target metadata; The creating of the aggregation table based on the metadata includes: Constructing a table creation statement based on the target metadata; Execute the table creation statement to generate the aggregate table.

3. The method according to claim 1, characterized in that The generating of a scheduling task based on the aggregation table and the to-be-reconciled data includes: Extracting aggregation table information corresponding to the aggregation table, and determining a batch number and a first data volume of the data to be reconciled; generating a data reconciliation application form based on the aggregation table information, the batch number, the first data volume, and the data to be reconciled; The scheduling task is generated based on the data reconciliation application form.

4. The method according to claim 1, wherein The reconciliation task includes a batch number of the to-be-reconciled data corresponding to the reconciliation task, and executing the data reconciliation task includes: Performing a validity check on the batch number; When it is determined that the validity check of the batch number passes, a reconciliation process corresponding to the reconciliation task is executed.

5. The method according to claim 4, characterized in that The legitimacy verification of the batch number includes: Based on the preset verification rules, the legitimacy of the field length corresponding to the batch number, the creation date corresponding to the batch number, and whether the batch number meets the preset requirements is verified; The performing of the reconciliation process corresponding to the reconciliation task includes: Matching the first metadata corresponding to the reconciliation task with the second metadata in the front-end database, and determining whether the metadata are successfully matched; When it is determined that the metadata match is successful, the first data volume of the to-be-reconciled data corresponding to the reconciliation task is matched with the second data volume of the to-be-reconciled data corresponding to the front-end database.

6. The method according to claim 1, characterized in that The method further comprises: Monitor in real time whether there are new reconciliation task application forms written; When a new reconciliation task application form is detected, a new target reconciliation task is generated based on the new reconciliation task application form and the new target reconciliation task is placed in the preset task queue; An execution order of the target reconciliation tasks is determined based on target batch numbers corresponding to the target reconciliation tasks.

7. The method according to claim 1, characterized in that The executing the data aggregation task to aggregate the to-be-reconciled data from the intermediate database to the business database includes: Determine target data corresponding to the data aggregation task in the intermediate database; performing reconciliation processing on the target data, and after determining that the reconciliation of the target data is successful, determining a target operation to be performed on the target data based on the data in the business database; The target operation is executed to aggregate the target data from the intermediate database into the business database, where the target operation includes at least one of an update operation and a new addition operation.

8. A data aggregation device, characterized in that: include: A data acquisition module, configured to acquire metadata and business data corresponding to at least one business system, and store the business data and metadata in a front-end database; a task generation module, configured to create an aggregation table based on the metadata, determine the data to be reconciled in the business data based on identification information corresponding to the business data, and generate a scheduling task based on the aggregation table and the data to be reconciled, the scheduling task including a data reconciliation task and the data aggregation task; A first task execution module is configured to execute the data reconciliation task and, after determining that the to-be-reconciled data corresponding to the data reconciliation task is reconciled successfully, store the to-be-reconciled data in an intermediate database; The second task execution module is used to execute the data aggregation task and aggregate the to-be-reconciled data from the intermediate database to the business database.

9. An electronic device, characterized in that: include: processor; a memory for storing executable instructions; The processor is configured to read the executable instructions from the memory and execute the executable instructions to implement the data aggregation method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that The storage medium stores a computer program, and when the computer program is executed by a processor, the processor implements the data aggregation method according to any one of claims 1 to 7.

Images