Multi-source data aggregation processing method and system

Abstract

The invention discloses a multi-source data aggregation processing method and system, and relates to a data processing technology, and the method comprises the steps: loading an original data stream from various types of data sources through a data source loading engine, and processing the original data stream into metadata through a corresponding data converter according to the types of the data sources, processing the metadata so as to record the processed metadata into a distributed storage; and registering the metadata in the distributed storage to a metadata management service of a rule configuration background, utilizing a rule configuration center of the rule configuration background to configure an association relationship of metadata depended under a current rule so as to pre-generate a fusion result according to a rule plan task after the configuration is completed; and issuing the latest rule to a unified interface service by utilizing the rule configuration background, so as to externally provide an output interface in a specified format by utilizing the unified interface service. According to the invention, multi-data integration processing is provided, and a universal middle table which shields bottom-layer complex logic based on a unified interface is provided.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a method and system for aggregate processing of multi-source data. Background Art

[0002] Nowadays, enterprise projects are showing a trend of multi-unit, multi-project, and multi-language joint work. The upstream and downstream call chains are long, the data transmission methods are complex, and there are many dependencies on interactions, but the requirements for timeliness are not low. If we refer to the working form of Internet companies, the interaction of upstream and downstream data mostly adopts the following methods:

[0003] The downstream puts forward data requirements, the upstream provides data interfaces, and the downstream calls customized interfaces for development;

[0004] The downstream synchronizes the upstream business library and develops directly based on the synchronized library;

[0005] Upstream data flows directly into big data services such as the Hadoop platform through middleware. Downstream data is processed by big data components such as Spark and Flink for secondary data transcription, and the results are stored in the relational database again.

[0006] However, all of the above cooperation methods have some disadvantages:

[0007] Due to the complex upstream and downstream dependencies, the development of new interfaces is time-consuming and laborious, and changes frequently. In addition, due to the large correlation between services, it is difficult to strictly separate the upstream and downstream relationships.

[0008] Database synchronization often has a certain delay, so it cannot meet the high requirements of business software for data real-time performance;

[0009] The big data platform has long dependencies and links, requires high configuration and cluster environment, and places too high demands on servers.

[0010] In addition, the input of multiple data sources increases the complexity of the project, reduces its robustness, disperses the business logic, and customized functions are often only used for a single purpose. Summary of the invention

[0011] The embodiments of the present application provide a method and system for aggregate processing of multi-source data, which are used to propose multi-data integration processing, customize dependency relationships, and use a universal middle platform based on a unified interface to shield the underlying complex logic.

[0012] The present application provides a method for aggregating multi-source data, including:

[0013] Using a data source loading engine, loading raw data streams from various types of data sources, and processing them into metadata through corresponding data converters according to the types of data sources, so as to record the processed metadata into distributed storage; and,

[0014] Register the metadata in the distributed storage to the metadata management service of the rule configuration background;

[0015] Utilize the rule configuration center of the rule configuration background to configure the association relationship of the metadata relied on by the current rule, so as to pre-generate the fusion result according to the rule planning task after the configuration is completed;

[0016] The rule configuration background is used to publish the latest rules to the unified interface service, so as to use the unified interface service to provide an output interface in a specified format to the outside.

[0017] Optionally, using a data source loading engine, loading raw data streams from multiple types of data sources is achieved by using corresponding engine plug-ins for different types of data sources. After the engine plug-in reads the data, the data is input into a corresponding data converter.

[0018] Optionally, it also includes utilizing a data source loading engine to convert metadata into a temporary cache and store it in a distributed cache of the rule configuration background through a scheduling service.

[0019] Optionally, processing the metadata through a corresponding data converter according to the type of the data source includes: setting different data converters for the same data source to adapt to file requirements of different businesses.

[0020] Optionally, it also includes using the rule configuration background to manually add subscriber information, and, when the subscribed information changes, the changed interface registers the tracked metadata to the registration center of the rule configuration background, so that the registration center performs heartbeat detection and maintains monitoring of the metadata online and offline.

[0021] The present application also provides a multi-source data aggregation processing system, including:

[0022] A data source loading engine, including a data converter and distributed storage, wherein the data source loading engine is used to load original data streams from various types of data sources, and process the original data streams into metadata through corresponding data converters according to the types of data sources, so as to record the processed metadata into distributed storage; and, to register the metadata in the distributed storage into the metadata management service of the rule configuration background;

[0023] A rule configuration backend, which is used to configure the association relationship of the metadata relied on by the current rule in the rule configuration center of the rule configuration backend, so as to pre-generate the fusion result according to the rule planning task after the configuration is completed; and publish the latest rules to the unified interface service;

[0024] Unified interface service, used to provide an output interface in a specified format to the outside world.

[0025] Optionally, the data source loading engine further includes engine plug-ins set for different data source types, and after the data is read by the engine plug-in, the data is input into a corresponding data converter.

[0026] Optionally, the data source loading engine is further used to convert the metadata into a temporary cache and store it in the distributed cache of the rule configuration background through a scheduling service.

[0027] Optionally, the data source loading engine is further used to set different data converters for the same data source to adapt to file requirements of different businesses.

[0028] Optionally, the rule configuration background is also used to manually add subscriber information, and when the subscribed information changes, the changed interface registers the tracked metadata to the registration center of the rule configuration background, so that the registration center performs heartbeat detection and maintains monitoring of the metadata online and offline.

[0029] The embodiment of the present application proposes a universal middle-office system for multi-data integration processing, dependency customization, and shielding of underlying complex logic based on a unified interface, thereby solving the pain points during upstream and downstream interactions. After the upstream access is completed, the downstream use is unaware of the upstream, thus reducing the tedious work of the upstream in providing support to multiple downstreams.

[0030] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present application. Also, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings:

[0032] Figure 1 The overall architecture of the aggregation processing system of the embodiment of the present application is shown;

[0033] Figure 2The following is a schematic diagram of the data source access process of the aggregation processing system according to an embodiment of the present application. DETAILED DESCRIPTION

[0034] The exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0035] The embodiment of the present application also proposes a multi-source data aggregation processing system, which includes a unified interface service, a rule configuration background, a data source loading engine and a basic service component. Figure 1 As shown, where:

[0036] The data source loading engine includes a data converter and distributed storage. The data source loading engine is used to load raw data streams from various types of data sources, and process them into metadata through corresponding data converters according to the types of data sources, so as to record the processed metadata into distributed storage; and register the metadata in the distributed storage with the metadata management service of the rule configuration background. In some specific embodiments, such as Figure 1 As shown, the data source loading engine also includes engine plug-ins set for different data source types. After the data is read by the engine plug-in, the data is input into the corresponding data converter.

[0037] In some examples, the data source loading engine is also used to set different data converters for the same data source to adapt to the file requirements of different businesses.

[0038] The rule configuration background is used to configure the association relationship of the metadata relied on by the current rule in the rule configuration center of the rule configuration background. For example, manual configuration can be performed, or further nested rule configuration can be performed based on rule logic. After the configuration is completed, the fusion results are pre-generated according to the rule planning tasks; and the latest rules are published to the unified interface service.

[0039] The unified interface service is used to provide an output interface in a specified format to the outside. In some embodiments, the latest rules are published to the unified interface service, and the unified interface service is used to provide an output interface in XML format to the outside.

[0040] In some embodiments, the data source loading engine is further used to convert metadata into a temporary cache and store it in the distributed cache of the rule configuration background through a scheduling service.

[0041] In some embodiments, the rule configuration background is also used to manually add subscriber information, and when the subscribed information changes, the changed interface registers the tracked metadata to the registration center of the rule configuration background, so that the registration center can perform heartbeat detection, maintain monitoring of metadata online and offline, and effectively supervise.

[0042] The embodiment of the present application also proposes an implementation case of the aggregation processing system of multi-source data. In this example, the access of a new data source is used as an example to describe the workflow of the aggregation system of the present application as a whole:

[0043] like Figure 2 As shown, the upstream service (data source provider) provides data table Table_A and text File_B for downstream services to use. The upstream service first connects the newly introduced data source to the aggregation center. The specific operation method is: according to the data source type, select the engine plug-in and add the engine plug-in to the engine library of the service. In the example, the upstream service selects DAMO and XML engine. The engine parses the original format into a data stream and loads it into the memory.

[0044] At the same time, the upstream writes the converters corresponding to Table_A and File_B. The converters are associated with the data source and will convert the proprietary data format into the unified metadata format of the aggregation platform during processing. After completing the metadata conversion, the upstream manually registers the system-unique registration name for the loaded metadata Table_A and File_B in the metadata management service, and associates the corresponding converters with the engine. At this time, the overall registration process is completed. After the registration is completed, the metadata information is loaded into the distributed cache in stages.

[0045] After the upstream configuration is completed, the downstream service (data source user) logs in to the rule configuration background, adds the subscribed metadata information Table_A and File_B, and adds the usage rules for Table_A and File_B in the rule configuration center, adding their field associations, filter conditions, and or non-hierarchical relationships, and the final required output result fields. After editing the above rules, publish the rules and publish them as interfaces. After the interface is published, in order to continuously track interface changes, manually add subscribers to the rule, configure the IP and port of the business service, and add a callback method in the business code to receive callback messages in real time and process update information.

[0046] The data provided by the upstream data in this application is in the original format. After the upstream access is completed, the downstream use is unaware of the upstream, reducing the tedious work of the upstream providing support to multiple downstreams. At the same time, the downstream can independently configure and process the original format into the required result, and finally only needs to parse the final published interface.

[0047] The multi-source data aggregation processing system of the embodiment of the present application decouples the database layer and the business layer, so that the business software is driven by the business and does not need to care about the implementation of the underlying data layer. At the same time, it simplifies the deployment process, and the complex underlying links are handled by a unified processing service and centrally managed; at the same time, the data layer is uniformly registered based on the registration center to realize the publishing, subscription and updating of data content. While all the data are elementized, the data is integrated, processed and processed according to the custom rule configuration, which solves the pain points of upstream and downstream interactions and no longer relies on upstream support.

[0048] The embodiment of the present application proposes a universal middle-office system for multi-data integration processing, dependency customization, and shielding of underlying complex logic based on a unified interface, thereby solving the pain points during upstream and downstream interactions. After the upstream access is completed, the downstream use is unaware of the upstream, thus reducing the tedious work of the upstream in providing support to multiple downstreams.

[0049] The present application provides a method for aggregating multi-source data, including:

[0050] Using a data source loading engine, loading raw data streams from various types of data sources, and processing them into metadata through corresponding data converters according to the types of data sources, so as to record the processed metadata into distributed storage; and,

[0051] Register the metadata in the distributed storage to the metadata management service of the rule configuration background;

[0052] Utilize the rule configuration center of the rule configuration background to configure the association relationship of the metadata relied on by the current rule, so as to pre-generate the fusion result according to the rule planning task after the configuration is completed;

[0053] The rule configuration background is used to publish the latest rules to the unified interface service, so as to use the unified interface service to provide an output interface in a specified format to the outside.

[0054] In some embodiments, using a data source loading engine, loading raw data streams from multiple types of data sources is achieved by using corresponding engine plug-ins for different types of data sources. After the data is read by the engine plug-in, the data is input into a corresponding data converter.

[0055] In some embodiments, the method further includes utilizing a data source loading engine to convert metadata into a temporary cache and store it in a distributed cache of the rule configuration background through a scheduling service.

[0056] In some embodiments, processing the data source into metadata through a corresponding data converter according to the type of the data source includes: setting different data converters for the same data source to adapt to file requirements of different businesses.

[0057] In some embodiments, it also includes utilizing the rule configuration background to manually add subscriber information, and, in the event that the subscribed information changes, the changed interface registers the tracked metadata to the registration center of the rule configuration background, so that the registration center performs heartbeat detection and maintains monitoring of the metadata going online and offline.

[0058] It should be noted that in the various embodiments of the present application, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "includes a ..." does not exclude the presence of other identical elements in the process, method, article or device including the element.

[0059] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0060] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0061] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

Claims

1. A method for aggregate processing of multi-source data, characterized in that: include: Using the data source loading engine, the original data stream is loaded from various types of data sources, and processed into metadata through the corresponding data converter according to the type of data source, so as to record the processed metadata into distributed storage; as well as, Register the metadata in the distributed storage to the metadata management service of the rule configuration background; Utilize the rule configuration center of the rule configuration background to configure the association relationship of the metadata relied on by the current rule, so as to pre-generate the fusion result according to the rule planning task after the configuration is completed; The rule configuration background is used to publish the latest rules to the unified interface service, so as to use the unified interface service to provide an output interface in a specified format to the outside.

2. The method for aggregate processing of multi-source data according to claim 1, characterized in that: Using the data source loading engine, loading the original data stream from various types of data sources is achieved by using corresponding engine plug-ins for different types of data sources. After the data is read by the engine plug-in, the data is input into the corresponding data converter.

3. The method for aggregate processing of multi-source data according to claim 2, characterized in that: It also includes utilizing a data source loading engine to convert metadata into a temporary cache and store it in a distributed cache of the rule configuration background through a scheduling service.

4. The method for aggregate processing of multi-source data according to claim 1, characterized in that: Processing the data source into metadata through a corresponding data converter according to the type of the data source includes: setting different data converters for the same data source to adapt to the file requirements of different businesses.

5. The method for aggregate processing of multi-source data according to claim 1, characterized in that: It also includes using the rule configuration background to manually add subscriber information, and, when the subscribed information changes, the changed interface registers the tracked metadata to the registration center of the rule configuration background, so that the registration center performs heartbeat detection and maintains monitoring of the metadata online and offline.

6. A multi-source data aggregation processing system, characterized in that: include: A data source loading engine, including a data converter and distributed storage, wherein the data source loading engine is used to load original data streams from various types of data sources, and process the original data streams into metadata through corresponding data converters according to the types of the data sources, so as to record the processed metadata into the distributed storage; And, register the metadata in the distributed storage to the metadata management service of the rule configuration background; A rule configuration backend, which is used to configure the association relationship of the metadata relied on by the current rule in the rule configuration center of the rule configuration backend, so as to pre-generate the fusion result according to the rule planning task after the configuration is completed; and publish the latest rules to the unified interface service; Unified interface service, used to provide an output interface in a specified format to the outside world.

7. The multi-source data aggregation processing system according to claim 6, characterized in that: The data source loading engine also includes engine plug-ins set for different data source types. After the data is read by the engine plug-in, the data is input into a corresponding data converter.

8. The multi-source data aggregation processing system according to claim 7, characterized in that: The data source loading engine is also used to convert metadata into a temporary cache and store it in the distributed cache of the rule configuration background through a scheduling service.

9. The multi-source data aggregation processing system according to claim 6, characterized in that: The data source loading engine is also used to set different data converters for the same data source to adapt to the file requirements of different businesses.

10. The multi-source data aggregation processing system according to claim 6, characterized in that: The rule configuration background is also used to manually add subscriber information, and when the subscribed information changes, the changed interface registers the tracked metadata to the registration center of the rule configuration background, so that the registration center performs heartbeat detection and maintains monitoring of the metadata online and offline.

Images