Universal query field aggregation method and device for multi-source heterogeneous data, electronic equipment and storage medium

Abstract

The invention discloses a general query field aggregation method and device for multi-source heterogeneous data, electronic equipment and a storage medium, and the method comprises the steps: employing a task flow annotation configurator to carry out annotation marking for each task chain in advance, and marking the task chain as a column aggregation processing flow task chain; managing an execution chain type task set marked by the same type through a task control manager, and loading a task processor to a system cache; loading a task processor in a system cache through a chained task processor, and executing a preset task based on a task process processor interface; and receiving a plurality of chained task processors through a column data aggregation executor, constructing processor request context parameters, executing the processors in sequence, and outputting an aggregation result. By means of the method and device, on one hand, flexible arrangement of the processing sequence of the function processors is achieved, dynamic expansion of the processors is achieved, on the other hand, the processing flow of field-level aggregation is unified, and the aggregation query efficiency is improved.

Description

Technical Field

[0001] The present application relates to the technical field of multi-source heterogeneous data query, and in particular to a method, device, electronic device, and storage medium for aggregating common query fields for multi-source heterogeneous data. Background Art

[0002] Multi-source heterogeneous data comes from different data sources in multiple different systems. Usually, the data generated by different systems have different storage modes and logical structures, and the field types and presentation forms of the data are also different, which results in the "multi-source" feature of data.

[0003] In the scenario of an enterprise-level query platform, the function of providing unified information query often requires obtaining heterogeneous data generated in the same scenario through different system query interfaces, and realizing the aggregation and fusion function of multi-source heterogeneous data on the query platform, and feeding back to the client system through online messages or asynchronous files.

[0004] Among them, the aggregation situations involved mainly include:

[0005] 1. Data obtained from different data sources are spliced ​​at the row level, that is, data row level aggregation.

[0006] 2. Data obtained from different data sources are spliced ​​at the data column level, that is, data field level aggregation. Summary of the invention

[0007] The embodiments of the present application provide a general query field aggregation method, device, electronic device, and storage medium for multi-source heterogeneous data, so as to provide a query aggregation and fusion method that is general, easy to expand, flexible, and supports field combinations of different main data sources and sub-data sources.

[0008] The present application embodiment adopts the following technical solutions:

[0009] In a first aspect, an embodiment of the present application provides a general query field aggregation method for multi-source heterogeneous data, wherein the method includes:

[0010] Use the task flow annotation configurator to pre-annotate each task chain and mark the task chain as a column aggregation processing flow task chain;

[0011] The task control manager manages the collection of execution chain tasks with the same type of tags and loads the task processor into the system cache;

[0012] Load the task processor in the system cache through the chain task processor, and execute the preset task based on the task flow processor interface;

[0013] The column data aggregation executor receives the plurality of chained task processors, constructs a processor request context parameter, executes the processors in sequence and outputs an aggregation result.

[0014] In some embodiments, the chain task processor is used to perform any one or more tasks of specific interface calling, message assembly, main source outbound call request, sub-source outbound call request, data callback processing, and data assembly post-processing, and the method further includes:

[0015] After loading the task processor in the system cache through the chain task processor, column data task processing is performed based on the task flow processor interface, and the aggregate task call method is called through an asynchronous thread. Then, the task processing parameters are combined through the context of the constructed object, and each task handle processing method in the chain task collection is traversed and executed in turn until the last one is processed.

[0016] In some embodiments, performing column data task processing based on the task flow processor interface includes one of the following:

[0017] The preprocessor of the primary and secondary source column data aggregation task is used to perform pre-processing operations on parameters, contexts, and logs before executing the primary and secondary source outbound calls and single source result data processing;

[0018] The main source execution processor uses the thread pool to receive the execution tasks of the main source, asynchronously executes and completes the parsing of the execution results, constructs standard objects and puts them into the cache;

[0019] The sub-source execution processor uses the thread pool to receive the execution tasks of the sub-source, parses the execution results after the asynchronous execution is completed, and constructs a standard object and puts it into the cache;

[0020] The multi-source aggregation processor uses a unique field to load the current batch of data information from the cache, calls back the business processing method to process and parse the data rows, and returns the parsed data in a unified manner.

[0021] In some embodiments, a task flow annotation configurator is used to pre-annotate each task chain, and the task chain is marked as a column aggregation processing flow task chain, including:

[0022] When the application service is started, the task flow annotation configurator is used to load the service objects marked with the task flow configuration annotation, and the objects are arranged in reverse order according to the order of the annotation configuration, and the sorted objects are loaded into the cache.

[0023] In some embodiments, managing a set of execution chained tasks with the same type of tags through a task control manager and loading a task processor into a system cache includes:

[0024] When the business service is called, a multi-source column data aggregation task is constructed through the task control manager, and context-related data and a chained task set are passed into the column data aggregation task.

[0025] In some embodiments, the task control manager is also used to

[0026] When the system is initialized, scan the objects annotated with the configurator, parse the type and order of the configuration, sort them according to different order values ​​of the same type, and load the sorted task processors into the system cache.

[0027] In some embodiments, the main source and sub-source aggregation scenarios of the application include at least one of the following:

[0028] A standard scenario of aggregating columns of two data sources;

[0029] The main source / sub-source data is composed of multiple data sources. The main source and multiple sources can support the combination of response data into one row, and also support the combination of multiple rows of data into multiple rows.

[0030] Scenarios where fields from multiple data sources are aggregated.

[0031] In a second aspect, an embodiment of the present application further provides a general query field aggregation device for multi-source heterogeneous data, wherein the device comprises:

[0032] The task flow annotation configuration module is used to use the task flow annotation configurator to pre-annotate and mark each task chain, marking the task chain as a column aggregation processing flow task chain;

[0033] A task control management module is used to manage a set of execution chain tasks with the same type of tags through a task control manager, and load the task processor into the system cache;

[0034] The chain task processing module is used to load the task processor in the system cache through the chain task processor and execute the preset task based on the task flow processor interface;

[0035] The column data aggregation execution module is used to receive multiple chain task processors through the column data aggregation executor, construct processor request context parameters, execute the processors in sequence and output aggregation results.

[0036] In a third aspect, an embodiment of the present application further provides an electronic device, comprising: a processor; and a memory arranged to store computer executable instructions, wherein the executable instructions, when executed, cause the processor to perform the above method.

[0037] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, which stores one or more programs. When the one or more programs are executed by an electronic device including multiple application programs, the electronic device executes the above method.

[0038] At least one of the above technical solutions adopted in the embodiments of the present application can achieve the following beneficial effects: a task flow annotation configurator is used to pre-annotate and label each task chain, and the task chain is marked as a column aggregation processing flow task chain. The task control manager is then used to manage the execution chain task set with the same type of label, and the task processor is loaded into the system cache. Furthermore, the task processor in the system cache is loaded by the chain task processor, and the preset task is executed based on the task flow processor interface. By defining the process processor processing behavior as a unified interface standard behavior, each actual working processor is controlled by the control manager and uniformly executed by the process task executor. Finally, a plurality of the chain task processors are received by the column data aggregation executor, and the processor request context parameters are constructed, the processor is executed in sequence and the aggregation result is output. Through the above method, a solution is provided for the scenario where the enterprise-level query platform needs to perform field-level data fusion when aggregating different data sources.

[0039] Through the above method, the processor is defined as a field aggregation processing type through annotation, and automatic scanning and loading into the manager memory are realized. At the same time, through the above method, each process processor is implemented according to a unified interface standard, and different implementation logics are used according to different processing functions to meet the work completed by the processor. In addition, the same executor executes the function processors that have been arranged in the memory, and continuously changes the context data information during the processing process, passing the latest data information to the next function processor until the last function processor completes the processing and outputs the result. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0041] Figure 1 It is a schematic diagram of data row-level aggregation processing in the related technology;

[0042] Figure 2 A diagram of hard-coded customized processing through different code classes for different businesses in related technologies according to different primary and secondary source logics;

[0043] Figure 3 This is a flow chart of a general query field aggregation method for multi-source heterogeneous data in an embodiment of the present application;

[0044] Figure 4 A schematic diagram of a data aggregation engine supporting multi-source data parallel callback service processing in an embodiment of the present application;

[0045] Figure 5 A schematic diagram of a specific processing flow of a general query field aggregation method for multi-source heterogeneous data in an embodiment of the present application;

[0046] Figure 6 This is a schematic diagram of the structure of a general query field aggregation device for multi-source heterogeneous data in an embodiment of the present application;

[0047] Figure 7 It is a schematic diagram of the structure of an electronic device in an embodiment of the present application. DETAILED DESCRIPTION

[0048] In order to make the purpose, technical solution and advantages of the present application clearer, the technical solution of the present application will be clearly and completely described below in combination with the specific embodiments of the present application and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present application.

[0049] In the multi-data source field-level aggregation scenario, most of the existing technologies are to hard-code and add data source interface call codes according to fixed business scenarios. After obtaining the query results of the main data source or sub-data source, the fixed secondary business rule processing is performed according to the data of the main and sub-sources, and finally the aggregated result data is obtained. Taking the query and aggregation field processing of four data sources as an example, the data line queries source A, and puts the results of source A into the cache after obtaining them. Then, the BC sources are queried. Among them, source B is the main data source of source C. It is necessary to obtain the results of source B for processing. After further processing, the request message of source C is constructed to call the source C interface. After obtaining the result data, the data of sources B and C are aggregated by field splicing to obtain a series of complete data lists, added to the cache list to be processed, and then the query of source D is performed. Finally, after obtaining the result data of all data sources, data row-level aggregation processing is performed, such as Figure 1 In the process of BC data source field aggregation, it is easiest to design the data information splicing of the main and sub-sources according to the business logic of a specific scenario. Different businesses are hard-coded and customized through different code classes according to different main and sub-source logics. Figure 2 shown.

[0050] For each business scenario, row and column-level aggregation of different source data is implemented through different codes according to data distribution and field aggregation and fusion rules. The following problems exist:

[0051] (1) High code duplication rate. Each business interface must aggregate different data according to specific scenarios, but the core aggregation logic is consistent, resulting in a large amount of repeated identical logic code, making maintenance difficult.

[0052] (2) Low scalability and flexibility. The column aggregation logic function is customized based on each business transaction interface. When the key business links are adjusted, the core code logic will be changed, which has a wide impact.

[0053] (3) The logic is complex and not unified, and not universal. The column aggregation function is a relatively universal function that can be adapted to the general processing flow according to certain rules. It is not universal if it is implemented alone according to business rules.

[0054] (4) Service instances consume large amounts of resources. They generate many class objects, occupying memory resources and reducing the system's online transaction processing capabilities. A single interface uses a separate set of logic to create objects, and the generated classes will be stored in the JVM memory until garbage collection is triggered. The entire service uses hundreds of interfaces, which will generate many identical or similar objects, occupying system memory resources, resulting in frequent garbage collection, increasing service response time, and thus reducing the system's transaction processing capabilities and throughput.

[0055] The technical solutions provided by various embodiments of the present application are described in detail below in conjunction with the accompanying drawings.

[0056] The present application embodiment provides a general query field aggregation method for multi-source heterogeneous data, such as Figure 1 As shown, a schematic diagram of a general query field aggregation method for multi-source heterogeneous data in an embodiment of the present application is provided, and the method at least includes the following steps S110 to S140:

[0057] Considering that big data query has become a key competitive factor for enterprises, extracting key information and developing applications based on data query has become an important area of ​​enterprise informatization construction, but there are still many problems in the construction of enterprise-level unified query platform:

[0058] First, the data structure is inconsistent, the quality is uneven, and the standards are not unified. Large enterprises have complex business lines and many types of businesses, and multiple business systems often use different data collection methods and statistical calibers. When performing aggregate queries on related business data on a unified platform, the difficulty of data structure adaptation is increased, and the complexity of collaborative processing of data consistency is increased.

[0059] Second, data is scattered and information is isolated. As the old and new systems run in parallel, each business department drives different business systems, and data is scattered and stored in these systems with inconsistent architectures, inconsistent development languages, and diverse databases. Data from different systems cannot be interconnected and cannot be efficiently combined and unified, resulting in the inability to fully realize the value of data, and ultimately preventing customers from directly viewing complete and comprehensive information data.

[0060] In view of the above two types of problems, combined with the background premise that the old and new systems run in parallel and generate business data at the same time, starting from the transaction query application layer, by aggregating data from different systems and different data sources, the row and column data are processed separately to form accurate data results and fed back to the client for display. The general query field aggregation method of multi-source heterogeneous data is adopted, mainly starting from the general and efficient extension processing method of column data processing, and using the general design pattern to achieve efficient and flexible aggregation function of column data processing.

[0061] Step S110, using the task flow annotation configurator to pre-annotate and label each task chain, marking the task chain as a column aggregation processing flow task chain.

[0062] The task process annotation configurator (TaskProcessAnotation) is used, which is mainly responsible for annotating and marking each task chain, and then marking the task chain as a column aggregation processing process task chain, and configuring the name of the task chain, the service type of the task, and the task priority.

[0063] Specifically, the priority of the task execution is configured in the task annotation. When chain execution is performed, the task with the highest priority will be selected for execution first according to the priority level of the task. The service type of a task refers to the processing of different task logics, which are divided into different service types according to classification.

[0064] By annotating the task flow annotation configurator, the processor is defined as the field aggregation processing type, and automatic scanning and loading into the manager memory can be achieved.

[0065] Step S120: managing the execution chain task set with the same type of mark through the task control manager, and loading the task processor into the system cache.

[0066] like Figure 4 As shown, a task control manager (TaskHandleManager) is used, which is mainly responsible for managing a collection of tasks with the same type of tags and initializing them to the system cache at startup.

[0067] Step S130, loading the task processor in the system cache through the chain task processor, and executing the preset task based on the task flow processor interface.

[0068] like Figure 4 As shown, for the chain task processor (TaskChainHandler), it is mainly responsible for executing specific interface calls, message assembly, main source outbound call requests, sub-source outbound call requests, data callback processing, data assembly post-processing and other tasks. Secondly, the task process processor interface (TaskProcessChainHandler): mainly for each process processing chain to implement task processing according to the interface standard, and is called uniformly by the task control manager. For each process processor, a unified interface standard is followed, and different implementation logics can be used according to the different processing functions to meet the work completed by the processor.

[0069] Step S140, receiving the plurality of chained task processors through a column data aggregation executor, constructing a processor request context parameter, executing the processors in sequence and outputting an aggregation result.

[0070] like Figure 4 As shown in the figure, the DataFieldAggregateExecutor is used as the processing entry of the column data aggregation engine and can receive multiple chained task processors. The processor request context parameters are constructed, the processors are executed in sequence and the results are output. The same executor executes the function processors that have been arranged in the memory, and the context data information is continuously changed during the processing, and the latest data information is passed to the next function processor until the last function processor completes the processing and outputs the result.

[0071] Through the above method, the processing order of the function processors can be flexibly arranged and the processors can be dynamically expanded. In addition, the processing flow of field-level aggregation is unified, which improves the efficiency of aggregate query.

[0072] The above method mainly solves the problems of different access and splicing rules for column and field data of various main data sources and sub-data sources, and inconsistent data assembly methods of different business application interfaces, which lead to the inability to unify and compatible processing when splicing and aggregating columns and fields of multiple data sources, rigid rule implementation, no unified standard for splicing logic, high coupling of functional modules, and low data aggregation efficiency. The above method has advantages in functional code scalability, heterogeneous response data processing, diversity of multi-source data aggregation rules, and high efficiency of field aggregation query:

[0073] Advantages (1) Functional scalability is based on various design patterns. It can quickly implement the expansion of primary and secondary source query rules for various scenarios without invading the original logic, achieving a high degree of functional decoupling.

[0074] Advantage (2) It adopts heterogeneous response data processing, and data processing is implemented by business interface functions. The inconsistent data processing logic in various scenarios is solved through callback method, and standard unification is achieved.

[0075] Advantage (3) Maintains the diversity of data aggregation rules, supports aggregation rules of data sources in different scenarios, supports dynamic organization and assembly, and supports flexible customization of processing logic sequence.

[0076] Advantage (4) Improves field concatenation efficiency. The engine uses a thread pool to make simultaneous outbound calls to multiple data sources, obtain response result data in parallel, and process callback data in parallel, thus achieving efficient and rapid aggregation of column data.

[0077] like Figure 4 As shown, it includes a task control manager, a task flow annotation configurator, a chain task processor and an aggregation executor. The chain task processor includes but is not limited to multiple chain processors, specifically, a sub-source execution processor, a main sub-source column data aggregation task pre-processor, a main source execution processor and a multi-source summary processor. The aggregation executor includes but is not limited to preparing context data, loop execution processors and output results. The same executor is used to execute the function processors that have been arranged in the memory, and the context data information is continuously changed during the processing process, and the latest data information is passed to the next function processor until the last function processor completes the processing and outputs the result.

[0078] In one embodiment of the present application, the chain task processor is used to execute any one or more tasks including specific interface calling, message assembly, main source outbound call request, sub-source outbound call request, data callback processing, and data assembly post-processing. The method also includes: after loading the task processor in the system cache through the chain task processor, column data task processing is performed based on the task flow processor interface, and the aggregate task call method is called through an asynchronous thread, and then the task processing parameters are combined through the context of the constructed object, and each task handle processing method in the chain task collection is traversed and executed in turn until the last one is processed.

[0079] Specifically, when executing column data task processing, the aggregate task call method is called through an asynchronous thread, and the task processing parameters are combined through the context of the constructed object, and each task handle processing method in the chain task collection is traversed and executed in sequence until the last one is processed. The final result output is obtained.

[0080] As you can understand, the call method is mainly used to call a method of an object, while allowing the execution context of the method to be changed. This method is more common in JavaScript and is used to implement context switching when calling a function.

[0081] It can be understood that the handle processing method mainly involves the steps of creating, using, error handling and releasing resources, and these steps ensure that the life cycle of resources is properly managed to avoid resource leakage and other potential problems.

[0082] In one embodiment of the present application, the column data task processing based on the task flow processor interface includes one of the following: a main-sub-source column data aggregation task pre-processor, which is used to perform pre-operations on parameters, contexts, and logs before executing main-sub-source outbound calls and single-source result data processing; a main source execution processor, which uses a thread pool to receive the execution tasks of the main source, asynchronously executes and parses the execution results, constructs a standard object and puts it into the cache; a sub-source execution processor, which uses a thread pool to receive the execution tasks of the sub-source, asynchronously executes and parses the execution results, constructs a standard object and puts it into the cache; a multi-source aggregation processor, which uses a unique field to load the data information of this batch from the cache, calls back the business processing method to process and parse the data rows, and returns the parsed data in a unified manner.

[0083] When executing column data task processing based on the task flow processor interface, it includes: the primary and secondary source column data aggregation task preprocessor, which is used to perform some pre-operations on parameters, context, and logs before executing the primary and secondary source outbound calls and single source result data processing;

[0084] When executing column data task processing based on the task flow processor interface, it includes: the main source execution processor uses the thread pool to receive the execution task of the main source, and after the asynchronous execution is completed, the execution result is parsed, and the standard object is constructed and put into the cache. Construct the next outbound call task set;

[0085] When executing column data task processing based on the task flow processor interface, it includes: the sub-source execution processor uses the thread pool to receive the execution task of the sub-source, and after the asynchronous execution is completed, the execution result is parsed, and the standard object is constructed and put into the cache.

[0086] Column data task processing based on the task flow processor interface includes: multi-source aggregation processor, using unique fields to load the batch data information from the cache, calling back the business processing method, processing and parsing the data rows and returning the parsed data uniformly.

[0087] In one embodiment of the present application, a task flow annotation configurator is used to pre-annotate and label each task chain, marking the task chain as a column aggregation processing flow task chain, including: when the application service is started, the task flow annotation configurator is used to load the service objects marked with the task flow configuration annotation, and arrange them in reverse order according to the order of the annotation configuration, and load the sorted objects into the cache.

[0088] like Figure 4As shown, when the application service starts, the task flow annotation configurator loads the service objects marked with the task flow configuration annotation, arranges them in reverse order according to the order of the annotation configuration, and loads the sorted objects into the cache.

[0089] In one embodiment of the present application, a set of chained tasks with the same type of tags are managed by a task control manager, and a task processor is loaded into a system cache, including: when a business service is called, a multi-source column data aggregation task is constructed by the task control manager, and context-related data and a chained task set are passed into the column data aggregation task.

[0090] like Figure 4 As shown, when the business service is called, the multi-source column data aggregation task is constructed through the task control manager, and the context-related data and the chained task set are passed into the column data aggregation task.

[0091] In one embodiment of the present application, the task control manager is also used to scan objects annotated with configurators when the system is initialized, parse the type and order of the configuration, sort them according to different sequence values ​​of the same type, and load the sorted task processors into the system cache.

[0092] like Figure 4 As shown, when the system is initialized, the service scans the objects annotated with the configurator, parses the type and order of the configuration through the task control manager, sorts them according to different order values ​​of the same type, and loads the sorted task processors into the system cache.

[0093] In one embodiment of the present application, the main source and sub-source aggregation scenarios of the application include at least one of the following: a standard scenario of two data source column field aggregation; the main source / sub-source data is composed of multiple data sources, and the main source multiple sources can support the response data to be combined into one row, and at the same time support the combination of multiple rows of data into multiple rows; a scenario of multiple data source field aggregation.

[0094] The main source and sub-source aggregation scenarios supported in the embodiments of the present application mainly include the following three:

[0095] The first type is the standard aggregation of two data source columns and fields (primary source + sub-source), for example, A->B;

[0096] The second type is that the main source / sub-source data is composed of multiple data sources. The main source multiple sources can support the combination of response data into one row (column field aggregation of different main sources), and also support the combination of multiple rows of data into multiple rows (row data aggregation of different main sources), for example, (A+B)->C, A->(B+C), (A+B)->(C+D);

[0097] The third type is the aggregation of multiple data source fields (ultimately manifested as one main source and one sub-source). For example, the simple structure is: A->B->C (A->....->F).

[0098] like Figure 5 As shown, PBS refers to a business processing unit, MultiSource Data Task refers to the general class of the current multi-source data processing task, and the function entry of this class object is the multi-source data processing entry.

[0099] In MultiSource Data Task, the field aggregation main source preprocessor is used to call back other callback methods in PBS.

[0100] Other processors in MultiSource Data Task. Construct subtask Tasks callback method in PBS.

[0101] The main source data processor in MultiSource Data Task executes the main source outbound call and data storage. The main source storage data constructor or sub-source outbound call task constructor in PBS.

[0102] The sub-source data processor in MultiSource Data Task executes sub-source outbound calls and data storage. The sub-source storage data constructor in PBS.

[0103] The row data aggregation endpoint in MultiSource Data Task is associated with query and data aggregation. The main aggregation result processor in PBS.

[0104] The build result is returned in MultiSource Data Task.

[0105] The embodiment of the present application also provides a general query field aggregation device 600 for multi-source heterogeneous data, such as Figure 6 As shown, a schematic diagram of the structure of a general query field aggregation device for multi-source heterogeneous data in an embodiment of the present application is provided. The general query field aggregation device 600 for multi-source heterogeneous data at least includes: a task flow annotation configuration module 610, a task control management module 620, a chain task processing module 630 and a column data aggregation execution module 640, wherein:

[0106] In one embodiment of the present application, the task flow annotation configuration module 610 is specifically used to: use the task flow annotation configurator to pre-annotate and mark each task chain, and mark the task chain as a column aggregation processing flow task chain.

[0107] The task process annotation configurator (TaskProcessAnotation) is used, which is mainly responsible for annotating and marking each task chain, and then marking the task chain as a column aggregation processing process task chain, and configuring the alias of the task chain, the service type of the task, and the task priority.

[0108] By annotating the task flow annotation configurator, the processor is defined as the field aggregation processing type, and automatic scanning and loading into the manager memory can be achieved.

[0109] In one embodiment of the present application, the task control management module 620 is specifically used to: manage a set of execution chain tasks with the same type of tags through a task control manager, and load the task processor into the system cache.

[0110] like Figure 4 As shown, a task control manager (TaskHandleManager) is used, which is mainly responsible for managing a collection of tasks with the same type of tags and initializing them to the system cache at startup.

[0111] In one embodiment of the present application, the chained task processing module 630 is specifically used to: load the task processor in the system cache through the chained task processor, and execute the preset task based on the task flow processor interface.

[0112] like Figure 4 As shown, for the chain task processor (TaskChainHandler), it is mainly responsible for executing specific interface calls, message assembly, main source outbound call requests, sub-source outbound call requests, data callback processing, data assembly post-processing and other tasks. Secondly, the task process processor interface (TaskProcessChainHandler): mainly for each process processing chain to implement task processing according to the interface standard, and is called uniformly by the task control manager. For each process processor, a unified interface standard is followed, and different implementation logics can be used according to the different processing functions to meet the work completed by the processor.

[0113] In one embodiment of the present application, the column data aggregation execution module 640 is specifically used to: receive multiple chain task processors through a column data aggregation executor, construct processor request context parameters, execute the processors in sequence and output aggregation results.

[0114] like Figure 4As shown in the figure, the DataFieldAggregateExecutor is used as the processing entry of the column data aggregation engine and can receive multiple chained task processors. The processor request context parameters are constructed, the processors are executed in sequence and the results are output. The same executor executes the function processors that have been arranged in the memory, and the context data information is continuously changed during the processing, and the latest data information is passed to the next function processor until the last function processor completes the processing and outputs the result.

[0115] In one embodiment of the present application, the chain task processor is used to execute any one or more tasks of specific interface call, message assembly, main source outbound call request, sub-source outbound call request, data callback processing, and data assembly post-processing, and also includes: executing column data task processing module for

[0116] After loading the task processor in the system cache through the chain task processor, column data task processing is performed based on the task flow processor interface, and the aggregate task call method is called through an asynchronous thread. Then, the task processing parameters are combined through the context of the constructed object, and each task handle processing method in the chain task collection is traversed and executed in turn until the last one is processed.

[0117] In one embodiment of the present application, the column data task processing module is used to execute the column data task processing based on the task flow processor interface, including one of the following:

[0118] The preprocessor of the primary and secondary source column data aggregation task is used to perform pre-processing on parameters, context, and logs before executing the primary and secondary source outbound calls and single source result data processing;

[0119] The main source execution processor uses the thread pool to receive the execution tasks of the main source, asynchronously executes and completes the parsing of the execution results, constructs standard objects and puts them into the cache;

[0120] The sub-source execution processor uses the thread pool to receive the execution tasks of the sub-source, parses the execution results after the asynchronous execution is completed, and constructs a standard object and puts it into the cache;

[0121] The multi-source aggregation processor uses a unique field to load the current batch of data information from the cache, calls back the business processing method to process and parse the data rows, and returns the parsed data in a unified manner.

[0122] In one embodiment of the present application, the task flow annotation configuration module 610 is also used to

[0123] When the application service is started, the task flow annotation configurator is used to load the service objects marked with the task flow configuration annotation, and the objects are arranged in reverse order according to the order of the annotation configuration, and the sorted objects are loaded into the cache.

[0124] In one embodiment of the present application, the task control management module 620 is also used to

[0125] When the business service is called, a multi-source column data aggregation task is constructed through the task control manager, and context-related data and a chained task set are passed into the column data aggregation task.

[0126] In one embodiment of the present application, the task control management module 620 is also used to

[0127] When the system is initialized, scan the objects annotated with the configurator, parse the type and order of the configuration, sort them according to different order values ​​of the same type, and load the sorted task processors into the system cache.

[0128] In one embodiment of the present application, the main source and sub-source aggregation scenarios of the application include at least one of the following:

[0129] A standard scenario of aggregating columns of two data sources;

[0130] The main source / sub-source data is composed of multiple data sources. The main source and multiple sources can support the combination of response data into one row, and also support the combination of multiple rows of data into multiple rows.

[0131] Scenarios where fields from multiple data sources are aggregated.

[0132] It can be understood that the above-mentioned universal query field aggregation device for multi-source heterogeneous data can implement the various steps of the universal query field aggregation method for multi-source heterogeneous data provided in the aforementioned embodiments, and the relevant explanations on the universal query field aggregation method for multi-source heterogeneous data are all applicable to the universal query field aggregation device for multi-source heterogeneous data, which will not be repeated here.

[0133] Figure 7 This is a schematic diagram of the structure of an electronic device according to an embodiment of the present application. Figure 7 At the hardware level, the electronic device includes a processor, and optionally also includes an internal bus, a network interface, and a memory. The memory may include a memory, such as a high-speed random access memory (RAM), and may also include a non-volatile memory (non-volatile memory), such as at least one disk storage. Of course, the electronic device may also include hardware required for other services.

[0134] The processor, network interface and memory can be interconnected through an internal bus, which can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 7 Only one bidirectional arrow is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0135] The memory is used to store the program. Specifically, the program may include a program code, and the program code includes a computer operation instruction. The memory may include a memory and a non-volatile memory, and provides instructions and data to the processor.

[0136] The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it, forming a general query field aggregation device for multi-source heterogeneous data at the logical level. The processor executes the program stored in the memory and is specifically used to perform the following operations:

[0137] Use the task flow annotation configurator to pre-annotate each task chain and mark the task chain as a column aggregation processing flow task chain;

[0138] The task control manager manages the collection of execution chain tasks with the same type of tags and loads the task processor into the system cache;

[0139] Load the task processor in the system cache through the chain task processor, and execute the preset task based on the task flow processor interface;

[0140] The column data aggregation executor receives the plurality of chained task processors, constructs a processor request context parameter, executes the processors in sequence and outputs an aggregation result.

[0141] The above application Figure 1The method performed by the general query field aggregation device of multi-source heterogeneous data disclosed in the illustrated embodiment can be applied to a processor or implemented by a processor. The processor may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by an integrated logic circuit of hardware in the processor or an instruction in software form. The above processor may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present application can be implemented or executed. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc. The steps of the method disclosed in conjunction with the embodiments of the present application can be directly embodied as a hardware decoding processor for execution, or a combination of hardware and software modules in the decoding processor for execution. The software module can be located in a storage medium mature in the art, such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware.

[0142] The electronic device may also perform Figure 3 A method for executing a general query field aggregation device for multi-source heterogeneous data in a multi-source heterogeneous data aggregation system, and realizing a general query field aggregation device for multi-source heterogeneous data in Figure 3 The functions of the illustrated embodiment will not be described in detail in the embodiments of the present application.

[0143] The present application also provides a computer-readable storage medium, which stores one or more programs, wherein the one or more programs include instructions, which, when executed by an electronic device including multiple application programs, enable the electronic device to execute Figure 3 The method performed by the general query field aggregation device for multi-source heterogeneous data in the embodiment shown is specifically used to perform:

[0144] Use the task flow annotation configurator to pre-annotate each task chain and mark the task chain as a column aggregation processing flow task chain;

[0145] The task control manager manages the collection of execution chain tasks with the same type of tags and loads the task processor into the system cache;

[0146] Load the task processor in the system cache through the chain task processor, and execute the preset task based on the task flow processor interface;

[0147] The column data aggregation executor receives the plurality of chained task processors, constructs a processor request context parameter, executes the processors in sequence and outputs an aggregation result.

[0148] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0149] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0150] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0151] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1The steps for the functions specified in one or more boxes.

[0152] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0153] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0154] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0155] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0156] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0157] The above is only an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.

Claims

1. A general query field aggregation method for multi-source heterogeneous data, wherein: The method comprises: Use the task flow annotation configurator to pre-annotate each task chain and mark the task chain as a column aggregation processing flow task chain; The task control manager manages the collection of execution chain tasks with the same type of tags and loads the task processor into the system cache; Load the task processor in the system cache through the chain task processor, and execute the preset task based on the task flow processor interface; The column data aggregation executor receives the plurality of chained task processors, constructs a processor request context parameter, executes the processors in sequence and outputs an aggregation result.

2. The method of claim 1, wherein: The chain task processor is used to perform any one or more tasks of specific interface calling, message assembly, main source outbound call request, sub-source outbound call request, data callback processing, and data assembly post-processing. The method also includes: After loading the task processor in the system cache through the chain task processor, column data task processing is performed based on the task flow processor interface, and the aggregate task call method is called through an asynchronous thread. Then, the task processing parameters are combined through the context of the constructed object, and each task handle processing method in the chain task collection is traversed and executed in turn until the last one is processed.

3. The method of claim 2, wherein: The performing column data task processing based on the task flow processor interface includes one of the following: The preprocessor of the primary and secondary source column data aggregation task is used to perform pre-processing operations on parameters, contexts, and logs before executing the primary and secondary source outbound calls and single source result data processing; The main source execution processor uses the thread pool to receive the execution tasks of the main source, asynchronously executes and completes the parsing of the execution results, constructs standard objects and puts them into the cache; The sub-source execution processor uses the thread pool to receive the execution tasks of the sub-source, parses the execution results after the asynchronous execution is completed, and constructs a standard object and puts it into the cache; The multi-source aggregation processor uses a unique field to load the current batch of data information from the cache, calls back the business processing method to process and parse the data rows, and returns the parsed data in a unified manner.

4. The method of claim 1, wherein: Use the task flow annotation configurator to pre-annotate each task chain and mark the task chain as a column aggregation processing flow task chain, including: When the application service is started, the task flow annotation configurator is used to load the service objects marked with the task flow configuration annotation, and the objects are arranged in reverse order according to the order of the annotation configuration, and the sorted objects are loaded into the cache.

5. The method of claim 1, wherein: The task control manager manages the execution chain task collection of the same type of tags and loads the task processor into the system cache, including: When the business service is called, a multi-source column data aggregation task is constructed through the task control manager, and context-related data and a chained task set are passed into the column data aggregation task.

6. The method of claim 1, wherein: The task control manager is also used to When the system is initialized, scan the objects annotated with the configurator, parse the type and order of the configuration, sort them according to different order values ​​of the same type, and load the sorted task processors into the system cache.

7. The method of claim 1, wherein: The main source and sub-source aggregation scenarios of the application include at least one of the following: A standard scenario of aggregating columns of two data sources; The main source / sub-source data is composed of multiple data sources. The main source and multiple sources can support the combination of response data into one row, and also support the combination of multiple rows of data into multiple rows. Scenarios where fields from multiple data sources are aggregated.

8. A general query field aggregation device for multi-source heterogeneous data, wherein: The device comprises: The task flow annotation configuration module is used to use the task flow annotation configurator to pre-annotate and mark each task chain, marking the task chain as a column aggregation processing flow task chain; A task control management module is used to manage a set of execution chain tasks with the same type of tags through a task control manager, and load the task processor into the system cache; The chain task processing module is used to load the task processor in the system cache through the chain task processor and execute the preset task based on the task flow processor interface; The column data aggregation execution module is used to receive multiple chain task processors through the column data aggregation executor, construct processor request context parameters, execute the processors in sequence and output aggregation results.

9. An electronic device, comprising: processor; as well as A memory arranged to store computer executable instructions, which when executed cause the processor to perform the method of any one of claims 1 to 7.

10. A computer-readable storage medium storing one or more programs, which, when executed by an electronic device including a plurality of application programs, causes the electronic device to execute any one of the methods of claims 1 to 7.

Images