Multi-data-source multi-service system query system and method

Abstract

The invention discloses a multi-data-source multi-service system query system and method, belongs to the technical field of multi-source data management, and aims to solve the technical problem of how to realize data sharing and efficient query between multiple data sources and multiple systems. The structure dynamic management module is used for managing and standardizing data tables and fields among the business systems, recording operation logs of the business systems through an operation log table, and recording lookup indexes of the data among the business systems through a definition index table; the data real-time synchronization engine is used for synchronizing an operation log of the business system to the operation log table and updating the public data table based on the operation log; the accurate data conversion tool is used for converting business data in the changed business system into a format adaptive to other business systems based on the operation log; the stable data writing module is used for writing the business data after format conversion into a system local data table of the corresponding business system; and the efficient retrieval module is used for providing query display of the business data.

Description

Technical Field

[0001] The present invention relates to the technical field of multi-source data management, and in particular to a multi-data source multi-business system query system and method. Background Art

[0002] At present, most of them use sharding components, large business tables or synchronization tools to synchronize data from multiple data sources multiple times to solve the current problem. However, since business tables generally use relational databases, when a certain amount of data is reached, the relational database may cause performance degradation and cannot effectively solve the problem of massive data. In some specific scenarios, sharding components may not be applicable and the performance is poor. In addition, when there are duplicate or cross-relationships between two business system data tables and the data table fields are different, the business system assembles them through large business tables. However, the merger of large business tables is only a union of fields, not a unification of fields. When fields in the business system data table are added, modified, or deleted, the large business table needs to be updated synchronously, and the field's autonomous scalability is poor.

[0003] How to achieve data sharing and efficient query among multiple data sources and multiple systems is a technical problem that needs to be solved. Summary of the invention

[0004] The technical task of the present invention is to address the above deficiencies and provide a multi-data source and multi-business system query system and method to solve the technical problem of how to achieve data sharing and efficient query between multiple data sources and multiple systems.

[0005] In a first aspect, the present invention provides a multi-data source multi-business system query system for realizing data sharing and data query between multiple business systems, the system comprising an interconnected data real-time synchronization engine, an efficient retrieval module, a structure dynamic management module, an accurate data conversion tool, a stable data writing module, a registration module and a public pool;

[0006] The registration module interacts with the outside world through a visual registration interface to support registration of the business system;

[0007] The public pool is configured with a public data table, which is used to store business data shared between various business systems;

[0008] The structure dynamic management module is used to manage and standardize the data tables and fields between various business systems, to record the system information, data source information, data table dictionary, field dictionary of each business system, and the correspondence between the business system, data table and field, to record the operation log of each business system through the operation log table, to record the search index of data between various business systems through the definition index table, and for the business system where the business data has changed, the operation log is used to achieve data synchronization between the public data table and the local data table of other business systems;

[0009] The real-time data synchronization engine is used to synchronize the operation log of the business system to the operation log table, and update the public data table based on the operation log;

[0010] The precise data conversion tool is used to convert the business data in the changed business system into a format that is compatible with other business systems based on the operation log;

[0011] The stable data writing module is used to update the public data table based on the operation log, and write the business data after format conversion into the local data table of the corresponding business system;

[0012] The efficient retrieval module is used to provide query display of business data.

[0013] Preferably, based on the definition index table, the efficient retrieval module is used to provide query operations for business data, including association, merging and aggregation of business data, query field values ​​that meet business conditions based on the definition index table, query data conversion query results for all fields in the target data table according to the data table dictionary, and query the corresponding business system according to the data source and feed back the query results to the target business system.

[0014] Preferably, the structure dynamic management module is used to support the business system to redefine the table structure of the data table;

[0015] When there is an intersection between data tables of business systems and the data is different, it is used to merge the intersecting data tables into one data table and redefine the merged data table;

[0016] The structure dynamic management module is used to merge multiple data tables of a business system into one data table and redefine the merged data table;

[0017] When each business system adds new data, the structure dynamic management module, data real-time synchronization engine, precise data conversion tool and stable data writing module work together to synchronize the new data into the merged data table, and synchronize the new data into the local data tables of other business systems.

[0018] Preferably, the business system detects the changed data in its data table, generates an operation log for the changed data, and sends a data synchronization signal to the real-time data synchronization engine, the precise data conversion tool, and the stable data writing module based on the operation log;

[0019] Based on the data synchronization signal, the real-time data synchronization engine is used to write the operation log into the operation log table, write the changed data into the definition index table, and send the changed data to the public data table. The precise data conversion tool is used to convert the changed data into a format compatible with the business system. Correspondingly, the stable data writing module is used to write the changed data after format conversion into the corresponding business system local data table.

[0020] Preferably, the stable data writing module is used to query the routing storage table information and field information in the structure dynamic management module, and to add, modify and delete data;

[0021] Among them, the routing storage information table is used to record the interface information of each business system.

[0022] In a second aspect, the present invention provides a multi-data source business system query method, which is used to realize data sharing and data query between multiple business systems through a multi-data source business system query system as described in any one of the first aspects, and the method comprises the following steps:

[0023] The dynamic management module is used to manage and standardize the data tables and fields between each business system, record the system information, data source information, data table dictionary, field dictionary, and the correspondence between the business system, data table, and field of each business system, record the operation log of each business system through the operation log table in the dynamic management module, and record the search index of data between each business system through the index table defined in the dynamic management module. For business systems with business data changes, the operation log is used to achieve data synchronization between public data tables and local data tables of other business systems;

[0024] The real-time data synchronization engine is used to synchronize the operation logs of the business system to the operation log table, and the public data table is updated based on the operation logs.

[0025] Based on the operation log, the business data in the changed business system is converted into a format that is compatible with other business systems through precise data conversion tools;

[0026] Based on the operation log, the public data table is updated through the stable data writing module, and the business data after format conversion is written into the local data table of the corresponding business system;

[0027] Provide query and display of business data through efficient retrieval module.

[0028] Preferably, based on the definition index table and efficient retrieval module, query operations for business data are provided, including association, merging and aggregation of business data. Field values ​​that meet business conditions are queried based on the definition index table, and data conversion query results for all fields in the target data table are queried based on the data table dictionary. The corresponding business system is queried based on the data source and the query results are fed back to the target business system.

[0029] As a preferred embodiment, the structure dynamic management module supports the business system to redefine the table structure of the data table;

[0030] When there is an intersection between the data tables of the business systems and the data is different, the cross-data tables are merged into one data table through the structure dynamic management module, and the merged data table is redefined;

[0031] Through the structure dynamic management module, multiple data tables of a business system can be merged into one data table, and the merged data table can be redefined;

[0032] When each business system adds new data, the structure dynamic management module, real-time data synchronization engine, precise data conversion tool and stable data writing module work together to synchronize the new data to the merged data table, and synchronize the new data to the local data tables of other business systems.

[0033] Preferably, the business system detects the changed data in its data table, generates an operation log for the changed data, and sends a data synchronization signal to the real-time data synchronization engine, the precise data conversion tool, and the stable data writing module based on the operation log;

[0034] Based on the data synchronization signal, the operation log is written into the operation log table through the real-time data synchronization engine, the changed data is written into the definition index table, and the changed data is sent to the public data table. The changed data is converted into a format compatible with the business system through the precise data conversion tool. Correspondingly, the changed data after format conversion is written into the corresponding business system local data table through the stable data writing module.

[0035] As a preferred method, the stable data writing module queries the routing storage table information and field information in the structure dynamic management module, and performs addition, modification and deletion operations on the data;

[0036] Among them, the routing storage information table is used to record the interface information of each business system.

[0037] The multi-data source business system query system and method of the present invention have the following advantages:

[0038] 1. Realize the storage of massive data, provide unified SDK query to realize fuzzy database type, database table joint query, multi-condition query, and improve query efficiency;

[0039] 2. Support the expansion of data table fields in multiple data sources, and support mutual recognition and conversion between multiple systems when the same field names are different;

[0040] 3. Multiple systems share data with each other, and business operations can directly change the data of related systems;

[0041] 4. The query method platform for multiple data sources and multiple systems and the application system provide an interface management page, and the application system can choose to call the interface according to needs;

[0042] 5. Apply low-code development and powerful one-click generation function (including controllers, models, views, menus, etc.;

[0043] 6. Support multiple data sources and multiple systems. Switching multiple data sources and multi-system queries can be realized through simple configuration;

[0044] 7. Physical isolation of data layer and business layer data can be achieved. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0046] The present invention is further described below in conjunction with the accompanying drawings.

[0047] Figure 1 This is a structural block diagram of a multi-data source multi-business system query system in Example 1;

[0048] Figure 2 This is a query data flow diagram corresponding to query mode 1 in a multi-data source multi-business system query system in Example 1;

[0049] Figure 3 This is a query data flow diagram corresponding to query mode 2 in a multi-data source multi-business system query system in embodiment 1;

[0050] Figure 4 This is a query data flow diagram corresponding to query mode 3 in a multi-data source multi-business system query system in embodiment 1;

[0051] Figure 5 This is an illustration of a data table in a structure dynamic management module in a multi-data source multi-business system query system in Example 1;

[0052] Figure 6This is an illustration of the index table data table n:1 in a multi-data source multi-business system query system in Example 1. DETAILED DESCRIPTION

[0053] The present invention is further described below in conjunction with the accompanying drawings and specific embodiments so that those skilled in the art can better understand the present invention and implement it. However, the embodiments are not intended to limit the present invention. In the absence of conflict, the embodiments of the present invention and the technical features in the embodiments may be combined with each other.

[0054] The embodiment of the present invention provides a multi-data source and multi-business system query system and method, which are used to solve the technical problem of how to achieve data sharing and efficient query among multiple data sources and multiple systems.

[0055] Embodiment 1:

[0056] The present invention discloses a multi-data source and multi-business system query system, which is used to realize data sharing and data query among multiple business systems. The system comprises an interconnected data real-time synchronization engine, an efficient retrieval module, a structure dynamic management module, an accurate data conversion tool, a stable data writing module, a registration module and a public pool.

[0057] The registration module interacts with the outside world through a visual registration interface to support registration of business systems.

[0058] The public pool is configured with public data tables, which are used to store business data shared among various business systems.

[0059] The structural dynamic management module is used to manage and standardize the data tables and fields between various business systems, to record the system information, data source information, data table dictionary, field dictionary of each business system, and the correspondence between business systems, data tables and fields, to record the operation logs of each business system through the operation log table, and to record the search index of data between various business systems by defining the index table. For business systems where business data changes occur, the operation log is used to achieve data synchronization between public data tables and local data tables of other business systems.

[0060] In this embodiment, the structure dynamic management module is used to support the business system to redefine the table structure of the data table; when there is an intersection between the data tables of the business systems and the data is different, it is used to merge the intersecting data tables into one data table and redefine the merged data table. The structure dynamic management module is used to merge multiple data tables of a business system into one data table and redefine the merged data table; when each business system adds new data, the structure dynamic management module, the real-time data synchronization engine, the precise data conversion tool and the stable data writing module cooperate to synchronize the new data into the merged data table, and synchronize the new data into the local data tables of other business systems.

[0061] The real-time data synchronization engine is used to synchronize the operation logs of the business system to the operation log table and update the public data table based on the operation logs.

[0062] The precise data conversion tool is used to convert the business data in the changed business system into a format that is compatible with other business systems based on the operation log.

[0063] The stable data writing module is used to update the public data table based on the operation log, and write the business data after format conversion into the local data table of the corresponding business system.

[0064] In this embodiment, the stable data writing module is used to query the routing entry table information and field information in the structure dynamic management module, and to add, modify and delete data; wherein the routing entry information table is used to record the interface information of each business system.

[0065] The efficient retrieval module is used to provide query and display of business data.

[0066] As a specific implementation of the efficient retrieval module, based on the definition index table, the efficient retrieval module is used to provide query operations for business data, including association, merging and aggregation of business data, and to query field values ​​that meet business conditions based on the definition index table. It is used to query all fields in the target data table according to the data table dictionary to convert data query results, and is used to query the corresponding business system according to the data source and feedback the query results to the target business system.

[0067] In this embodiment, the defined index table can be directly queried for data query, and query operations such as association, merging, and aggregation can be performed to query the data field values ​​that meet the business conditions. The query results for data conversion of all fields in the table can be queried based on the query data table dictionary, or further based on the data ID set, based on the query source data source set, and the query results can be fed back to the business module.

[0068] In this embodiment, the business system detects the changed data in its data table, generates an operation log for the changed data, and sends a data synchronization signal to the real-time data synchronization engine, the precise data conversion tool, and the stable data writing module based on the operation log; based on the data synchronization signal, the real-time data synchronization engine is used to write the operation log into the operation log table, write the changed data into the definition index table, and send the changed data to the public data table, the precise data conversion tool is used to convert the changed data into a format compatible with the business system, and correspondingly, the stable data writing module is used to write the changed data after format conversion into the corresponding business system local data table.

[0069] The system of this embodiment performs the following operations before implementing the query:

[0070] Step 1: In the structure dynamic management unit, configure the system information, data source information, data table dictionary, and field dictionary of each business system, and standardize the data source, data table, and data field between business systems.

[0071] Step 2: Configure the correspondence between business systems, data tables, and fields in the structure dynamic management unit. The business system can redefine the table structure. When the data tables between data sources have intersections and the data are different, the intersecting data tables need to be merged into one data table, and the data table name of the merged table can be redefined. Multiple tables in one data source can be merged into one table in a multi-data source and multi-system query platform, and the data table name of the merged table can be redefined. When each business system adds new data, the data needs to be synchronized to the merged table. Multiple business systems are synchronized, and the merged table obtains a complete query data. There is no intersection between data source data tables, so there is no need to merge. The data source data table can be split into multiple business tables on a multi-data source and multi-system query platform, and the data table name of the split table can be redefined. There are many common fields in separate tables to improve utilization efficiency.

[0072] Step 3: Configure the field correspondence information in the structure dynamic management unit. The business system can redefine the field name and type. The data sources express the same meaning, but the field names and types are not unified. All data cannot be queried through one field. The system achieves field unification through field standardization, and can query all the same field data at once.

[0073] The system performs the following operations to achieve data sharing and query:

[0074] Step 1: Stable data writing system has a registered business module, and the system needs to be registered. During the operation execution, the business data is written into the corresponding data source and a synchronous data signal is sent to the multi-data source, multi-system query method platform and application system, and the synchronous data is written into the operation log table.

[0075] Step 2: When data changes are detected in the business system, the specific data that has changed can be located. Through the carefully designed query structure dynamic management unit, these changed data will be intelligently routed to the registered system, and the data structure will be defined in the system, the query results will be converted, and finally these updates will be synchronized to the defined index table.

[0076] Step 3: Query the query model interface information provided by the system in advance, find the query model that meets the needs, submit a query request to the system, query the corresponding query model interface method in the efficient retrieval module, and query the corresponding definition index table to obtain the query results.

[0077] Step 4: The system converts and splices the original data of the data result according to the format required for conversion by querying the data, and provides the processed data to the query platform.

[0078] Embodiment 2:

[0079] The present invention provides a multi-data source business system query method, which realizes data sharing and data query between multiple business systems through the system disclosed in Example 1. The method includes the following steps:

[0080] Step S100: Manage and standardize the data tables and fields between the business systems through the dynamic management module, record the system information, data source information, data table dictionary, field dictionary of each business system, and the correspondence between the business system, data table and field, record the operation log of each business system through the operation log table in the dynamic management module, and record the search index of data between each business system through the index table defined in the dynamic management module. For the business system where the business data changes, the operation log is used to realize the data synchronization of the public data table and the local data tables of other business systems.

[0081] The structure dynamic management module of this embodiment supports the business system to redefine the table structure of the data table; when there is an intersection between the data tables of the business systems and the data is different, the cross-data tables are merged into one data table through the structure dynamic management module, and the merged data table is redefined; multiple data tables of a business system can be merged into one data table through the structure dynamic management module, and the merged data table is redefined; when each business system adds new data, the structure dynamic management module, the real-time data synchronization engine, the precise data conversion tool and the stable data writing module cooperate to synchronize the new data to the merged data table, and synchronize the new data to the local data tables of other business systems.

[0082] Step S200: synchronize the operation log of the business system to the operation log table through the real-time data synchronization engine, and update the public data table based on the operation log.

[0083] Step S300: Based on the operation log, the business data in the changed business system is converted into a format that is compatible with other business systems through an accurate data conversion tool.

[0084] Step S400: Based on the operation log, the public data table is updated through the stable data writing module, and the business data after format conversion is written into the local data table of the corresponding business system.

[0085] Step S500: providing query display of business data through an efficient retrieval module.

[0086] In step S500 of this embodiment, the stable data writing module queries the routing entry table information and field information in the structure dynamic management module, and performs addition, modification and deletion operations on the data; wherein the routing entry information table is used to record the interface information of each business system.

[0087] Among them, based on the definition index table and efficient retrieval module, query operations of business data are provided, including association, merging and aggregation of business data. Field values ​​that meet business conditions are queried based on the definition index table, and data conversion query results of all fields in the target data table are queried according to the data table dictionary. The corresponding business system is queried according to the data source and the query results are fed back to the target business system.

[0088] The business system detects the changed data in its data table, generates an operation log for the changed data, and sends a data synchronization signal to the data real-time synchronization engine, the precise data conversion tool, and the stable data writing module based on the operation log; based on the data synchronization signal, the operation log is written into the operation log table through the data real-time synchronization engine, the changed data is written into the definition index table, and the changed data is sent to the public data table, and the changed data is converted into a format compatible with the business system through the precise data conversion tool. Correspondingly, the changed data after format conversion is written into the corresponding business system local data table through the stable data writing module.

[0089] The present invention has been shown and described in detail above through the accompanying drawings and preferred embodiments. However, the present invention is not limited to these disclosed embodiments. Based on the above multiple embodiments, those skilled in the art can know that the means in the above different embodiments can be combined to obtain more embodiments of the present invention, and these embodiments are also within the protection scope of the present invention.

Claims

1. A multi-data source multi-business system query system, characterized in that: Used to realize data sharing and data query between multiple business systems, the system includes an interconnected data real-time synchronization engine, an efficient retrieval module, a structure dynamic management module, an accurate data conversion tool, a stable data writing module, a registration module and a public pool; The registration module interacts with the outside world through a visual registration interface to support registration of the business system; The public pool is configured with a public data table, which is used to store business data shared between various business systems; The structure dynamic management module is used to manage and standardize the data tables and fields between various business systems, to record the system information, data source information, data table dictionary, field dictionary of each business system, and the corresponding relationship between the business system, data table and field, to record the operation log of each business system through the operation log table, to record the search index of data between various business systems through the definition index table, and for the business system where the business data has changed, the operation log is used to realize the data synchronization of the public data table and the local data table of other business systems; The real-time data synchronization engine is used to synchronize the operation log of the business system to the operation log table, and update the public data table based on the operation log; The precise data conversion tool is used to convert the business data in the changed business system into a format that is compatible with other business systems based on the operation log; The stable data writing module is used to update the public data table based on the operation log, and write the business data after format conversion into the local data table of the corresponding business system; The efficient retrieval module is used to provide query display of business data.

2. The multi-data source multi-business system query system according to claim 1, characterized in that: Based on the definition index table, the efficient retrieval module is used to provide query operations for business data, including association, merging and aggregation of business data, query field values ​​that meet business conditions based on the definition index table, query data conversion query results for all fields in the target data table according to the data table dictionary, and query the corresponding business system according to the data source and feedback the query results to the target business system.

3. The multi-data source multi-business system query system according to claim 1, characterized in that: The structure dynamic management module is used to support the business system to redefine the table structure of the data table; When there is an intersection between data tables of business systems and the data is different, it is used to merge the intersecting data tables into one data table and redefine the merged data table; The structure dynamic management module is used to merge multiple data tables of a business system into one data table and redefine the merged data table; When each business system adds new data, the structure dynamic management module, data real-time synchronization engine, precise data conversion tool and stable data writing module work together to synchronize the new data into the merged data table, and synchronize the new data into the local data tables of other business systems.

4. The multi-data source multi-business system query system according to claim 1, characterized in that: The business system detects the changed data in its data table, generates an operation log for the changed data, and sends a data synchronization signal to the real-time data synchronization engine, the precise data conversion tool, and the stable data writing module based on the operation log; Based on the data synchronization signal, the real-time data synchronization engine is used to write the operation log into the operation log table, write the changed data into the definition index table, and send the changed data to the public data table. The precise data conversion tool is used to convert the changed data into a format compatible with the business system. Correspondingly, the stable data writing module is used to write the changed data after format conversion into the corresponding business system local data table.

5. The multi-data source multi-business system query system according to claim 1, characterized in that: The stable data writing module is used to query the routing storage table information and field information in the structure dynamic management module, and to add, modify and delete data; Among them, the routing storage information table is used to record the interface information of each business system.

6. A multi-data source business system query method, characterized in that: The method is used to realize data sharing and data query between multiple business systems through a multi-data source business system query system as described in any one of claims 1 to 5, and the method comprises the following steps: The dynamic management module is used to manage and standardize the data tables and fields between each business system, record the system information, data source information, data table dictionary, field dictionary, and the correspondence between the business system, data table, and field of each business system, record the operation log of each business system through the operation log table in the dynamic management module, and record the search index of data between each business system through the index table defined in the dynamic management module. For business systems with business data changes, the operation log is used to achieve data synchronization between public data tables and local data tables of other business systems; The real-time data synchronization engine is used to synchronize the operation logs of the business system to the operation log table, and the public data table is updated based on the operation logs. Based on the operation log, the business data in the changed business system is converted into a format that is compatible with other business systems through precise data conversion tools; Based on the operation log, the public data table is updated through the stable data writing module, and the business data after format conversion is written into the local data table of the corresponding business system; Provide query and display of business data through efficient retrieval module.

7. The multi-data source business system query method according to claim 6, characterized in that: Based on the definition index table and efficient retrieval module, business data query operations are provided, including the association, merging and aggregation of business data. Field values ​​that meet business conditions are queried based on the definition index table. All fields in the target data table are queried based on the data table dictionary to convert the query results of the data. The corresponding business system is queried based on the data source and the query results are fed back to the target business system.

8. The multi-data source business system query method according to claim 6, characterized in that: The structure dynamic management module supports the business system to redefine the table structure of the data table; When there is an intersection between the data tables of the business systems and the data is different, the cross-data tables are merged into one data table through the structure dynamic management module, and the merged data table is redefined; Through the structure dynamic management module, multiple data tables of a business system can be merged into one data table, and the merged data table can be redefined; When each business system adds new data, the structure dynamic management module, real-time data synchronization engine, precise data conversion tool and stable data writing module work together to synchronize the new data to the merged data table, and synchronize the new data to the local data tables of other business systems.

9. The multi-data source business system query method according to claim 6, characterized in that: The business system detects the changed data in its data table, generates an operation log for the changed data, and sends a data synchronization signal to the real-time data synchronization engine, the precise data conversion tool, and the stable data writing module based on the operation log; Based on the data synchronization signal, the operation log is written into the operation log table through the real-time data synchronization engine, the changed data is written into the definition index table, and the changed data is sent to the public data table. The changed data is converted into a format compatible with the business system through the precise data conversion tool. Correspondingly, the changed data after format conversion is written into the corresponding business system local data table through the stable data writing module.

10. The multi-data source business system query method according to claim 6, characterized in that: Through the stable data writing module, query the routing table information and field information in the structure dynamic management module, and add, modify and delete data; Among them, the routing storage information table is used to record the interface information of each business system.

Images