Method, system, electronic device and computer-readable medium for acquiring associated data at one time

Abstract

A data aggregation acquisition method is used to obtain all called data in multiple database tables at once according to the calling relationship between multiple microservices that call each other. The method includes: configuring multiple configuration items, configuring the calling operation of each microservice on each database table through multiple configuration items, each configuration item includes a calling template and the corresponding database table information; configuring the calling template, specifying the data to be called by the calling operation corresponding to the corresponding configuration item in the calling template; and data aggregation acquisition, based on the calling templates and the corresponding database table information of the multiple configuration items and the multiple database tables, all the called data are obtained in an aggregated manner. The above method of the present invention can obtain all related data in multiple exclusive databases corresponding to different microservices in an aggregated manner at one time through configuration, which is conducive to quickly and accurately viewing data and improving data analysis efficiency.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a data acquisition method, system, electronic device and computer-readable medium for acquiring all called data in all databases involved in multiple microservices that have a mutual calling relationship at one time. Background Art

[0002] In recent years, Internet business has developed rapidly, and distributed systems have gradually become standard. In a distributed system, many microservices often call each other to complete a business. In order to avoid the complexity of operation and maintenance, multiple microservices usually use different database tables under the same database. As the business develops, there are more and more database tables under a database, and it is difficult for technicians to distinguish which database tables are actually used by a business, which greatly affects the efficiency of data analysis.

[0003] To address the above issues, there is currently a database partitioning technology, which is to split the database according to the microservice dimension, and put the database table used by a microservice into a dedicated database. By logging into the database through visualization software, you can view all the data in all the database tables. Alternatively, you can enter and view it through the database connection command. Summary of the invention

[0004] Technical issues

[0005] However, as the business becomes more and more complex, the number of libraries and tables in a microservice's dedicated database will continue to increase. When a business is associated with multiple tables, and the data in the tables are dependent and form a deep tree structure, if you need to check whether the business data is normal, you often need to write complex calls and manually judge the data. This is a large workload, inefficient, and easy to miss. It is impossible to efficiently and intuitively check the libraries and tables affected by a specific business, and the complexity and operation and maintenance costs are also increased.

[0006] The data produced by actual business is often distributed in different databases and different database tables, and there is a direction and dependency between the data, and the dependency also has a sequence. Therefore, when analyzing data, it is necessary to be able to connect related data across different databases based on certain configuration rules for visualization to meet the needs of different business customization.

[0007] In view of this, the present invention provides a data aggregation acquisition method, which aggregates all related data in different databases or library tables at one time through a visual configuration, so that even in business scenarios with complex calling relationships, all related data can be accurately extracted and visualized, thereby improving analysis efficiency.

[0008] Problem Solution

[0009] According to one aspect of the present invention, a data aggregation acquisition method is provided, which is used to obtain all called data in multiple database tables at one time according to the calling relationship between multiple microservices that call each other, and is characterized in that the method includes:

[0010] Configure multiple configuration items, and configure the calling operation of each microservice on each database table through the multiple configuration items, each of the configuration items including a calling template and the corresponding database table information;

[0011] Configuring the calling template, specifying in the calling template the data to be called by the calling operation corresponding to the corresponding configuration item; and

[0012] Data aggregation acquisition: based on the calling templates of the multiple configuration items and the corresponding database table information and the multiple database tables, all the called data are obtained in an aggregated manner.

[0013] Optionally, according to a method of one aspect of the present invention, the calling template is a SQL template, and configuring the calling template includes specifying, in a custom part of the SQL template, data to be called by the calling operation corresponding to the corresponding configuration item.

[0014] Optionally, according to a method of one aspect of the present invention, each of the configuration items further includes a unique number, and configuring the call template further includes specifying the number in a custom part of the SQL template to specify a configuration item associated with the corresponding configuration item, thereby establishing the call relationship.

[0015] Optionally, according to a method of one aspect of the present invention, configuring the call template further comprises specifying field information in a custom part of the SQL template to specify the field of specific data to be acquired.

[0016] Optionally, according to a method of one aspect of the present invention, standard SQL is generated by replacing the custom part of the SQL template with target data according to predetermined parsing rules, and the target data is loaded based on the standard SQL, thereby obtaining the data to be called by the calling operation corresponding to the corresponding configuration item.

[0017] Optionally, according to a method of one aspect of the present invention, the step of obtaining all the called data in an aggregated manner based on the calling templates of the multiple configuration items and the corresponding database table information and the multiple database tables comprises:

[0018] The plurality of configuration items are stored as a complete set of configurations in a business database of the business that is to call all of the microservices, and

[0019] When processing the service, the entire set of configurations is loaded from the service database for processing.

[0020] Optionally, according to a method of one aspect of the present invention, the step of obtaining all the called data in an aggregated manner based on the calling templates of the multiple configuration items and the corresponding database table information and the multiple database tables comprises:

[0021] Each of the configuration items loads data sets from the multiple database tables according to its own database table information and stores the data sets in a data structure in the memory for calling by other configuration items.

[0022] According to another aspect of the present invention, a data aggregation acquisition system is provided, which is used to acquire all called data in multiple database tables at once according to the calling relationship between multiple microservices that call each other, characterized in that the system includes:

[0023] A configuration item configuration module, used to configure multiple configuration items, and to configure the calling operation of each microservice on each database table through the multiple configuration items, each of the configuration items including a calling template and the corresponding database table information;

[0024] A calling template configuration module, which specifies in the calling template the data to be called by the calling operation corresponding to the corresponding configuration item; and

[0025] The data aggregation acquisition module aggregates and obtains all the called data based on the calling templates of the multiple configuration items, the corresponding database table information, and the multiple database tables.

[0026] According to another aspect of the present invention, there is provided an electronic device having a kitchenware function, comprising:

[0027] one or more processors; and

[0028] a storage device for storing one or more programs,

[0029] When the one or more programs are executed by the one or more processors, the one or more processors implement the data aggregation acquisition method described in any one of the above aspects of the present invention.

[0030] According to another aspect of the present invention, there is provided a computer-readable medium having a computer program stored thereon, wherein when the program is executed by a processor, the program implements any one of the data aggregation acquisition methods according to one aspect of the present invention.

[0031] Advantageous Effects of the Invention

[0032] According to the data aggregation acquisition method of the present invention, by configuring each call operation with each microservice in the visualization panel in the form of configuration items, it is possible to configure the calls between data with complex dependencies in different database tables involved in multiple microservices as a whole, so as to achieve one-time aggregation acquisition of all related data in different microservices and multiple databases and perform visual display, which is conducive to fast and accurate viewing of data and improving data analysis efficiency. In addition, through the above-mentioned configuration method, the system is easy to build, supports fast deletion and addition, and has high scalability. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] The accompanying drawings are used to better understand the present invention and do not constitute an improper limitation of the present invention.

[0034] Figure 1 is a schematic diagram of the main process of a data aggregation acquisition method according to a preferred embodiment of the present invention;

[0035] Figure 2 is a diagram of instances of the configuration items of the visualization panel;

[0036] Figure 3 is a schematic diagram showing the SQL template of the configuration item and the dependency relationship of the configuration item;

[0037] Figure 4 An example of data acquired by using a data aggregation acquisition method according to a preferred embodiment of the present invention is shown;

[0038] Figure 5 It is a schematic diagram of main modules of a data aggregation acquisition system according to a preferred embodiment of the present invention. DETAILED DESCRIPTION

[0039] The technical solution of the present invention will be clearly and completely described below in conjunction with the specific embodiments of the present invention and the corresponding drawings. Obviously, the described embodiments are only part of the preferred embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0040] Figure 1It is a schematic diagram of the main process of the data aggregation acquisition method according to a preferred embodiment of the present invention. The data aggregation acquisition method according to this embodiment mainly includes the following steps: configuring multiple configuration items, each of which corresponds to a call operation of a microservice to a database table, and each of which includes a call template and the corresponding database table information; configuring the call template, specifying the data to be called by the call operation corresponding to the corresponding configuration item in the call template; data aggregation acquisition, based on the call templates of the multiple configuration items and the corresponding database table information and the multiple database tables, all the called data are obtained in an aggregated manner.

[0041] The steps of the data aggregation acquisition method according to this embodiment are described in detail below.

[0042] First, for businesses that call multiple microservices, set up a set of visualization panels based on the different data sources associated with the business scenarios. Figure 2 As shown in the figure, the visualization panel includes multiple configuration items. Each configuration item corresponds to a call operation of a microservice to a database table, and contains the assigned unique number, call template, database table information and data description, etc. Note that there is no strict one-to-one correspondence between configuration items and microservices. The core target of configuration items is library tables. If a microservice involves multiple library tables, that is, a microservice uses multiple database tables, then the microservice is correspondingly configured with multiple configuration items, each of which is used to obtain data from a database table.

[0043] In this embodiment, an SQL template is used as a calling template, and supports references to data sets between different configuration items in a numbered manner. An SQL template is a calling template that can be parsed by a syntax parser, which includes a custom part inserted in SQL, that is, embedding special non-SQL official symbols in SQL statements, and finally obtaining standard SQL to obtain data by parsing these symbols into the desired data using a syntax parser.

[0044] like Figure 3As shown, the custom syntax type of the SQL template is, for example, {number.field}, and the part enclosed by "{}" represents the custom part, and the content is "number.field". "Number" refers to the unique number assigned to the configuration item on which this call operation depends, and "field" is the field contained in the database and / or library table to which the configuration item corresponding to the number belongs. Through "number.field", the data set of the database to be referenced can be pointed to. In addition to "{}", unofficial symbols that can be used as custom parts also include some other custom symbols, such as "if". The reason for using unofficial symbols is to avoid replacing valid SQL syntax keywords, so that only by replacing the custom unofficial symbols, the entire template can be translated into a standard SQL statement for execution. In addition, the above-mentioned parsing refers to finding the custom part in the SQL template through unofficial symbols, replacing it with the target data to be called according to the predetermined parsing rules and backfilling it into the SQL template to generate standard SQL, so as to further load the required data.

[0045] It should be noted that the references to the numbers in the SQL templates of configuration items with different numbers do not have to be sequential, but can cross-reference any existing numbers, but there cannot be a circular reference.

[0046] After completing the configuration of the SQL template for each configuration item, the reference relationship between the data of all configuration items is created. Then, all the configured configuration items are stored in a specific business database as a complete set of configurations for the business to which they belong. When the business needs to be processed, the complete set of configurations is loaded from the business database into the processing system through the business type field of the business for subsequent processing.

[0047] In business processing, for example, when viewing the data of a business, the processing system loads the entire set of configurations corresponding to the business from the business database to obtain all configuration items involved in the business. The system can parse the SQL templates of each configuration item based on the configuration item information, such as the number, the database table information, the database description, etc., to obtain the data reference relationship between microservices. Subsequently, the system loads the called target data according to the database to which each configuration item belongs, and stores it in the data structure in the memory for other configuration items to call. Finally, the system outputs the results of all configuration item parsing to the visualization system to aggregate and display the related data of all microservice calls involved in the current business at one time.

[0048] Since each configuration item may read different databases, the actual parsing will read data according to the database specified during configuration. For data whose referenced objects do not exist, it will be implicitly considered that the result data set does not exist, indicating that the business has not actually occurred at this stage and no data has been generated.

[0049] Figure 4 An example of data obtained by applying the data aggregation acquisition method according to this embodiment is shown. In this example, first, the specific configuration items of the microservices involved are selectively added by configuring the visualization panel, and the SQL template, the name of the database to which it belongs, the data usage description, etc. of each configuration item are configured. Then, in the processing system, based on the information of each configuration item and each database, a well-known syntax parser, such as SQLParser, is used to parse the SQL template into a specific target data set. Finally, all parsed data sets are output as result sets to the visualization system for display. The display result is as follows: Figure 4 shown.

[0050] The data aggregation processing method according to the preferred embodiment of the present invention is described above.

[0051] According to the data aggregation acquisition method of the present invention, by configuring one or more configuration items of each microservice in the visualization panel, the data reference relationship between microservices can be configured as a whole, so as to achieve a one-time aggregation acquisition of all related data in multiple databases corresponding to different microservices and perform visual display, thereby facilitating fast and accurate data viewing and improving data analysis efficiency. In addition, through the above-mentioned configuration method, the system is easy to build, supports fast deletion and addition, and has high scalability.

[0052] The present invention also provides a data aggregation acquisition system. Figure 5 As shown, the system mainly includes:

[0053] A configuration item configuration module, used to configure multiple configuration items, and to configure the calling operation of each microservice on each database table through the multiple configuration items, each of the configuration items including a calling template and the corresponding database table information;

[0054] A calling template configuration module, which specifies in the calling template the data to be called by the calling operation corresponding to the corresponding configuration item; and

[0055] The data aggregation acquisition module aggregately obtains all the called data based on the calling templates of the multiple configuration items and the corresponding database tables and multiple database tables.

[0056] As another aspect, the present invention further provides an electronic device with a data aggregation acquisition function, comprising: one or more processors; and a storage device for storing one or more programs. When the one or more programs are executed by the one or more processors, the electronic device enables the one or more processors to implement the data aggregation acquisition method as described in the preferred embodiment of the present invention.

[0057] As another aspect, the present invention also provides a computer-readable medium, which carries one or more programs. When the above one or more programs are executed by a device, the program is executed by a processor to implement the data aggregation acquisition method as described in the preferred embodiment of the present invention.

[0058] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may occur depending on design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A data aggregation acquisition method, used to obtain all called data in multiple database tables at once according to the calling relationship between multiple microservices that call each other, wherein the multiple microservices call each other to jointly complete a business, characterized in that: The method comprises: Configure multiple configuration items, and configure the calling operation of each microservice on each database table through the multiple configuration items, each of the configuration items includes a calling template and the corresponding database table information, and each of the configuration items is used to obtain data from one of the database tables; configuring the calling template, specifying in the calling template the data to be called by the calling operation corresponding to the corresponding configuration item; and Data aggregation acquisition, based on the calling templates of the multiple configuration items and the corresponding database table information and the multiple database tables, all the called data are obtained in an aggregated manner, Wherein, a SQL template is used as the calling template. Each of the configuration items also includes a unique number, and Configuring the call template further includes specifying the number in the custom part of the SQL template to specify the configuration item associated with the corresponding configuration item, thereby establishing the call relationship, The step of obtaining all the called data in an aggregated manner based on the calling template of the multiple configuration items, the corresponding database table information, and the multiple database tables includes: The plurality of configuration items are stored as a complete set of configurations in a business database of the business that is to call all of the microservices, and When processing the service, the entire set of configurations is loaded from the service database for processing.

2. The method according to claim 1, characterized in that in, The calling template is a SQL template, and Configuring the call template includes specifying, in the custom part of the SQL template, data to be called by the call operation corresponding to the corresponding configuration item.

3. The method according to claim 2, characterized in that in, Configuring the calling template also includes specifying field information in the custom part of the SQL template to specify the field of the specific data to be obtained.

4. The method according to claim 3, characterized in that: in, The custom part of the SQL template is replaced with the target data according to a predetermined parsing rule to generate a standard SQL, and the target data is loaded based on the standard SQL, so as to obtain the data to be called by the calling operation corresponding to the corresponding configuration item.

5. The method according to claim 1 or 2, characterized in that: in, The step of obtaining all the called data in an aggregated manner based on the calling template of the multiple configuration items, the corresponding database table information, and the multiple database tables includes: Each of the configuration items loads data sets from the multiple database tables according to its own database table information and stores the data sets in a data structure in the memory for calling by other configuration items.

6. A data aggregation acquisition system, used to acquire all called data in multiple database tables at once according to the calling relationship between multiple microservices that call each other, wherein the multiple microservices call each other to jointly complete a business, characterized in that: The system comprises: A configuration item configuration module, used to configure multiple configuration items, and to configure the calling operation of each microservice on each database table through the multiple configuration items, each of the configuration items includes a calling template and the corresponding database table information, and each of the configuration items is used to obtain data from one of the database tables; A calling template configuration module, which specifies in the calling template the data to be called by the calling operation corresponding to the corresponding configuration item; and A data aggregation acquisition module, based on the calling templates of the multiple configuration items and the corresponding database table information and the multiple database tables, aggregately obtains all the called data, Wherein, a SQL template is used as the calling template. Each of the configuration items also includes a unique number, and The calling template configuration module further specifies the number in the custom part of the SQL template to specify the configuration item associated with the corresponding configuration item, thereby establishing the calling relationship. The data aggregation acquisition module stores the multiple configuration items as a complete set of configurations in a business database of the business that is to call all of the microservices, and when processing the business, loads the complete set of configurations from the business database for processing.

7. An electronic device for acquiring data, comprising: one or more processors; as well as a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method as claimed in any one of claims 1 to 5.

8. A computer-readable medium having a program stored thereon, characterized in that: The program is executed by a processor to implement the method according to any one of claims 1 to 5.

Images