A penetrating data query method and system based on microservice call

Abstract

The present invention provides a penetrating data query method and system based on microservice call, including: step M1: a data middle-end system collects business source-end data of a source-end business system, and integrates the collected business source-end data to obtain summary data; step M2: a business model is established through the data middle-end system, and the summary data is queried through the service interface provided by the business model; step M3: a data model action is established through the data middle-end system, microservices are associated through the data model action, url information for registering the current microservice is obtained by mobilizing the microservice, and business source-end data is called through the url information of the current microservice; the penetrating data query of the present invention is implemented based on the v6 framework microservice method, and can not only query data through sql, but also realize data services through web interface, web interface and sql query combination, etc.

Description

Technical Field

[0001] The present invention relates to the field of big data, and specifically, to a penetrating data query method and system based on microservice calls, and more specifically, to a penetrating data query method and implementation based on microservice calls, which can be widely used in group enterprise data sharing and data governance. Background Art

[0002] China Baowu Ecological Technology Platform is based on the pilot requirements of state-owned capital investment companies. The Baowu Group headquarters is transformed into a capital operation headquarters that combines value creation and management functions. Focusing on the company's 2019 management theme of "Innovation to enhance system capabilities and collaboratively build a steel ecosystem", it provides technical support for the construction of the data operation collaboration platform and the penetrating supervision platform for the entire steel ecosystem. On October 8, 2018, the Corporate Governance Department, together with Baosight Software and other units, formally established the Baowu Ecological Technology Platform Planning Working Group for collaborative construction.

[0003] The essence of the data middle platform is to give full play to the storage and computing capabilities of the big data platform; centering on the enterprise development strategy, it realizes the business and management goals of serving the business, gaining insight into the business, driving the business, and creating value, organizes and integrates the data for modeling, and continuously accumulates standardized data service capabilities and data asset operation capabilities, empowers the front-end business, and provides flexible and efficient data services.

[0004] By integrating the accumulated data after modeling, data operation and maintenance can be performed efficiently. However, when data quality problems are found, the accumulated data cannot indicate which detailed data caused the problem during the integration process. At this time, it is necessary to drill from the business source and read the original detailed data until it is found which detailed data caused the data quality problem.

[0005] Patent document CN104050181A (application number: 201310079947.1) discloses an electronic form penetration query method and device, the method comprising: obtaining all related form unit information of the queried form unit, the queried form unit is calculated based on the contents of all the related form units; obtaining the source data of each related form unit respectively; combining the source data obtained by the query as the penetration query result of the queried form unit. The present invention also provides a corresponding device. Summary of the invention

[0006] In view of the defects in the prior art, the object of the present invention is to provide a penetrating data query method and system based on microservice call.

[0007] A penetrating data query method based on microservice call provided by the present invention includes:

[0008] Step M1: The data middle-end system collects the business source data of the source business system, and integrates the collected business source data to obtain summary data;

[0009] Step M2: Establish a business model through the data middle platform system, and query the summary data through the service interface provided by the business model;

[0010] Step M3: Establish a data model action through the data middle platform system, associate microservices through the data model action, obtain the URL information of registering the current microservice by mobilizing the microservice, and call the business source end data through the URL information of the current microservice;

[0011] The business model is to abstract the data set generated by the business process into a business model, and call the data interface through the abstract business model to query the data;

[0012] The data model action is a business model-associated micro-event, and the micro-event is associated with multiple micro-services.

[0013] Preferably, the step M1 comprises:

[0014] Step M1.1: In the data center system, configure a scheduled task to synchronize the table data of the specified business source data to the ODS table of the local database of the data center system;

[0015] Step M1.2: In the data middle-office system, configure a scheduled task to perform statistical calculations on the data in the ODS table of the local database of the data middle-office system, and store the statistical results in the entity table of the data middle-office system.

[0016] Preferably, the step M2 comprises:

[0017] Step M2.1: Establish a business model in the data middle platform system, configure the service interface of the business model, and provide external data services;

[0018] Step M2.2: Add the association between the business model and the entity table, and query the data in the entity table through the service interface of the business model.

[0019] Preferably, the service interface provided by the business model includes: for a business model associated with only one entity table, a single table query of the associated entity table data; for a business model associated with multiple entity tables, configuring SQL to query the entity table multiple tables; model action query, by calling a remote microservice to obtain data.

[0020] Preferably, step M3 includes: the data middle-end system obtains the business source data URL address where the current microservice is specifically implemented through the microservice information registered in the data middle-end system, requests the source-end business system to call the current microservice, and after the source-end business system obtains the microservice identifier, it queries the class file name and method name of the current microservice, and calls the specific implementation method.

[0021] A penetrating data query system based on microservice call provided by the present invention includes:

[0022] Module M1: The data middle platform system collects business source data from the source business system and integrates the collected business source data to obtain summary data;

[0023] Module M2: Establish a business model through the data middle platform system, and query the summary data through the service interface provided by the business model;

[0024] Module M3: Establish data model actions through the data middle platform system, associate microservices through data model actions, obtain the URL information of registering the current microservice by mobilizing microservices, and call the business source data through the URL information of the current microservice;

[0025] The business model is to abstract the data set generated by the business process into a business model, and call the data interface through the abstract business model to query the data;

[0026] The data model action is a business model-associated micro-event, and the micro-event is associated with multiple micro-services.

[0027] Preferably, the module M1 comprises:

[0028] Module M1.1: In the data center system, configure a scheduled task to synchronize the table data of the specified business source data to the ODS table of the local database of the data center system;

[0029] Module M1.2: In the data middle-office system, configure scheduled tasks to perform statistical calculations on the data in the ODS table of the local database of the data middle-office system, and store the statistical results in the entity table of the data middle-office system.

[0030] Preferably, the module M2 comprises:

[0031] Module M2.1: Establish a business model in the data middle platform system, configure the service interface of the business model, and provide external data services;

[0032] Module M2.2: Add the association between the business model and the entity table, and query the data of the entity table through the service interface of the business model.

[0033] Preferably, the service interface provided by the business model includes: for a business model associated with only one entity table, a single table query of the associated entity table data; for a business model associated with multiple entity tables, configuring SQL to query the entity table multiple tables; model action query, by calling a remote microservice to obtain data.

[0034] Preferably, the module M3 includes: the data middle-end system obtains the business source data URL address where the current microservice is specifically implemented through the microservice information registered in the data middle-end system, requests the source-end business system to call the current microservice, and after the source-end business system obtains the microservice identifier, it queries the class file name and method name of the current microservice and calls the specific implementation method.

[0035] Compared with the prior art, the present invention has the following beneficial effects:

[0036] 1. It simplifies the process of providing data services. Penetrating data query is the third data service interface of the data center model. The other two are model custom SQL single-table query and model API multi-table query. They all require the data from the business source to be extracted to the center. Penetrating data query can directly penetrate to the business source to directly query the original business data, omitting the inherent data extraction process.

[0037] 2. Data service implementation methods are richer, with penetrating data query, which is implemented based on the v6 framework microservice method. Data services can be implemented not only through SQL query, but also through web interface, web interface and SQL query combination, etc. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Other features, objects and advantages of the present invention will become more apparent from the detailed description of non-limiting embodiments made with reference to the following drawings:

[0039] Figure 1 This is a schematic diagram of a penetrating data query system based on microservice calls;

[0040] Figure 2 This is a flow chart of the penetrating data query method based on microservice calls;

[0041] Figure 3 This is a schematic diagram of a penetrating data query system based on microservice calls;

[0042] Figure 4 This is a schematic diagram of a penetrating data query system based on microservice calls. DETAILED DESCRIPTION

[0043] The present invention is described in detail below in conjunction with specific embodiments. The following embodiments will help those skilled in the art to further understand the present invention, but are not intended to limit the present invention in any form. It should be noted that, for those of ordinary skill in the art, several changes and improvements can also be made without departing from the concept of the present invention. These all belong to the protection scope of the present invention.

[0044] The integrated sedimentation data is stored in the data middle platform, while the original detailed data is stored in the business source end. In order to meet the needs of the data middle platform to both sediment data and drill down to the original detailed data, we provide a penetrating data query method based on microservice calls. The demander customizes the implementation method to read the business source end data, allowing the data middle platform to perform data governance and query from top to bottom.

[0045] Example 1

[0046] According to the present invention, a penetrating data query method based on microservice call is provided, comprising: Figure 1-Figure 4 As shown,

[0047] Step M1: The data middle-end system collects the business source data of the source business system, and integrates the collected business source data to obtain summary data;

[0048] Step M2: Establish a business model through the data middle platform system, and query the summary data through the service interface provided by the business model;

[0049] Step M3: Establish a data model action through the data middle platform system, associate microservices through the data model action, obtain the URL information of registering the current microservice by mobilizing the microservice, and call the business source end data through the URL information of the current microservice;

[0050] The business model is to abstract the data set generated by the business process into a business model, and call the data interface through the abstract business model to query the data;

[0051] The data model action is a business model-associated micro-event, and the micro-event is associated with multiple micro-services.

[0052] Specifically, the step M1 includes:

[0053] Step M1.1: In the data center system, configure a scheduled task to synchronize the table data of the specified business source data to the ODS table of the local database of the data center system;

[0054] Step M1.2: In the data middle-office system, configure a scheduled task to perform statistical calculations on the data in the ODS table of the local database of the data middle-office system, and store the statistical results in the entity table of the data middle-office system.

[0055] Specifically, the step M2 includes:

[0056] Step M2.1: Establish a business model in the data middle platform system, configure the service interface of the business model, and provide external data services;

[0057] Step M2.2: Add the association between the business model and the entity table, and query the data in the entity table through the service interface of the business model.

[0058] Specifically, the service interfaces provided by the business model include: for a business model associated with only one entity table, a single table query of the associated entity table data; for a business model associated with multiple entity tables, SQL is configured to query the entity table multiple tables; model action query, by calling remote microservices to obtain data.

[0059] Specifically, step M3 includes: the data middle-end system obtains the business source data URL address where the current microservice is specifically implemented through the microservice information registered in the data middle-end system, requests the source-end business system to call the current microservice, and after the source-end business system obtains the microservice identifier, it queries the class file name and method name of the current microservice, and calls the specific implementation method.

[0060] A penetrating data query system based on microservice call provided by the present invention includes:

[0061] Module M1: The data middle platform system collects business source data from the source business system and integrates the collected business source data to obtain summary data;

[0062] Module M2: Establish a business model through the data middle platform system, and query the summary data through the service interface provided by the business model;

[0063] Module M3: Establish data model actions through the data middle platform system, associate microservices through data model actions, obtain the URL information of registering the current microservice by mobilizing microservices, and call the business source data through the URL information of the current microservice;

[0064] The business model is to abstract the data set generated by the business process into a business model, and call the data interface through the abstract business model to query the data;

[0065] The data model action is a business model-associated micro-event, and the micro-event is associated with multiple micro-services.

[0066] Specifically, the module M1 includes:

[0067] Module M1.1: In the data center system, configure a scheduled task to synchronize the table data of the specified business source data to the ODS table of the local database of the data center system;

[0068] Module M1.2: In the data middle-office system, configure scheduled tasks to perform statistical calculations on the data in the ODS table of the local database of the data middle-office system, and store the statistical results in the entity table of the data middle-office system.

[0069] Specifically, the module M2 includes:

[0070] Module M2.1: Establish a business model in the data middle platform system, configure the service interface of the business model, and provide external data services;

[0071] Module M2.2: Add the association between the business model and the entity table, and query the data of the entity table through the service interface of the business model.

[0072] Specifically, the service interfaces provided by the business model include: for a business model associated with only one entity table, a single table query of the associated entity table data; for a business model associated with multiple entity tables, SQL is configured to query the entity table multiple tables; model action query, by calling remote microservices to obtain data.

[0073] Specifically, the module M3 includes: the data middle-end system obtains the business source data URL address where the current microservice is implemented through the microservice information registered in the data middle-end system, requests the source business system to call the current microservice, and after the source business system obtains the microservice identifier, it queries the class file name and method name of the current microservice, and calls the specific implementation method.

[0074] Example 2

[0075] Example 2 is a variation of Example 1

[0076] The legal person penetration project (CTJD) developed a project web application based on the iplat4j v6 framework to connect to the data center; the main business scenario of the CTJD project team is to produce Fanruan reports. The following takes the "Meeting and Meeting Minutes" report as an example to describe the details of the project team's implementation of penetration query.

[0077] Meetings and Minutes report:

[0078] Display summary data on the home page, i.e., meeting topics and meeting attachments;

[0079] Click the meeting name, expand the attachment hyperlink, and click the hyperlink to jump to the attachment preview page.

[0080] In this scenario, the project team first created a business model through the collection-fusion-modeling-customization services of the data center, providing a data service interface for project meeting records: meeting name, date, agenda, meeting attachment ID, etc., which is summary data.

[0081] Business source data refers to the original business data, including all data generated during the daily operation of the business, including detailed data of each business link;

[0082] 1. Collection

[0083] (1) Source data configuration

[0084] In the data middle-end system, configure the JDBC information of the business source database, including database type, IP, port, data domain, account, password, etc.

[0085] (2) Local database configuration

[0086] In the data middle-office system, configure a private database of the data middle-office system to extract and store database information of source data.

[0087] (3) Data entity design

[0088] In the data middle-end system, configure the ODS (Operational Data Store) table, which mainly stores data at or close to the source and detailed data.

[0089] Configure the entity table, which is used to store data in the ODS table that is summarized in various dimensions (such as time, location, unit, etc.).

[0090] The information for configuring ODS tables and entity tables includes: table name, local database information where the table is located, table field information, etc.

[0091] (4) Configure the collection task

[0092] In the data middle-end system, configure a scheduled task to synchronize the table data of the specified source database to the local database ODS table.

[0093] 2. Fusion

[0094] (1) Configure entity computing tasks

[0095] In the local database, configure a scheduled task to perform statistics (SQL calculation) on the data in the ODS table and store the statistical results in the entity table.

[0096] 3. Modeling

[0097] (1) Configure basic model information

[0098] Establish a business model. A business model is just a concept that represents a data set with a certain business meaning. The business model can configure service interfaces and provide external data services.

[0099] (2) Business model related entity table

[0100] Add a business model-entity table association relationship, that is, you can query the data of the entity table through the service interface of the business model.

[0101] 4. Customized services

[0102] (1) Registration model service interface

[0103] The business model can be configured with three types of service interfaces: 1. For a business model associated with only one entity table, a single table query is used to query the associated entity table data; 2. For a business model associated with multiple entity tables, SQL is configured to query the entity table multiple tables; 3. Business model action query obtains data by calling remote microservices. Remote microservices can be understood as Java interfaces of remote systems.

[0104] The above are the premises in daily scenarios. Through customized services, you can obtain the summarized data and form a report with the summarized data, which can be used to supervise and monitor business data. In the process of monitoring and operation, when abnormalities are found in business data, the underlying raw data that forms the summarized data can be analyzed and judged through the penetrating data query function to locate the specific cause or location of the abnormality.

[0105] The data center provides an SDK jar package that supports querying source data through data model actions. Fanruan reports can obtain data sets by calling data model actions for report production and presentation. The CTJD project team uses the SDK provided by the Fanruan report production software and the data center. In the data center, you can view summary data and query detailed data in a penetrating manner.

[0106] 5. Model Action

[0107] (1) Calling model actions

[0108] Through the customized data model actions in the customized service, you can call the micro-events associated with the action;

[0109] 6. Events

[0110] (1) Data model action associated (micro) events

[0111] Micro-event is a concept in the data middle platform system, which requires user configuration. (Micro) event is a collection of ordered microservices. By calling micro-events, micro-events can be called in the defined order.

[0112] 7. Microservices

[0113] (1) Micro-events associated with micro-services

[0114] Microservice is a concept in the data middle-end system, which provides basic support for calling data services of remote systems.

[0115] By calling the microservice and obtaining the URL information when registering the microservice, you can call the corresponding data service interface in the remote system, that is, the Java interface.

[0116] 8. Source-side microservices

[0117] (1) Source-side microservice implementation and registration

[0118] The source end refers to the source business system. The source business system needs to implement the microservice code and register the microservice information in the source business system: service class file name, service method name.

[0119] (2) The data center system calls the source microservice

[0120] The data middle-end system obtains the URL address of the source business system where the microservice is specifically implemented through the microservice information registered in the data middle-end system, and requests the source business system to call the microservice. After the source business system obtains the microservice identifier, it queries the class file name and method name of the microservice and calls the specific implementation method.

[0121] (3) Query the source details table

[0122] The implementation of the source business system microservice is to query the business details table of the source business system, that is, the source details table.

[0123] At this point, through the data middle platform, the source detailed data is summarized from the source business system to the entity table to form summary data, and then through the penetrating data query method, it penetrates back from the data middle platform system to the source business system to query the source detailed data.

[0124] Example 3

[0125] Embodiment 3 is a variation of Embodiment 1 and / or Embodiment 2.

[0126] Eplat middle platform is the data middle platform system, and source application is the source application system. It is developed using the v6 framework and uses penetrating queries.

[0127] The V6 framework provides a microservice module, and the data middle platform system additionally implements a model service module

[0128] Figure 2 The penetration query process is shown:

[0129] Step 1: In the data middle-end system, penetrating query is to call the data model action.

[0130] Step 2: Model actions are associated with micro-events, micro-events are associated with micro-services, and calling data model actions is to call micro-services.

[0131] Step 3: In the business scenario of penetrating query, in the data middle-end system, the microservice type associated with the data model action is URL, that is, the microservice of the remote source application system can be called through the URL registered by the microservice. This is the meaning of penetration.

[0132] Step 4: The penetrating query penetrates to the source application system to call the microservice, which is a platform-hosted type.

[0133] The source application system receives the call request and parses the English names of the secondary module and primary module corresponding to the microservice according to the requested microservice unique identifier and the naming rule of the microservice unique identifier.

[0134] The secondary module refers to business links, such as warehousing, transportation, etc., and the primary module refers to business domains, such as logistics.

[0135] Step 5: Get the application name according to the first-level module. This mainly deals with the distributed source application system, where one application is deployed in one business domain.

[0136] Step 6: Get the IP and port information of the application deployment node according to the application name;

[0137] Step 7: Assemble and concatenate the URL address of the microservice call based on the node IP and port information, and forward the microservice call information to the node.

[0138] In a distributed system, if an application deployed on a node finds that the first- and second-level modules of a microservice do not belong to itself, it determines that there is no implementation code for the microservice locally, and then finds the node that implements the microservice and forwards the request to it.

[0139] Example 4

[0140] Embodiment 4 is a variation of Embodiment 1, Embodiment 2 and / or Embodiment 3.

[0141] 1. Microservice implementation at the business source end

[0142] Penetrating query of the original detailed data of the conference attachment: attachment ID, name, download address, etc. This is detailed data (1), querying data from the data environment of the business source application.

[0143] Call the document preview interface of the data center, and obtain the attachment preview URL address through the attachment download address. This is the detailed data (2).

[0144] 2. Data center configuration microservice information

[0145] Configure the microservice in the data center, set the microservice ID to [S_JD_ZY_010], type to "url", and manually fill in the microservice call address: http: / / {ip}:{port} / ctjd / service.

[0146] 3. Configure the corresponding event information

[0147] Configure micro-event information in the data center, set the event ID to [E_JD_ZY_010], and the associated service to [S_JD_ZY_010].

[0148] 4. Data Modeling

[0149] Create a new data model action [M_CW_SPES_ATTACHMENT_FILE] in the data center to provide an attachment preview interface.

[0150] 5. Configure the penetration query action

[0151] Configure the model action [A_JD_ZY_010] of [M_CW_SPES_ATTACHMENT_FILE] and associate it with the micro event [E_JD_ZY_010]

[0152] In the Fanruan report, by using the model action SDK, you can call the model action [A_JD_ZY_010] through the model ID, thereby calling the microservice [S_JD_ZY_010] of the project source application to obtain the attachment preview address.

[0153] Those skilled in the art know that, in addition to implementing the system, device and its various modules provided by the present invention in a purely computer-readable program code, it is entirely possible to implement the same program in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers and embedded microcontrollers by logically programming the method steps. Therefore, the system, device and its various modules provided by the present invention can be considered as a hardware component, and the modules included therein for implementing various programs can also be considered as structures within the hardware component; the modules for implementing various functions can also be considered as both software programs for implementing the method and structures within the hardware component.

[0154] The above describes the specific embodiments of the present invention. It should be understood that the present invention is not limited to the above specific embodiments, and those skilled in the art can make various changes or modifications within the scope of the claims, which does not affect the essence of the present invention. In the absence of conflict, the embodiments of the present application and the features in the embodiments can be combined with each other arbitrarily.

Claims

1. A penetrating data query method based on microservice call, characterized in that: include: Step M1: The data middle-end system collects the business source data of the source business system, and integrates the collected business source data to obtain summary data; Step M2: Establish a business model through the data middle platform system, and query the summary data through the service interface provided by the business model; Step M3: Establish a data model action through the data middle platform system, associate microservices through the data model action, obtain the URL information of registering the current microservice by mobilizing the microservice, and call the business source end data through the URL information of the current microservice; The business model is to abstract the data set generated by the business process into a business model, and call the data interface through the abstract business model to query the data; The data model action is a business model-related micro-event, and the micro-event is associated with multiple micro-services; The step M2 comprises: Step M2.1: Establish a business model in the data middle platform system, configure the service interface of the business model, and provide external data services; Step M2.2: Add the association between the business model and the entity table, and query the data of the entity table through the service interface of the business model; The service interfaces provided by the business model include: for a business model associated with only one entity table, query the associated entity table data by a single table; for a business model associated with multiple entity tables, configure SQL to query the entity table multiple tables; model action query, obtain data by calling remote microservices; The step M3 includes: the data middle-end system obtains the business source data URL address where the current microservice is specifically implemented through the microservice information registered in the data middle-end system, requests the source-end business system to call the current microservice, and after the source-end business system obtains the microservice identifier, it queries the class file name and method name of the current microservice, and calls the specific implementation method.

2. The microservice call-based penetrating data query method according to claim 1 is characterized in that: The step M1 comprises: Step M1.1: In the data center system, configure a scheduled task to synchronize the table data of the specified business source data to the ODS table of the local database of the data center system; Step M1.2: In the data middle-office system, configure a scheduled task to perform statistical calculations on the data in the ODS table of the local database of the data middle-office system, and store the statistical results in the entity table of the data middle-office system.

3. A penetrating data query system based on microservice calls, characterized in that: include: Module M1: The data middle platform system collects business source data from the source business system and integrates the collected business source data to obtain summary data; Module M2: Establish a business model through the data middle platform system, and query the summary data through the service interface provided by the business model; Module M3: Establish data model actions through the data middle platform system, associate microservices through data model actions, obtain the URL information of registering the current microservice by mobilizing microservices, and call the business source data through the URL information of the current microservice; The business model is to abstract the data set generated by the business process into a business model, and call the data interface through the abstract business model to query the data; The data model action is a business model-related micro-event, and the micro-event is associated with multiple micro-services; The module M2 comprises: Module M2.1: Establish a business model in the data middle platform system, configure the service interface of the business model, and provide external data services; Module M2.2: Add the relationship between the business model and the entity table, and query the data of the entity table through the service interface of the business model; The service interfaces provided by the business model include: for a business model associated with only one entity table, query the associated entity table data by a single table; for a business model associated with multiple entity tables, configure SQL to query the entity table multiple tables; model action query, obtain data by calling remote microservices; The module M3 includes: the data middle-end system obtains the business source data URL address where the current microservice is implemented through the microservice information registered in the data middle-end system, requests the source business system to call the current microservice, and after the source business system obtains the microservice identifier, it queries the class file name and method name of the current microservice and calls the specific implementation method.

4. The microservice call-based penetrating data query system according to claim 3 is characterized in that: The module M1 comprises: Module M1.1: In the data center system, configure a scheduled task to synchronize the table data of the specified business source data to the ODS table of the local database of the data center system; Module M1.2: In the data middle-office system, configure scheduled tasks to perform statistical calculations on the data in the ODS table of the local database of the data middle-office system, and store the statistical results in the entity table of the data middle-office system.

Images