Self-service Report Data Processing Method and Self-service BI System Based on Middleware Mode

Abstract

The present invention relates to a self-service report data processing method and a self-service BI system based on a middleware mode. The method includes the following steps: obtaining a query instruction; performing encapsulation and table building processing on the query instruction to generate a timed scheduling task and a data logic processing execution instruction. When the timed scheduling task automatically responds, call the big data scheduling platform interface to obtain a data set, and perform logic processing on the data by using the data logic processing execution instruction; export the logically processed data set to a first database; generate an operable interface, and the operable interface calls the data in the first database based on a user operation instruction to generate a report. Compared with the prior art, the present invention has the advantages of improving data call and report production efficiency, etc.

Description

Technical Field

[0001] The present invention relates to a BI system, and more particularly to a self-service report data processing method and a self-service BI system based on a middle platform mode. Background Art

[0002] When existing BI (Business Intelligence) systems are connected to big data platforms, they generally adopt the method of opening ODBC and JDBC interfaces or importing data into an internal database through manual operations. The method of directly providing interfaces ignores the incoordination between the slowness of batch processing tasks on the big data platform and the fast response requirements of the BI system, while the method of importing into an internal database abandons the advantages of distributed computing on the big data platform and limits the data processing capabilities of the data platform.

[0003] Therefore, it is necessary to improve the existing BI system. Summary of the Invention

[0004] The purpose of the present invention is to provide a self-service report data processing method based on a middle platform mode that improves data call and report production efficiency to overcome the defects of the existing technologies described above.

[0005] The purpose of the present invention can be achieved by the following technical solutions:

[0006] A self-service report data processing method based on a middle platform mode includes the following steps:

[0007] Obtain a query instruction;

[0008] Perform encapsulation and table creation processing on the query instruction to generate a timed scheduling task and a data logic processing execution instruction. When the timed scheduling task automatically responds, call the big data scheduling platform interface to obtain a data set, and perform logic processing on the data using the data logic processing execution instruction;

[0009] Export the logically processed data set to a first database;

[0010] Generate an operable interface, and the operable interface calls the data in the first database based on a user operation instruction to generate a report.

[0011] Further, the query instruction is an SQL statement.

[0012] Further, the encapsulation and table creation processing is specifically as follows:

[0013] Encapsulate the query instruction into a table creation statement, and create a corresponding temporary table based on the table creation statement;

[0014] Create a partitioned table in the first database based on the data and data characteristics in the temporary table.

[0015] Further, the data features include field type and length.

[0016] Further, the operable interface is generated based on the data visualization tool Tableau.

[0017] Further, an operation box and draggable data are generated within the operable interface.

[0018] Further, the operation box includes one or more of new creation, chart generation, data operation, saving, saving as, modification, deletion, sharing, downloading, subscription, data update, and data rerun.

[0019] The present invention also provides a self-service BI system based on the middle platform mode, including:

[0020] A query module for receiving query instructions;

[0021] A query processing module for performing encapsulation and table creation processing on the query instructions, generating a timed scheduling task and a data logic processing execution instruction, calling the big data scheduling platform interface when the timed scheduling task automatically responds, obtaining a data set, and performing logic processing on the data using the data logic processing execution instruction;

[0022] A data export module for exporting the logically processed data set to a first database;

[0023] A visualization module connected to the first database, generating an operable page, and calling the data in the first database based on user operation instructions to generate a report.

[0024] Further, the query module is established based on the web-edit tool.

[0025] Further, the operable interface is generated based on the data visualization tool Tableau.

[0026] Compared with the prior art, the present invention has the following beneficial effects:

[0027] 1. By integrating and calling an interface, the present invention generates a timed scheduling task, stores the exported data in a first database, has an operable page, supports the business to combine its own data skills, uses methods such as dragging, functions, and SQL language to go deeper step by step, adopts a working mode of cooperation within the department or independent completion, uses ClickHouse to calculate massive data, and has a fast calculation ability with a millisecond response for summarization, simplifying the originally complex application process and the report-making work of cross-departmental communication into a short process of team cooperation within the department or employees independently making reports within one day.

[0028] 2. The BI system of the present invention integrates various data components or platforms. After establishing a complete data exploration using the computing power of the big data platform, the results can be quickly displayed on the front-end web page. In the data source creation link, big data batch tasks are automatically generated to make full use of the data processing capabilities of the big data platform; in the report display link, the BI tool connects to the batch result data set imported into ClickHouse and responds to the requirements of users' dragged data within milliseconds.

[0029] 3. The present invention breaks through the barriers between technology and business. Through the middle platform mode, the original mode of separate division of labor and cooperation between technology and business is transformed into a mode of technology providing skills training + business self-service operation, fully releasing the subjective ability of business and mining the potential value of data, and having good application prospects for the application. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 It is a schematic flowchart of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0031] The present invention will be described in detail below with reference to the drawings and specific embodiments. This embodiment is implemented on the premise of the technical solution of the present invention, and the detailed implementation manner and specific operation process are given, but the protection scope of the present invention is not limited to the following embodiments.

[0032] Embodiment 1

[0033] The middle platform is an enterprise architecture method designed to effectively improve the reuse ability. The middle platform is a set of methods that combines Internet technology and industry characteristics to precipitate the core capabilities of the enterprise in a shared service center. The purpose of the middle platform is to provide the enterprise with the ability to innovate quickly and at low cost, and can strongly support the digital operation capabilities and product technology capabilities of the entire enterprise for each business front desk.

[0034] As Figure 1 shown, this embodiment provides a self-service report data processing method based on the middle platform mode, including:

[0035] Step S1, obtaining a query instruction. In this embodiment, the query instruction is an SQL statement, and a front-end query tool of the big data platform, such as tools like web-edit, can be used to obtain it.

[0036] Step S2, performing encapsulation and table creation processing on the query instruction to generate a timed scheduling task and a data logic processing execution instruction. When the timed scheduling task responds, the big data scheduling platform interface is called to obtain a data set, and the data is logically processed using the data logic processing execution instruction.

[0037] After receiving the query instruction, the background application encapsulates the statement into a table creation statement, creates a temporary table, checks the data in the temporary table, determines the field type and length in combination with the data lineage, creates a partitioned table in the first database, calls the interface of the big data scheduling platform to generate a task execution box, generates a task dependency box based on the dependency table timeliness filled in by the user, calls the interface to set the batch date, and executes the task to perform logical processing on the data. In this step, a big data batch task is automatically generated through the generated timed scheduling task, making full use of the data processing capabilities of the big data platform. The timed scheduling task sets the timing period based on the task dependency box.

[0038] In this embodiment, the first database uses ClickHouse.

[0039] Step S3: Export the logically processed data set to the first database.

[0040] In this embodiment, the open-source component SQOOP of the big data platform can be used to export the logically processed data set to the OLAP database Click House.

[0041] Step S4: Generate an operable interface, which calls the data in the first database based on the user operation instruction to generate a report.

[0042] In this embodiment, the operable interface is generated based on the data visualization tool Tableau. Tableau automatically connects to the database Click House through the background application, calls the API interface of Tableau, generates a drag-and-drop page for the user to operate Tableau, explores the data independently, and creates a report.

[0043] An operation box and draggable data are generated in the operable interface. The operation box includes one or more of new, generate chart, data operation, save, save as, modify, delete, share, download, subscribe, data update, and data rerun.

[0044] The above method realizes an overall solution for processing massive data, generating timed scheduling tasks, importing into an OLAP high-performance database, and using a BI software to drag and create reports, improving the data call and report creation efficiency.

[0045] When the above method is implemented in the form of software functional units and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes: various media such as USB flash drives, mobile hard disks, read-only memories (ROM, Read-Only Memory), random access memories (RAM, Random Access Memory), magnetic disks, or optical discs that can store program codes.

[0046] Embodiment 2

[0047] This embodiment provides a self-service BI system based on the middle platform mode, including: a query module for receiving query instructions; a query processing module for encapsulating and table-building processing of the query instructions, generating a timed scheduling task and a data logic processing execution instruction, calling the big data scheduling platform interface when the timed scheduling task responds, obtaining a data set, and performing logical processing on the data using the data logic processing execution instruction; a data export module for exporting the logically processed data set to a first database; a visualization module connected to the first database to generate an operable page, and this operable interface calls the data in the first database based on user operation instructions to generate a report.

[0048] The above query module is built based on front-end query tools of the big data platform, such as tools like web-edit; the data export module is built based on the open-source component SQOOP of the big data platform; the operable interface is generated based on the data visualization tool Tableau. The rest is the same as in Embodiment 1.

[0049] The above has described in detail the preferred specific embodiments of the present invention. It should be understood that those of ordinary skill in the art can make many modifications and variations based on the concept of the present invention without creative work. Therefore, all technical solutions that can be obtained by those skilled in the art in the technical field of the present invention through logical analysis, reasoning, or limited experiments based on the concept of the present invention on the basis of the prior art should be within the protection scope determined by the claims.

Claims

1. A self-service report data processing method based on the middle platform mode, characterized in that The following steps are involved: Get query instructions; The query instruction is encapsulated and table-created to generate a scheduled scheduling task and a data logic processing execution instruction. When the scheduled scheduling task automatically responds, the big data scheduling platform interface is called to obtain a data set, and the data logic processing execution instruction is used to perform logical processing on the data; specifically: after obtaining the query instruction, the background application encapsulates the statement as a table-creating statement, creates a temporary table, checks the data in the temporary table, and determines the field type and length in combination with the data lineage relationship, creates a partition table in the first database, calls the big data scheduling platform interface, generates a task execution box, generates a task dependency box based on the timeliness of the dependency table filled in by the user, calls the interface to set the batch date, and executes the task to perform logical processing on the data; automatically generates a big data batch task through the generated scheduled scheduling task, wherein the scheduled scheduling task sets the timing period based on the task dependency box; Exporting the logically processed data set to the first database; An operable interface is generated, which calls the data in the first database based on the user operation instruction to generate a report.

2. The self-service report data processing method based on the middle platform mode according to claim 1, wherein The query instruction is a SQL statement.

3. The self-service report data processing method based on the middleware mode according to claim 1, characterized in that The encapsulation table creation process is specifically as follows: Encapsulate the query instruction into a table creation statement, and create a corresponding temporary table based on the table creation statement; Based on the data and data features in the temporary table, a partition table is created in the first database.

4. The self-service report data processing method based on the middle platform mode according to claim 3, wherein, The data characteristics include field type and length.

5. The self-service report data processing method based on the middleware platform mode according to claim 1, wherein The operational interface is generated based on the data visualization tool Tableau.

6. The self-service report data processing method based on the middle platform mode according to claim 1, characterized in that An operation box and draggable data are generated in the operable interface.

7. The self-service report data processing method based on the middle platform mode according to claim 6, wherein The operation box includes one or more of creating, generating charts, data calculation, saving, saving as, modifying, deleting, sharing, downloading, subscribing, updating data, and re-running data.

8. A self-service BI system based on the middle platform mode, characterized in that, include: A query module, used for receiving query instructions; The query processing module is used to encapsulate and create tables for the query instructions, generate timed scheduling tasks and data logic processing execution instructions, call the big data scheduling platform interface when the timed scheduling task automatically responds, obtain the data set, and use the data logic processing execution instructions to perform logical processing on the data; specifically: after obtaining the query instruction, the background application encapsulates the statement into a table creation statement, creates a temporary table, checks the data in the temporary table, and determines the field type and length based on the data blood relationship, creates a partition table in the first database, calls the big data scheduling platform interface, generates a task execution box, generates a task dependency box based on the timeliness of the dependency table filled in by the user, calls the interface to set the batch date, and executes the task to perform logical processing on the data; automatically generates big data batch tasks through the generated timed scheduling tasks, wherein the timed scheduling tasks set the timing period based on the task dependency box; A data export module, used to export the logically processed data set to the first database; The visualization module is connected to the first database to generate an operable interface. The operable interface calls the data in the first database based on user operation instructions to generate a report.

9. The self-service BI system based on the middle platform mode according to claim 8, characterized in that, The query module is established based on the web-edit tool.

10. The self-service BI system based on the middle platform mode according to claim 8, characterized in that, The operable interface is generated based on the data visualization tool Tableau.

Images