Report generation methods, devices, terminal equipment, and storage media
By building a report generation system, the problems of insufficient data management and task monitoring in report generation were solved, achieving efficient data report generation and task management, and improving data query performance and security.
Patent Information
- Application Number
- CN202311861783.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-12-29
- Publication Date
- 2026-11-17
- Estimated Expiration
- 2043-12-29
AI Technical Summary
Existing technologies lack data management and task monitoring during report generation, resulting in low efficiency, inability to effectively manage the data task lifecycle and scheduling, poor data task query performance, lack of data security hierarchical management, and complex task process links.
By constructing a report generation system, including a data management platform, a template management platform, and a task management platform, data is extracted, cleaned, type-defined, and security-classified to generate data reports. The system also uses a task execution queue and computing cluster to sort and execute tasks, thereby enabling the generation and monitoring of data reports.
It improves the efficiency of report generation, solves the problems of insufficient data management and task monitoring, ensures data security and task process visualization, and enhances data task query performance.
Smart Images

Figure CN117807971B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of computer technology, and in particular to a report generation method, apparatus, terminal device, and storage medium. Background Technology
[0002] As the business expands and develops, the scale of data and the demand for data usage are increasing. In various scenarios such as customer complaint investigation, temporary business data statistics, submission of regulatory data reports, weekly report data statistics, and value review data, it is necessary to perform statistical queries on data and generate reports. At the same time, business data involves user privacy information, and strict access control is required for data use. The traditional data report generation solution is as follows: apply for access permissions for each data table, write SQL or SQL-like statements, perform manual queries on the built-in command-line interface (CLI) tool, and export the results to generate the corresponding reports.
[0003] However, traditional data report generation solutions have the following drawbacks: (1) lack of data task lifecycle management and scheduling, mainly including the need for users to manually follow up on task execution, which affects work efficiency and does not support task scheduling; (2) poor data task query performance; (3) lack of data security hierarchical management; (4) complex task process links, mainly including data dispersion and lack of data result visualization.
[0004] Therefore, a report generation method is needed to solve the problem of low efficiency caused by the lack of corresponding data management and task monitoring during report generation.
[0005] The above content is only used to help understand the technical solution of the present invention and does not represent an admission that the above content is prior art. Summary of the Invention
[0006] The main objective of this invention is to provide a report generation method, apparatus, terminal device, and storage medium, aiming to solve the technical problem of low efficiency caused by the lack of corresponding data management and task monitoring during report generation.
[0007] To achieve the above objectives, the present invention provides a report generation method, which includes the following steps:
[0008] Receive report generation request;
[0009] The report generation request is input into a pre-built report generation system, which generates a data report based on the report generation request. The report generation system is based on a preset data management platform, template management platform, and task management platform.
[0010] Optionally, the step of inputting the report generation request into a pre-built report generation system may include:
[0011] Data is extracted from a pre-defined database to obtain initial metadata, and a platform is built using the initial metadata to obtain a data management platform.
[0012] Configure templates according to the data management platform, obtain report task templates, and build a template platform using the report task templates to obtain a template management platform.
[0013] The report generation system is constructed by using the data management platform, template management platform, and preset task management platform.
[0014] Optionally, the step of building the platform using the initial metadata and obtaining the data management platform includes:
[0015] The initial metadata is cleaned and its type is defined to obtain the data type of the initial metadata;
[0016] The initial metadata is classified into security levels according to preset data characteristics to obtain the data security level of the initial metadata.
[0017] The platform is constructed based on the data type and data security level to obtain a data management platform.
[0018] Optionally, the step of generating a data report by the report generation system based on the report generation request includes:
[0019] The report generation request is parsed to obtain a data query strategy, and the data is queried through the data management platform according to the data query strategy to obtain report data;
[0020] Based on the report data, a task is generated through the template management platform to obtain the report generation task;
[0021] The report generation task is sent to the task management platform for task execution to obtain the data report.
[0022] Optionally, the step of sending the report generation task to the task management platform for task execution and obtaining the data report includes:
[0023] The report generation task is sent to the execution queue through the task management platform for sorting, and the sorting result is obtained.
[0024] If the sorting result is successful, the report generation task is sent to the computing cluster for task execution, and the execution result is obtained.
[0025] If the execution result is successful, then obtain the initial data report;
[0026] The initial data report is anonymized and categorized according to the data security level of the data management platform. The data report is then obtained, and notification information is generated and sent to the corresponding users through the message center.
[0027] Optionally, after the step of sending the report generation task to the execution queue for queue sorting through the task management platform and obtaining the sorting result, the method further includes:
[0028] If the sorting result is a sorting failure, then obtain the queue status of the computing cluster;
[0029] An exception message is generated based on the queue status, and then sent to the corresponding user through the message center.
[0030] Optionally, after the step of generating a data report through the report generation system based on the report generation request, the method further includes:
[0031] Perform scenario analysis on the report generation request to obtain the report scenario;
[0032] Based on the reported scenario, a missing data query is performed on the data report through the data management platform to obtain the query results;
[0033] If the query result indicates missing data, then the data report will be compensated to obtain the final data report.
[0034] This invention also proposes a report generation device, the report generation device comprising:
[0035] The receiving module is used to receive report generation requests;
[0036] The generation module is used to input the report generation request into a pre-built report generation system, and the report generation system generates a data report based on the report generation request. The report generation system is based on a preset data management platform, template management platform, and task management platform.
[0037] This invention also proposes a terminal device, which includes a memory, a processor, and a report generation program stored in the memory and executable on the processor. When the report generation program is executed by the processor, it implements the steps of the report generation method described above.
[0038] This invention also proposes a computer-readable storage medium storing a report generation program, which, when executed by a processor, implements the steps of the report generation method described above.
[0039] This invention proposes a report generation method, apparatus, terminal device, and storage medium. The method involves receiving a report generation request, inputting the request into a pre-built report generation system, and then having the system generate a data report based on the request. The report generation system is based on a pre-defined data management platform, template management platform, and task management platform. By using the pre-built data management platform, template management platform, and task management platform within the report generation system to generate the data report, the invention solves the problem of low efficiency caused by the lack of corresponding data management and task monitoring during report generation, thus improving the efficiency of report generation. Attached Figure Description
[0040] Figure 1 This is a schematic diagram of the functional modules of the terminal device to which the report generation device of this invention belongs;
[0041] Figure 2 This is a flowchart illustrating an exemplary embodiment of the report generation method of the present invention;
[0042] Figure 3 This is a schematic diagram of the overall framework of the report generation method of the present invention;
[0043] Figure 4 This is a flowchart illustrating another exemplary embodiment of the report generation method of the present invention;
[0044] Figure 5 This is a schematic diagram illustrating the report generation task management involved in the report generation method of the present invention;
[0045] Figure 6 This is a schematic diagram illustrating the method for generating reports according to the present invention, which involves sending abnormal information to the user.
[0046] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation
[0047] It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.
[0048] The main solution of this invention is as follows: Data is extracted from a preset database to obtain initial metadata; a platform is built using the initial metadata to obtain a data management platform; templates are configured based on the data management platform to obtain report task templates; a template platform is built using the report task templates to obtain a template management platform; a system is built using the data management platform, the template management platform, and a preset task management platform to obtain a report generation system. The initial metadata is cleaned and its type defined to obtain its data type; the initial metadata is classified for security based on preset data characteristics to obtain its data security level; a platform is built using the data type and data security level to obtain the data management platform. The report generation request is parsed to obtain a data query strategy; data is queried using the data query strategy through the data management platform to obtain report data; a task is generated using the template management platform based on the report data to obtain a report generation task; the report generation task is sent to the task management platform for execution to obtain a data report. The task management platform sends the report generation task to an execution queue for sorting and obtains the sorting result. If the sorting result is successful, the report generation task is sent to the computing cluster for execution and obtains the execution result. If the execution result is successful, an initial data report is obtained. The initial data report is then anonymized according to the data security level of the data management platform, and a notification is generated and sent to the corresponding user via the message center. If the sorting result is a sorting failure, the queue status of the computing cluster is obtained. An exception message is generated based on the queue status and sent to the corresponding user via the message center. The report generation request is analyzed to obtain the report scenario. Based on the report scenario, a missing data query is performed on the data report through the data management platform, and the query result is obtained. If the query result indicates missing data, task compensation is performed on the data report to obtain the final data report. This solves the problem of low efficiency caused by the lack of corresponding data management and task monitoring during report generation, thus improving the efficiency of report generation.Based on the solution of this invention, given the increasing scale and demand for data in reality, various scenarios such as customer complaint investigation, temporary business data statistics, submission of regulatory data reports, weekly report statistics, and value review data require statistical queries and report generation. Furthermore, business data involves user privacy information, necessitating strict access control. Traditional generation methods cannot solve the problems of data task lifecycle management and scheduling, as well as poor data task query performance, resulting in low efficiency. Therefore, a report generation method is designed, and its effectiveness is verified during report generation. Finally, the efficiency of report generation using this method is significantly improved.
[0049] Technical terms involved in the embodiments of this invention:
[0050] HDFS: HDFS (Hadoop Distributed File System) is a distributed file system in the Apache Hadoop project designed to handle the storage and processing of large-scale datasets. It is a reliable, high-throughput distributed file system suitable for storing and processing large-scale datasets. Its design goal is to provide high reliability, high performance, and high fault tolerance, enabling users to store, access, and analyze large-scale data.
[0051] Hive: Hive is a data warehouse tool based on Hadoop, used for data extraction, transformation, and loading. It's a mechanism for storing, querying, and analyzing large-scale data stored in Hadoop. Hive data warehouse tools can map structured data files to database tables and provide SQL query functionality, transforming SQL statements into MapReduce tasks for execution. Hive's advantages include a low learning curve; it allows for fast MapReduce statistics through SQL-like statements, simplifying MapReduce without requiring the development of dedicated MapReduce applications. Hive is well-suited for statistical analysis of data warehouses.
[0052] MR (MapReduce) queries are a query method based on the MapReduce framework. They typically consist of two phases: mapping (Map) and reduction (Reduce). MR queries can be implemented using the MapReduce programming model. Developers need to write custom Mapper and Reducer code, implementing the map and reduce functions to define the specific query logic. Input data is usually stored in HDFS. The Mapper reads the data from HDFS, performs mapping processing, and then the Reducer performs a reduction operation on the mapping results to finally obtain the query result.
[0053] SQL: SQL (Structured Query Language) is a standardized language for manipulating relational databases. It can be used to query, insert, update, and delete data, as well as create and manage database objects such as tables, views, and indexes.
[0054] ClickHouse: ClickHouse is an open-source columnar database management system specifically designed for efficient large-scale data analysis. It is suitable for scenarios involving large-scale data analysis and querying, and it excels in performance, scalability, and ease of use, making it one of the preferred tools for many enterprises and organizations to conduct data analysis.
[0055] The embodiments of the present invention take into account that, when generating reports, related technologies cannot solve the problems of data task lifecycle management and scheduling and poor data task query performance, so this method has the problem of low accuracy.
[0056] Therefore, this invention addresses the growing demand for data in reality, such as customer complaint investigation, temporary business data statistics, regulatory data report submission, weekly report statistics, and value review data. Various scenarios require statistical queries and report generation, and business data often involves user privacy information, necessitating strict access control. Traditional generation methods fail to address the issues of data task lifecycle management and scheduling, as well as poor data task query performance, resulting in low efficiency. This invention proposes a report generation method, and its effectiveness is verified during report generation. Finally, the efficiency of report generation using this method is significantly improved.
[0057] Specifically, refer to Figure 1 , Figure 1 This is a schematic diagram of the functional modules of the terminal device to which the report generation device of the present invention belongs. This report generation device can be an independent device capable of generating reports, separate from the terminal device, and can be implemented on the terminal device in hardware or software form. The terminal device can be a smart mobile device with data processing capabilities, such as a mobile phone or tablet computer, or a fixed terminal device or server with data processing capabilities.
[0058] In this embodiment, the terminal device to which the report generation device belongs includes at least an output module 110, a processor 120, a memory 130, and a communication module 140.
[0059] The memory 130 stores the operating system and a report generation program. The report generation device can receive report generation requests; the report generation request is input into a pre-built report generation system, which generates a data report based on the request. The report generation system is based on a preset data management platform, template management platform, and task management platform. The report generation program generates reports, and the generated results and other information are stored in the memory 130. The output module 110 can be a display screen, etc. The communication module 140 can include a WIFI module, a mobile communication module, and a Bluetooth module, etc., and communicates with external devices or servers through the communication module 140.
[0060] When the report generation program in memory 130 is executed by the processor, it performs the following steps:
[0061] Receive report generation request;
[0062] The report generation request is input into a pre-built report generation system, which generates a data report based on the report generation request. The report generation system is based on a preset data management platform, template management platform, and task management platform.
[0063] Furthermore, when the report generation program in memory 130 is executed by the processor, it also performs the following steps:
[0064] Data is extracted from a pre-defined database to obtain initial metadata, and a platform is built using the initial metadata to obtain a data management platform.
[0065] Configure templates according to the data management platform, obtain report task templates, and build a template platform using the report task templates to obtain a template management platform.
[0066] The report generation system is constructed by using the data management platform, template management platform, and preset task management platform.
[0067] Furthermore, when the report generation program in memory 130 is executed by the processor, it also performs the following steps:
[0068] The initial metadata is cleaned and its type is defined to obtain the data type of the initial metadata;
[0069] The initial metadata is classified into security levels according to preset data characteristics to obtain the data security level of the initial metadata.
[0070] The platform is constructed based on the data type and data security level to obtain a data management platform.
[0071] Furthermore, when the report generation program in memory 130 is executed by the processor, it also performs the following steps:
[0072] The report generation request is parsed to obtain a data query strategy, and the data is queried through the data management platform according to the data query strategy to obtain report data;
[0073] Based on the report data, a task is generated through the template management platform to obtain the report generation task;
[0074] The report generation task is sent to the task management platform for task execution to obtain the data report.
[0075] Furthermore, when the report generation program in memory 130 is executed by the processor, it also performs the following steps:
[0076] The report generation task is sent to the execution queue through the task management platform for sorting, and the sorting result is obtained.
[0077] If the sorting result is successful, the report generation task is sent to the computing cluster for task execution, and the execution result is obtained.
[0078] If the execution result is successful, then obtain the initial data report;
[0079] The initial data report is anonymized and categorized according to the data security level of the data management platform. The data report is then obtained, and notification information is generated and sent to the corresponding users through the message center.
[0080] Furthermore, when the report generation program in memory 130 is executed by the processor, it also performs the following steps:
[0081] If the sorting result is a sorting failure, then obtain the queue status of the computing cluster;
[0082] An exception message is generated based on the queue status, and then sent to the corresponding user through the message center.
[0083] Furthermore, when the report generation program in memory 130 is executed by the processor, it also performs the following steps:
[0084] Perform scenario analysis on the report generation request to obtain the report scenario;
[0085] Based on the reported scenario, a missing data query is performed on the data report through the data management platform to obtain the query results;
[0086] If the query result indicates missing data, then the data report will be compensated to obtain the final data report.
[0087] This embodiment, through the above-described scheme, specifically receives a report generation request; inputs the report generation request into a pre-built report generation system, which then generates a data report based on the request. The report generation system is based on a pre-built data management platform, template management platform, and task management platform. By generating the report using the pre-built data management platform, template management platform, and task management platform within the report generation system, the problem of low efficiency caused by the lack of corresponding data management and task monitoring during report generation can be solved. Based on this invention, considering the increasing scale and demand for data in reality, such as customer complaint investigation, temporary business data statistics, regulatory data report submission, weekly report statistics, and value review data, various scenarios require statistical queries and report generation. Furthermore, business data involves user privacy information, requiring strict access control. Traditional generation schemes cannot solve the problems of data task lifecycle management and scheduling, as well as poor data task query performance, resulting in low efficiency. Therefore, a report generation method is designed, and its effectiveness is verified during report generation. Finally, the efficiency of report generation using this method is significantly improved.
[0088] Based on, but not limited to, the above-described terminal device architecture, embodiments of the present invention are proposed.
[0089] Reference Figure 2 , Figure 2 This is a flowchart illustrating an exemplary embodiment of the report generation method of the present invention. The report generation method includes:
[0090] Step S01: Receive report generation request;
[0091] The execution subject of the method in this embodiment can be a report generation device, a report generation terminal device, or a server. This embodiment takes a report generation device as an example, which can be integrated into a terminal device with data processing functions.
[0092] Before explaining the report generation method proposed in this embodiment, it should be understood that as business expands and develops, the scale of data and the demand for data usage increase. In various scenarios such as customer complaint investigation, temporary business data statistics, submission of regulatory data reports, weekly report data statistics, and value review data, it is necessary to perform statistical queries on data and generate reports. At the same time, business data involves user privacy information, and strict access control is required for data use. The known traditional data report generation scheme is as follows: apply for access permissions for each data table, write SQL or SQL-like statements, perform manual queries on the built-in CLI (command-line interface) tool, and export the results to generate the corresponding report.
[0093] However, traditional data report generation solutions suffer from problems such as lack of data task lifecycle management and scheduling, poor data task query performance, lack of data security hierarchical management, and complex task process links. Therefore, the method proposed in this embodiment is to use a pre-built report generation system to generate data reports. Specifically, a report generation request can be received first. The received report generation request can be issued by a user when they need to view data, or it can be used for the periodic generation of reports by government systems, enterprises, schools, etc.
[0094] Step S05: Input the report generation request into a pre-built report generation system, and the report generation system generates a data report based on the report generation request. The report generation system is based on a preset data management platform, template management platform, and task management platform.
[0095] Upon receiving a report generation request, it is input into a pre-built report generation system, which includes, but is not limited to, a data management platform, a template management platform, a task management platform, and a visualization module.
[0096] The report generation system generates reports based on report generation requests, resulting in data reports.
[0097] Based on the above solution, it is known that the generation of data reports in this embodiment mainly relies on the report generation system. Therefore, before step S05, which involves inputting the report generation request into the pre-built report generation system, the following method is also included:
[0098] Step S02: Extract data from a preset database to obtain initial metadata, and build a platform using the initial metadata to obtain a data management platform;
[0099] Step S03: Configure the template according to the data management platform, obtain the report task template, and build the template platform through the report task template to obtain the template management platform;
[0100] Step S04: The system is constructed using the data management platform, template management platform, and preset task management platform to obtain the report generation system.
[0101] The report generation system described in this embodiment is based on a data management platform and a template management platform. Therefore, the data management platform should be obtained first. Specifically, this can be achieved by analyzing the data storage situation. Since most of the data is mainly distributed in Hive clusters, data warehouses, and ClickHouse, it is necessary to define and manage the data. Therefore, data can be extracted from the databases (i.e., Hive clusters, data warehouses, and ClickHouse) to extract metadata. The metadata includes, but is not limited to, business metadata, technical metadata, and management metadata. After obtaining the specific metadata, in order to facilitate the management of the subsequently received data, the platform is built according to the obtained data to obtain the data management platform. Since Hive clusters, data warehouses, and ClickHouse correspond to different data warehouses, their query strategies and storage methods may be different. Therefore, this embodiment collects and manages data from multiple sources to avoid problems with data extraction.
[0102] After obtaining the data management platform, since the data reports generated in this embodiment are all generated in the form of task scheduling, a corresponding template management platform is also needed to manage the templates of the report generation tasks. Since the data management platform has corresponding data types, namely business metadata, technical metadata, and management metadata, various types of data are configured with private templates and public templates to obtain report task templates. The platform is then built using the report task templates to obtain the template management platform. Among them, public templates can be used by all system users, while private templates are for individual users to use. After a user creates a data retrieval template for a certain requirement, they can directly create a query task by modifying the query parameters configured in the template. The query task can then obtain the corresponding data and generate a data report.
[0103] After obtaining the data management platform and template management platform through the above scheme, the system can be built using the preset task management platform, data management platform and template management platform to obtain the report generation system.
[0104] More specifically, step S02, the step of building the platform using the initial metadata and obtaining the data management platform, includes:
[0105] Step S021: Clean and define the type of the initial metadata to obtain the data type of the initial metadata;
[0106] Step S022: Perform security classification on the initial metadata according to preset data characteristics to obtain the data security level of the initial metadata;
[0107] Step S023: Construct the platform based on the data type and data security level to obtain the data management platform.
[0108] After extracting from the database, the initial metadata (i.e., raw data) is cleaned and its types are defined to obtain (1) business metadata: defining the business meaning and business rules of the data, which allows users to quickly understand the business background of the data; (2) technical metadata: defining data related to data warehouse development, management and maintenance, including: data source information; description of data transformation; rules used for data cleaning and data updates; mapping from source data to destination data, etc.; (3) management metadata: defining data management related information, including table business person in charge, technical person in charge, etc. It should be clear that due to different objects, the defined data types may also be different. Therefore, the definition of data should be set according to the actual business needs. This embodiment does not make specific limitations.
[0109] The report generation method in this embodiment is applied to government and enterprise systems, such as banks. Therefore, there may be corresponding sensitive data, which requires sensitivity level marking. This embodiment uses an actual bank transaction system as an example, but it should be understood that the solution proposed in this embodiment should not be limited to bank systems, but may also include government systems, corresponding enterprise and educational systems, etc. The data classification strategies and measures adopted should be classified according to their specific data levels. This embodiment classifies data according to the field dimension based on the "Financial Data Security - Data Security Classification Guide". According to the affected objects and the degree of impact after the data security of financial institutions is compromised, the data security level is divided into 5 levels from high to low, as shown in the table below:
[0110]
[0111] Thus, the corresponding data security level was obtained;
[0112] Finally, the platform is built based on data type and data security level to obtain a data management platform.
[0113] Furthermore, such as Figure 3 As shown, Figure 3 This is a schematic diagram of the overall framework of the report generation method of the present invention.
[0114] The operation flow of the report generation system described in this embodiment is as follows:
[0115] First, data is stored through the data storage module, including MySQL and HBase, while the received metadata comes from Hive, ClickHouse, and unified logs, etc.
[0116] Then, the core services that the report generation system can provide include task processing, task monitoring, message notification, access control, data processing, sensitive data de-identification, API callback, and task compensation.
[0117] Then, the gateway services corresponding to the report generation system include login authentication, routing forwarding, traffic control, and log recording;
[0118] Then, the web services corresponding to the report generation system include task management, template management, data editing, permission management, result export, task monitoring, field examples, and tag management, etc.
[0119] Finally, in this embodiment, the operation objects provided by the report generation system are developers and business personnel. Based on the above services, they complete login authentication, message notification, scheduled task management, data anonymization, OLAP (Online Analytical Processing), and data collection.
[0120] This embodiment, through the above-described scheme, specifically receives a report generation request; inputs the report generation request into a pre-built report generation system, which then generates a data report based on the request. The report generation system is based on a pre-defined data management platform, template management platform, and task management platform. Thus, by generating the report using the pre-built data management platform, template management platform, and task management platform within the report generation system, the efficiency of report generation is improved, solving the problem of low efficiency caused by the lack of corresponding data management and task monitoring during report generation.
[0121] Reference Figure 4 , Figure 4 This is a flowchart illustrating another exemplary embodiment of the report generation method of the present invention.
[0122] Based on the above Figure 2 In the embodiment shown, step S05, whereby the report generation system generates a data report based on the report generation request, includes:
[0123] Step S051: Parse the report generation request, obtain the data query strategy, and perform data query through the data management platform according to the data query strategy to obtain report data;
[0124] Step S052: Based on the report data, generate a report generation task through the template management platform;
[0125] Step S053: Send the report generation task to the task management platform for task execution and obtain the data report.
[0126] Through the above Figure 2 The embodiment shown shows that the report generation system has been built. Therefore, the received report generation request is parsed to know the data type and other information that the user wants to obtain, and a data query strategy is generated accordingly. Since a data management system is configured in the report generation system, the data that the user wants to obtain is extracted. Specifically, the data management platform is queried through the pre-configured data processing component Mapper and query statement. In this embodiment, the data query component and statement are also optimized. Specifically, the following can be done: (1) Job optimization. Since the log data is stored in HDFS in files of different sizes, the performance impact and resource waste caused by too many small files during the MR query process need to be considered. Therefore, the task submission needs reasonable parameter configuration. This system configures the merging of small files and the configuration of single-time By adjusting the upper limit of Mapper memory and configuring the number of Mappers and Reducers, parameter tuning was completed, ensuring efficient querying of tasks and effective utilization of cluster resources. (2) SQL tuning: Since the amount of data processed by Hive is usually at the TB level, SQL tuning is very important. It mainly includes: 1) Improving query performance: By optimizing query plans, index design, data partitioning, etc., unnecessary data scanning and calculation can be reduced, thereby speeding up the query. 2) Reducing resource consumption: By reasonably designing data partitioning and using indexes, unnecessary data movement and reading can be reduced, thereby reducing resource consumption. 3) Optimizing data storage and compression: For large-scale datasets, data compression technology can be used to reduce storage space and I / O overhead. Compressed data can be loaded and queried faster.
[0127] After obtaining report data through the aforementioned data processing components and query statements, the task is generated through the template management platform to obtain the report generation task. After obtaining the report generation task, it is input into the task management platform for task execution, and finally the data report is obtained.
[0128] Specifically, step S053 above, which involves sending the report generation task to the task management platform for task execution and obtaining the data report, includes:
[0129] Step S0531: Through the task management platform, the report generation task is sent to the execution queue for queue sorting, and the sorting result is obtained;
[0130] Step S0534: If the sorting result is successful, the report generation task is sent to the computing cluster for task execution and the execution result is obtained.
[0131] Step S0535: If the execution result is successful, then obtain the initial data report;
[0132] Step S0536: Based on the data security level of the data management platform, the initial data report is anonymized and classified to obtain the data report, and a notification message is generated through the message center and sent to the corresponding user.
[0133] In the task management platform, there may be a large number of report generation tasks. Therefore, the previously generated report generation tasks need to be sent to the execution queue for sorting. During the sorting process, the resources of the computing cluster will be calculated. If the resources in the computing cluster can execute the current report generation task, then the report generation task will be sent to the computing cluster for calculation to avoid causing system failure. Priority can also be set for tasks before sorting. This way, when dealing with more urgent tasks, they can be processed first (for example, clearing resources for currently unimportant tasks to obtain more resources for processing), solving the problem of untimely business feedback.
[0134] Once the sorting is complete and the task is executed, an initial data report is obtained. However, there is a data sensitivity issue at this point. In the above embodiment, multiple data levels have been set, so the obtained initial data report should be anonymized. Data anonymization is based on data hierarchy, and the hierarchical anonymization rules are as follows:
[0135]
[0136] The higher the data level, the more stringent the data anonymization algorithm. For example, this system uses a generalized algorithm to anonymize addresses and years, an encryption algorithm to anonymize highly sensitive data such as card numbers, ID numbers and phone numbers, and an inhibition algorithm to anonymize sensitive information such as email addresses and SMS messages.
[0137] It should be clear that by building a complete report generation system, the task lifecycle from submission to termination is divided into fine-grained segments. The task list adopts a timed refresh strategy, allowing users to intuitively see the stage of the task and determine whether the task has encountered an execution error or has been completed. Once a task is successfully or unsuccessfully executed, the system will promptly send information to the data user via push notification, informing them to view the report results.
[0138] More specifically, after step S0531 above, which involves sending the report generation task to the execution queue for sorting through the task management platform and obtaining the sorting result, the method further includes:
[0139] Step S0532: If the sorting result is a sorting failure, then obtain the queue status of the computing cluster;
[0140] Step S0533: Generate exception information based on the queue status, and send the exception information to the corresponding user through the message center.
[0141] The execution and sorting of tasks may encounter task exceptions and timeouts. Therefore, when sorting fails, the queue and computing cluster are monitored for anomalies to obtain the queue status.
[0142] Anomaly information is generated based on queue status and then sent to the corresponding user via the message center.
[0143] like Figure 5 As shown, Figure 5 This is a schematic diagram illustrating the method for generating reports according to the present invention, which involves sending abnormal information to the user.
[0144] First, submit tasks to the queue. If the submission is successful, sort the queue and then allocate cluster resources for execution.
[0145] Then, if resource allocation times out or the execution of a computing cluster task times out, a task exception message is sent.
[0146] Finally, the task exception information is sent to the corresponding user through the message center.
[0147] like Figure 6 As shown, Figure 6 This is a schematic diagram illustrating the report generation task management involved in the report generation method of the present invention.
[0148] First, users submit report generation tasks to a queue for sorting;
[0149] Then, when a task is executed, it is submitted to the computing cluster for task execution. If there are too many tasks in the queue, the queue overflows and no more tasks can be added to the queue.
[0150] Then, after the task is executed, the system can output that the task is completed, or that there is an error or timeout. If the task fails or times out, it is considered a failure, and the system will send the information to the corresponding user through the message center.
[0151] Then, once the task is completed, the data is anonymized, and the final data report and a notification of successful report generation are sent to the corresponding user.
[0152] Finally, this embodiment also adds a task delay mechanism to more reliably release cluster resources occupied by invalid tasks. There is a certain time interval between submitting a task to the cluster via JDBC and the task being received and executed by the cluster. If the task is stopped using the applicationId according to the traditional method, the tasks during the interval cannot be stopped. These tasks will still be received and executed by the cluster later, resulting in a waste of cluster resources. To address this situation, this system adds a task delay stop mechanism. When a task receives a stop signal but has not yet generated an applicationId, the task is placed in a false stop state. At this time, the task execution process log is queried, and the cluster task is stopped only after the task generates an applicationId. In addition, if a task is in the queue and has not yet been submitted to the cluster for processing, and a task stop signal is received at this time, the task will not be submitted.
[0153] Furthermore, based on the above Figure 2 In the embodiment shown, after step S05, where the report generation system generates a data report based on the report generation request, the method further includes:
[0154] Step S06: Perform scenario analysis on the report generation request to obtain the report scenario;
[0155] Step S07: Based on the report scenario, perform a missing data query on the data report through the data management platform to obtain the query results;
[0156] Step S08: If the query result indicates missing data, perform task compensation on the data report to obtain the final data report.
[0157] Since scheduled tasks may fail to execute successfully on a particular day due to various reasons, resulting in data loss, self-service reports need to be generated periodically for scenarios such as development weekly reports, value review, and MGM weekly reports. Therefore, this embodiment adds a task compensation mechanism to rerun the task and replenish the data for instances where scheduled tasks fail to execute, reducing tedious data replenishment work. Specifically, it can be done as follows:
[0158] By performing scenario analysis on report generation, such as the aforementioned development weekly report, value review, and MGM weekly report, missing data queries can be performed on the data management platform.
[0159] If the query results show missing data, the data report will be compensated to obtain the final data report, thus avoiding missing data.
[0160] This embodiment also provides visualization processing for the generated data reports. The system provides different types of visualization templates such as dashboards, line charts and radar charts. Users can select different templates to visualize the data according to their specific needs and scenarios.
[0161] This embodiment, through the above-described scheme, specifically parses the report generation request to obtain a data query strategy, and performs data querying through the data management platform according to the data query strategy to obtain report data; based on the report data, it generates a task through the template management platform to obtain a report generation task; and sends the report generation task to the task management platform for task execution to obtain the data report. Thus, it realizes the generation of data reports, solves the problem of low efficiency caused by the lack of corresponding data management and task monitoring during report generation, and improves the efficiency of report generation.
[0162] Furthermore, embodiments of the present invention also propose a report generation device, the report generation device comprising:
[0163] The receiving module is used to receive report generation requests;
[0164] The generation module is used to input the report generation request into a pre-built report generation system, and the report generation system generates a data report based on the report generation request. The report generation system is based on a preset data management platform, template management platform, and task management platform.
[0165] Furthermore, this embodiment of the invention also proposes a terminal device, which includes a memory, a processor, and a report generation program stored in the memory and executable on the processor. When the report generation program is executed by the processor, it implements the steps of the report generation method described above.
[0166] Since this report generation program employs all the technical solutions of all the aforementioned embodiments when executed by the processor, it possesses at least all the beneficial effects brought about by all the technical solutions of all the aforementioned embodiments, which will not be elaborated upon here.
[0167] Furthermore, embodiments of the present invention also propose a computer-readable storage medium storing a report generation program, which, when executed by a processor, implements the steps of the report generation method described above.
[0168] Since this report generation program employs all the technical solutions of all the aforementioned embodiments when executed by the processor, it possesses at least all the beneficial effects brought about by all the technical solutions of all the aforementioned embodiments, which will not be elaborated upon here.
[0169] Compared to existing technologies, the report generation method, apparatus, terminal device, and storage medium proposed in this invention receive a report generation request; input the report generation request into a pre-built report generation system, which then generates a data report based on the request. The report generation system is based on a preset data management platform, template management platform, and task management platform. This solves the problem of low efficiency caused by the lack of corresponding data management and task monitoring during report generation, thus improving the efficiency of report generation. Based on this invention, given the increasing scale and demand for data in reality, such as customer complaint investigation, temporary business data statistics, regulatory data report submission, weekly report statistics, and value review data, various scenarios require statistical queries and report generation. Furthermore, business data involves user privacy information, requiring strict access control. Traditional generation methods cannot solve the problems of data task lifecycle management and scheduling, as well as poor data task query performance, resulting in low efficiency. Therefore, this invention designs a report generation method, and verifies its effectiveness during report generation. Finally, the efficiency of report generation using this invention is significantly improved.
[0170] Compared with existing technologies, the solutions of the embodiments of the present invention have the following advantages:
[0171] 1. Complete task lifecycle and task scheduling management;
[0172] 2. Optimize task execution performance;
[0173] 3. Data security hierarchical management;
[0174] 4. Standardize the report generation process.
[0175] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or system. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or system that includes that element.
[0176] The sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.
[0177] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, controlled terminal, or network device, etc.) to execute the methods of each embodiment of the present invention.
[0178] The above are merely preferred embodiments of the present invention and do not limit the scope of the patent. Any equivalent structural or procedural transformations made based on the description and drawings of the present invention, or direct or indirect applications in other related technical fields, are similarly included within the scope of patent protection of the present invention.
Claims
1. A report generation method characterized by, The report generation method includes the following steps: Receive report generation request; The report generation request is input into a pre-built report generation system, which generates a data report based on the report generation request. The report generation system is based on a preset data management platform, template management platform, and task management platform. The step of inputting the report generation request into the pre-built report generation system also includes: Data is extracted from a pre-defined database to obtain initial metadata, and a platform is built using the initial metadata to obtain a data management platform. Configure templates according to the data management platform, obtain report task templates, and build a template platform using the report task templates to obtain a template management platform. A report generation system is obtained by constructing the data management platform, template management platform, and preset task management platform. The steps of building the platform using the initial metadata and obtaining the data management platform include: The initial metadata is cleaned and its type is defined to obtain the data type of the initial metadata. The data type includes business metadata, technical metadata and management metadata. The technical metadata includes data source information, description of data transformation, rules used for data cleaning and updating, and mapping from source data to destination data. The initial metadata is classified into security levels according to preset data characteristics to obtain the data security level of the initial metadata. A data management platform is constructed based on the data type and data security level described above. The step of generating a data report by the report generation system based on the report generation request includes: The report generation request is parsed to obtain a data query strategy, and the data is queried through the data management platform according to the data query strategy to obtain report data. This includes querying the data management platform through a pre-configured data processing component Mapper and a query statement, and optimizing the data processing component and the query statement. Based on the report data, a task is generated through the template management platform to obtain the report generation task; The report generation task is sent to the task management platform for task execution to obtain the data report.
2. The report generating method of claim 1, wherein, The step of sending the report generation task to the task management platform for task execution and obtaining the data report includes: The report generation task is sent to the execution queue through the task management platform for sorting, and the sorting result is obtained. If the sorting result is successful, the report generation task is sent to the computing cluster for task execution, and the execution result is obtained. If the execution result is successful, then obtain the initial data report; The initial data report is anonymized and categorized according to the data security level of the data management platform. The data report is then obtained, and notification information is generated and sent to the corresponding users through the message center.
3. The report generation method according to claim 2, characterized in that, After the step of sending the report generation task to the execution queue for sorting and obtaining the sorting result through the task management platform, the method further includes: If the sorting result is a sorting failure, then obtain the queue status of the computing cluster; An exception message is generated based on the queue status, and then sent to the corresponding user through the message center.
4. The report generation method according to claim 1, characterized in that, Following the step of generating a data report by the report generation system based on the report generation request, the following further steps are included: Perform scenario analysis on the report generation request to obtain the report scenario; Based on the reported scenario, a missing data query is performed on the data report through the data management platform to obtain the query results; If the query result indicates missing data, then the data report will be compensated to obtain the final data report.
5. A report generation device, characterized in that, The report generation device includes: The receiving module is used to receive report generation requests; The generation module is used to input the report generation request into a pre-built report generation system, and the report generation system generates a data report based on the report generation request. The report generation system is based on a preset data management platform, template management platform, and task management platform. The module is used to extract data from a preset database to obtain initial metadata, and then build a platform using the initial metadata to obtain a data management platform; configure templates based on the data management platform to obtain report task templates, and then build a template platform using the report task templates to obtain a template management platform; build a system using the data management platform, template management platform, and preset task management platform to obtain a report generation system; clean and define the types of the initial metadata to obtain its data type, which includes business metadata, technical metadata, and management metadata. The technical metadata includes data source information, descriptions of data transformation, rules used for data cleaning and updating, and mapping from source data to destination data; classify the initial metadata for security based on preset data characteristics to obtain its data security level; and build a platform using the data type and data security level to obtain the data management platform. The generation module is further configured to parse the report generation request, obtain a data query strategy, and perform data querying through the data management platform according to the data query strategy to obtain report data. This includes querying the data management platform using a pre-configured data processing component (Mapper) and a query statement, and optimizing the data processing component and the query statement. Based on the report data, a task is generated through the template management platform to obtain a report generation task. The report generation task is then sent to the task management platform for execution to obtain the data report.
6. A terminal device, characterized in that, The terminal device includes a memory, a processor, and a report generation program stored in the memory and executable on the processor. When the report generation program is executed by the processor, it implements the steps of the report generation method as described in any one of claims 1-4.
7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a report generation program, which, when executed by a processor, implements the steps of the report generation method as described in any one of claims 1-4.
Citation Information
Patent Citations
Report making method and system
CN102081757A
Report generation system and method based on big data platform
CN108804402A
Data management method, system, device and equipment and storage medium
CN116611080A
