Statistical report generation method and system, electronic equipment and storage medium

Abstract

The invention relates to the technical field of data processing, and discloses a statistical report generation method and system, electronic equipment and a storage medium, and the method comprises the steps of counting a configuration request, formulating a statistical statement template, and creating a statistical task; triggering a batch processing process according to timing configuration, executing a statistical task, performing batch calculation on the source data set associated with the statistical statement template, generating a statistical result, and persistently storing the statistical result in a specified database; selecting a query mode according to the visual query request, an identifier corresponding to the statistical task and a time range selected by a user, and performing formatting and merging calculation on a query result to generate a visual data set; performing rendering calculation on the visual data set according to the visual template configuration, generating chart data and displaying the chart data; and identifying and migrating the historical statistical data, receiving a query request for the archived data through an independent query interface, and returning a corresponding result. According to the invention, diversified statistical requirements can be met, and the integrity and flexibility of data tracing are guaranteed.

Description

Technical Field

[0001] This invention relates to the field of data processing, and in particular to a method, system, electronic device, and storage medium for generating statistical reports. Background Technology

[0002] Currently, data has become one of the core assets of enterprises and organizations. Statistical reporting systems, as a key component of project management, undertake important tasks such as data collection, filtering, quantitative statistics, and visual monitoring, playing an indispensable role in ensuring the normal operation of business and the subsequent operation and maintenance of projects. Promoting visual management of statistical reports, providing refined tracking of user habits, and quantitative statistics of user access to ultimately achieve visualized data management has become an inevitable trend and a crucial responsibility.

[0003] With the continuous development of business and the increasing number of user visits, the statistical reporting system faces higher requirements and challenges: 1) The traditional reporting mode with a single data flow can no longer meet the dual requirements of real-time performance and flexibility, and is difficult to cope with complex and ever-changing business scenarios; 2) The display and calculation logic of statistical reports are tightly coupled, making it difficult to adapt to diverse display needs, affecting user experience and system scalability; 3) The rapid expansion of the source dataset has led to increased system response latency and low data processing efficiency; 4) The same calculation scenario is executed multiple times following business queries, wasting a lot of computing and communication resources.

[0004] Therefore, this application provides a method for generating statistical reports to solve the above-mentioned technical problems. Summary of the Invention

[0005] The purpose of this invention is to provide a statistical report generation method, system, electronic device, and storage medium to solve the technical problems of low data processing volume and efficiency, and poor adaptability to business scenarios in the prior art.

[0006] To address the aforementioned technical problems, this invention provides a method for generating statistical reports, comprising:

[0007] In response to a statistical configuration request, a statistical statement template is defined, and a statistical task is created based on the statistical statement template. The statistical statement template includes statistical logic, result definition, dynamic parameter placeholders, and associated data source information.

[0008] The batch processing procedure is triggered according to the timed configuration, the statistical task is executed, the source dataset associated with the statistical statement template is calculated in batches, the statistical results are generated, and the statistical results are persistently stored in the specified database.

[0009] In response to a visualization query request, based on the identifier corresponding to the statistical task and the time range selected by the user, a query mode is selected, and real-time statistical queries are performed on the source dataset and / or the batch processing results are queried. The query results are then formatted and merged to generate a visualization dataset.

[0010] Based on the visualization template configuration, the visualization dataset is rendered and calculated to generate chart data, which is then displayed on the front-end page.

[0011] Based on the preset archiving strategy, historical statistical data is identified and migrated, data that meets the archiving conditions is transferred to the archiving storage area, archived data is cleaned up in the database, and query requests for archived data are received through an independent query interface and the corresponding results are returned.

[0012] In some specific embodiments, in response to a statistical configuration request, a statistical statement template is formulated, and a statistical task is created based on the statistical statement template, further including:

[0013] Define a statistical statement template, including statistical logic, statistical dimensions, indicator types, dynamic parameter placeholders, and associated data source identifiers;

[0014] Create a statistics task, and configure its name, description, execution priority, associated statistics statement template identifier, and statistics engine type;

[0015] Dynamically replace parameter placeholders in the statistical statement template according to business needs to generate executable statistical statements;

[0016] Associate the statistics task with the timer executor and configure the execution cycle and trigger conditions.

[0017] In some specific embodiments, a batch processing procedure is triggered according to a timed configuration to execute the statistical task, perform batch calculations on the source dataset associated with the statistical statement template, generate statistical results, and persistently store the statistical results in a specified database, further including:

[0018] Statistical tasks are triggered by a timer to perform statistical calculations.

[0019] Perform batch queries and calculations on the source dataset to generate statistical results.

[0020] The statistical results data are classified to distinguish between total results data and grouped statistical results data, and different parsing and storage strategies are adopted accordingly.

[0021] The statistical results are persistently stored in the batch processing results database, and the statistical dimensions and values ​​are recorded.

[0022] In some specific embodiments, in response to a visualization query request, a query mode is selected based on the identifier corresponding to the statistical task and the time range selected by the user. Real-time statistical queries are performed on the source dataset and / or on the batch processing results. The query results are then formatted and merged to generate a visualization dataset, further including:

[0023] Receive query requests and extract statistical task identifiers and time range parameters;

[0024] Determine the query mode based on the time range parameter, and select real-time query, batch result query, or mixed query;

[0025] The data returned by the query is standardized in format and merged and classified according to statistical dimensions to generate a visualization dataset that meets the requirements of the visualization template.

[0026] In some specific embodiments, the visualization dataset is rendered and calculated according to the visualization template configuration to generate chart data, which is then displayed on the front-end page, further including:

[0027] Define the display type, chart style, and data rendering engine type based on the visualization template;

[0028] Perform personalized calculations on visualized datasets, including dimensional classification and summarization, time series analysis, and proportion calculation;

[0029] The calculated visualization data is organized into display groups to generate chart plotting data;

[0030] Charts and data are rendered and displayed via the front-end page according to the configuration.

[0031] In some specific embodiments, historical statistical data is identified and migrated according to a preset archiving strategy, data that meets the archiving conditions is transferred to the archiving storage area, archived data is cleaned up in the database, and query requests for archived data are received and returned through an independent query interface, further including:

[0032] Execute data migration tasks according to the preset archiving cycle and rules;

[0033] Identify historical statistical data to be archived and migrate it to the designated archive storage area;

[0034] Clean up migrated data records in the database to free up storage space;

[0035] The system receives query requests for archived data through an independent query interface and returns the corresponding results.

[0036] In some specific embodiments, the method further includes:

[0037] A multi-threaded parallel strategy is used to execute multiple statistical tasks during batch processing;

[0038] Cache the statistical results of multiple accesses during the data query process;

[0039] Perform breakpoint resume and exception retry operations during the archiving process;

[0040] Perform data drill-down, linked analysis, and conditional filtering operations within the visualization.

[0041] Based on the same concept, the present invention also provides a statistical report generation system, comprising:

[0042] The statistics request configuration module is configured to respond to a statistics configuration request, specify a statistics statement template, and create a statistics task based on the statistics statement template. The statistics statement template includes statistics logic, result definition, dynamic parameter placeholders, and associated data source information.

[0043] The statistical result generation module is configured to trigger a batch processing procedure according to a timer configuration, execute the statistical task, perform batch calculations on the source dataset associated with the statistical statement template, generate statistical results, and persistently store the statistical results in a specified database.

[0044] The visualization data generation module is configured to respond to visualization query requests, select a query mode based on the identifier corresponding to the statistical task and the time range selected by the user, perform real-time statistical queries on the source dataset and / or query the batch processing results, and format and merge the query results to generate a visualization dataset.

[0045] The chart data display module is configured to render and calculate the visualization dataset according to the visualization template configuration, generate chart data, and display it on the front-end page;

[0046] The archiving module is configured to identify and migrate historical statistical data according to a preset archiving strategy, transfer data that meets the archiving conditions to the archiving storage area, clean up archived data in the database, and receive query requests for archived data through an independent query interface and return the corresponding results.

[0047] Based on the same concept, the present invention also provides an electronic device, including: a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; the memory stores a computer program, and when the computer program is executed by the processor, the processor performs the steps of a statistical report generation method.

[0048] Based on the same concept, the present invention also provides a computer-readable storage medium storing a computer program executable by an electronic device, which, when run on the electronic device, causes the electronic device to perform the steps of a statistical report generation method.

[0049] Compared with existing technologies, its advantages are as follows:

[0050] This invention discloses a statistical report generation method, system, electronic device, and storage medium. By separating real-time stream processing and offline batch processing modes, it achieves a two-layer architecture design for the reporting system, balancing real-time performance and accuracy requirements. The concept of a display view is introduced, making the final visualization more flexible and configurable. The concepts of statistical templates and statistical tasks are introduced, allowing batch processing tasks for data statistics to be largely decoupled from business logic, functioning solely as a data calculation process. A timed executor performs pre-calculation for offline batch processing and persists the calculation results, reducing the frequency of repetitive calculations and improving query response speed. A data archiving engine is designed and implemented, migrating cold data to a low-cost storage area according to a preset strategy, freeing up main storage resources and optimizing system performance. It can efficiently support diverse statistical needs under massive data volumes while ensuring the integrity and flexibility of data traceability. Attached Figure Description

[0051] Other features, objects, and advantages of this application will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings:

[0052] Figure 1 This is a flowchart illustrating some specific embodiments of the statistical report generation method of the present invention;

[0053] Figure 2 This is one of the flowcharts of a statistical report generation method according to another embodiment of the present invention;

[0054] Figure 3 This is a second flowchart illustrating another embodiment of the statistical report generation method of the present invention;

[0055] Figure 4 This is a third flowchart illustrating another embodiment of the statistical report generation method of the present invention;

[0056] Figure 5 This is a fourth flowchart illustrating another embodiment of the statistical report generation method of the present invention;

[0057] Figure 6 This is the fifth flowchart of another embodiment of the statistical report generation method of the present invention;

[0058] Figure 7This is a flowchart of another embodiment of the statistical report generation method of the present invention;

[0059] Figure 8 This is a schematic diagram of the structure of a statistical report generation system according to some specific embodiments of the present invention;

[0060] Figure 9 This is a schematic diagram of the structure of an electronic device according to some specific embodiments of the present invention;

[0061] In the diagram, 710 is the processor; 720 is the memory; 730 is the input device; and 740 is the output device. Detailed Implementation

[0062] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0063] The terminology used in the embodiments of this application is for the purpose of describing particular embodiments only and is not intended to limit the application. The singular forms “a,” “said,” and “the” used in the embodiments of this application and the appended claims are also intended to include the plural forms, and “multiple” generally includes at least two unless the context clearly indicates otherwise.

[0064] It should be understood that the term "and / or" used in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, the character " / " in this article generally indicates that the preceding and following related objects have an "or" relationship.

[0065] It should be understood that although the terms first, second, third, etc., may be used in the embodiments of this application, these descriptions should not be limited to these terms. These terms are only used to distinguish the descriptions. For example, first may also be referred to as second without departing from the scope of the embodiments of this application, and similarly, second may also be referred to as first.

[0066] Depending on the context, the words “if” or “suppose” as used here can be interpreted as “when” or “in response to determination” or “in response to detection.” Similarly, depending on the context, the phrases “if determination” or “if detection (of the stated condition or event)” can be interpreted as “when determination” or “in response to determination” or “when detection (of the stated condition or event)” or “in response to detection (of the stated condition or event).”

[0067] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that an article or device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such an article or device. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the article or device that includes said element.

[0068] It should be noted that any symbols and / or numbers present in the specification that are not marked in the accompanying drawings are not reference numerals.

[0069] Reference Figure 1 A method for generating statistical reports, comprising:

[0070] S101, in response to the statistical configuration request, a statistical statement template is formulated, and a statistical task is created based on the statistical statement template, wherein the statistical statement template includes statistical logic, result definition, dynamic parameter placeholders, and associated data source information;

[0071] S102, according to the timed configuration, the batch processing process is triggered to execute the statistical task, perform batch calculations on the source dataset associated with the statistical statement template, generate statistical results, and persistently store the statistical results in the specified database;

[0072] S103, in response to the visualization query request, according to the identifier corresponding to the statistical task and the time range selected by the user, select the query mode, perform real-time statistical query on the source dataset and / or query the batch processing results, and format and merge the query results to generate a visualization dataset;

[0073] S104. According to the visualization template configuration, the visualization dataset is rendered and calculated to generate chart data, which is then displayed on the front-end page.

[0074] S105 identifies and migrates historical statistical data according to the preset archiving strategy, transfers data that meets the archiving conditions to the archiving storage area, cleans up the archived data in the database, and receives query requests for archived data through an independent query interface and returns the corresponding results.

[0075] Specifically, in this embodiment of the invention, in response to a statistical configuration request, a statistical statement template is created. This template includes a statistical logical structure, result data definitions, parameter placeholders for dynamic replacement, and associated data source information identifiers. Based on this template, a statistical task is created, and its execution attributes and triggering conditions are configured. A batch processing process is triggered according to a preset timer configuration, executing the created statistical task, performing batch calculations on the source dataset associated with the statistical statement template, generating statistical result data, and persistently storing the results in a designated result database. When a visualization query request is received, the query mode is automatically selected based on the statistical task identifier and the user-selected time range. This allows for real-time statistical queries on the source dataset, queries on batch processing results, or mixed queries. The returned results are uniformly formatted and merged and categorized according to statistical dimensions to generate a dataset for visualization. The dataset is then rendered and calculated according to a pre-configured visualization template, including data conversion and graph generation based on the display type, and chart drawing and display are completed on the front-end page. The archiving process is executed periodically. Historical statistical data is identified according to preset strategies. Data that meets the migration conditions is transferred to the archive storage area. At the same time, archived data in the original database is cleaned up. The system receives retrieval requests for archived data through an independent query interface, locates the data, and returns the corresponding results.

[0076] For example, the statistical statement template can be an SQL query template defined in the ClickHouse database, such as: "SELECT COUNT(DISTINCT user_id) AS uv, event_date FROM user_events WHERE event_date BETWEEN{start_date}AND{end_date}GROUP BY event_date", where "{start_date}" and "{end_date}" are dynamic parameter placeholders. The statistical task is set to execute at 2:00 AM daily, replacing the placeholders with specific date values ​​such as "2024-01-01" and "2024-01-07". It performs batch queries on the user event table for this time period, calculates the daily user visits, and stores the results in a batch processing result table. When a user requests to view the visit trend for the first week of January 2024, they select the batch processing result query mode to obtain the pre-calculated results, which are then merged into a line chart by day. During rendering, dates and user counts are mapped to coordinate points, such as (2024-01-01, 1200) and (2024-01-02, 1350), which are then displayed as line charts on the front end. The archiving strategy is set to migrate data older than three months to S3 storage while deleting the original data from ClickHouse. When users query historical data, they access S3 through a separate interface and receive the results.

[0077] In some applications, in response to a statistical configuration request, a statistical statement template is defined, and a statistical task is created based on the statistical statement template. This includes defining the statistical statement template, which includes statistical logic, statistical dimensions, indicator types, dynamic parameter placeholders, and associated data source identifiers; creating the statistical task, configuring its name, description, execution priority, associated statistical statement template identifier, and statistical engine type; dynamically replacing parameter placeholders in the statistical statement template according to business requirements to generate executable statistical statements; and associating the statistical task with a timer executor, configuring the execution cycle and triggering conditions.

[0078] Understandably, in response to a statistical configuration request, a statistical statement template is defined. This template includes the statistical logical structure, statistical dimension division method, indicator value type definition, dynamic placeholders for parameterized replacement, and associated source data identification information. Based on this template, a statistical task is created, configuring its task name, function description, execution priority order, unique identifier of the associated statistical statement template, and type attribute of the statistical calculation engine. According to the actual business scenario requirements, the preset parameter placeholders in the statistical statement template are dynamically replaced to generate a complete and executable statistical statement. The statistical task is associated with a scheduled execution mechanism, configuring its execution cycle frequency and task triggering conditions, thereby completing the creation and scheduling configuration of the statistical task.

[0079] For example, a statistical statement template can be defined as follows: In a structured query, the statistical logic is set to calculate the number of user visits within a specified time period. The statistical dimensions are date and user type, the metric type is integer count, and the dynamic parameter placeholders include a start time {start_date} and an end time {end_date}. The associated data source is identified as the user behavior log table user_access_log. The statistical task is named "Daily User Access Statistics," described as "Statistics on various user visits per day," with high priority, associated template ID TEMPLATE_01, and the statistical engine type is batch processing engine. Replace parameter placeholders according to business requirements. For example, replace {start_date} with 2024-03-01 and {end_date} with 2024-03-31 to generate an executable statement: SELECT access_date,user_type,COUNT(*)AS access_countFROM user_access_log WHERE access_date BETWEEN'2024-03-01'AND'2024-03-31'GROUP BY access_date,user_type. Associate the statistics task with a scheduled executor, configure the execution cycle to run once a day at 2 AM, and trigger it automatically when the system is idle.

[0080] In some applications, a batch processing procedure is triggered based on a timed configuration to execute the statistical task, perform batch calculations on the source dataset associated with the statistical statement template, generate statistical results, and persistently store the statistical results in a designated database. This includes triggering the statistical task based on a timed executor to perform statistical calculations; performing batch queries and calculations on the source dataset to generate statistical result data; determining the type of the statistical result data, distinguishing between total result data and grouped statistical result data, and adopting different parsing and storage strategies; persistently storing the statistical result data in a batch processing result database, and recording the statistical dimensions and statistical values.

[0081] Understandably, the statistical task is triggered by a scheduled execution mechanism based on the preset timer configuration, and the statistical calculation process is executed; batch query and calculation operations are performed on the source dataset associated with the statistical statement template to generate statistical result data; the generated statistical result data is judged by type, distinguishing between total result data and grouped statistical result data, and different data parsing methods and storage strategies are adopted according to the type differences; the processed statistical result data is persistently stored in the specified batch processing result database, and the corresponding statistical dimension information and statistical values ​​are recorded during the storage process.

[0082] For example, the scheduled configuration is set to trigger the statistical task at 3 AM daily, initiating the statistical calculation process through a scheduled mechanism. A batch query is performed on the source data table `user_behavior_log`, executing the statistical statement "SELECT COUNT(*) AS total_visits,visit_date FROM user_behavior_log WHERE visit_date='2024-05-20' GROUP BY visit_date", generating statistical results. The results are then type-checked to distinguish between total results (e.g., total visits of 15000) and grouped statistical results (e.g., grouped by date). Total results are stored as a single record, labeled "Total"; grouped statistical results are stored as multiple records, each containing the specific grouping dimension value (e.g., 2024-05-20) and the corresponding statistical value (e.g., 15000). The statistical results are persistently stored in the daily_visits table of the batch processing results database stat_result_db, with the statistical dimension field being visit_date and the statistical value field being total_visits, and the corresponding numerical data is saved.

[0083] In some applications, in response to a visualization query request, a query mode is selected based on the identifier corresponding to the statistical task and the time range selected by the user. Real-time statistical queries are performed on the source dataset and / or on the batch processing results. The query results are then formatted and merged to generate a visualization dataset. This process includes receiving the query request, extracting the statistical task identifier and time range parameters; determining the query mode based on the time range parameters, selecting real-time query, batch processing result query, or mixed query; standardizing the format of the data returned by the query; and merging and classifying the data according to statistical dimensions to generate a visualization dataset that meets the requirements of the visualization template.

[0084] Understandably, in response to a received visualization query request, the system extracts the statistical task identifier and the user-selected time range parameter; it automatically determines and selects the query mode based on the length of the time range parameter and business needs, including real-time statistical queries on the source dataset, queries on batch processing results, or hybrid queries combining the two query modes; it performs format standardization processing on the raw data returned by the query, including unifying data field formats, converting numerical units, and handling null values, and performs data merging and classification calculations according to the statistical dimension definition, ultimately generating a structured visualization dataset that meets the requirements of the visualization template.

[0085] For example, upon receiving a user query request, the system extracts the statistical task identifier "TASK_UV_DAILY" and the time range parameter "2024-06-01" to "2024-06-07". Determining that this time range belongs to recent data, a mixed query mode is selected: data from 2024-06-01 to 2024-06-05 is queried from the batch processing result table `precomputed_uv`, while data from 2024-06-06 to 2024-06-07 is directly queried from the source data table `user_access_log` in real time. The returned data is standardized in format: date fields are uniformly converted to YYYY-MM-DD format, access count values ​​are uniformly converted to integers, and missing data is filled with a default value of 0. The batch processing results and real-time query results are merged by date dimension. The total daily visits are calculated, and a visualization dataset containing 7 records is generated. Each record contains a date field and a number field of visits. For example, 2024-06-01 corresponds to 12,500 visits, 2024-06-02 corresponds to 11,800 visits, and so on up to 2024-06-07 corresponds to 13,200 visits. The dataset is then output for use by the visualization component.

[0086] In some applications, the visualization dataset is rendered and calculated according to the visualization template configuration to generate chart data, which is then displayed on the front-end page. This includes defining the display type, chart style, and data rendering engine type according to the visualization template; performing personalized calculations on the visualization dataset, including dimensional classification and summarization, time series analysis, and ratio calculation; organizing the calculated visualization data into display groups to generate chart drawing data; and rendering and displaying the charts and data through the front-end page according to the configuration.

[0087] Understandably, the display type, chart style specifications, and data rendering engine type are determined based on a predefined visualization template; personalized calculations are performed on the input visualization dataset, including data classification and summarization by specified dimensions, trend analysis calculations by time series, and proportional relationship calculations by component; the processed visualization data is organized and arranged according to display grouping requirements to generate a standardized data format that meets the chart drawing requirements; and the charts are rendered and the data content is displayed through front-end page components according to the specific requirements configured in the visualization template.

[0088] For example, the visualization template is configured to display a pie chart, with the chart style showing the percentage of visits for each product category, and the data rendering engine is ECharts. Personalized calculations are performed on the input visualization dataset, which includes product category and visit volume fields. The dataset is categorized and summarized by product category: Education products had 18,500 visits, Academic products had 14,200 visits, and Journal products had 9,800 visits. Then, the percentage for each category is calculated: Total visits: 18,500 + 14,200 + 9,800 = 42,500; Education category percentage: 18,500 / 42,500 = 43.53%; Academic category percentage: 14,200 / 42,500 = 33.41%; Journal category percentage: 9,800 / 42,500 = 23.06%. The calculated data is organized into display groups to generate a pie chart in the following format: [{value:18500,name:'Education'},{value:14200,name:'Academic'},{value:9800,name:'Journals'}]. The front-end page uses the ECharts engine to receive this data, renders the pie chart, and displays it in the specified area. Each sector of the chart displays the product category name and its corresponding percentage value.

[0089] In some applications, historical statistical data is identified and migrated according to a preset archiving strategy. Data that meets the archiving conditions is transferred to the archiving storage area. Archived data is cleaned up in the database. Query requests for archived data are received through an independent query interface and the corresponding results are returned. This includes performing data migration tasks according to preset archiving cycles and rules; identifying historical statistical data to be archived and migrating it to the designated archiving storage area; cleaning up migrated data records in the database to release storage space; and receiving query requests for archived data through an independent query interface and returning the corresponding results.

[0090] Understandably, based on the preset archiving period and data retention rules, the system performs historical data migration tasks; identifies historical statistical data in the source database that meet the archiving conditions and migrates them completely to the designated archiving storage area; cleans up the migrated data records in the source database to release the occupied storage space; and receives query requests for archived data through an independently deployed query interface, locates the corresponding data in the archiving storage area, and returns the query results.

[0091] For example, the default archiving strategy is to automatically archive historical statistics from three months ago, performing a migration task monthly. Identify the data to be archived, such as migrating user access statistics records from January 1, 2024 to March 31, 2024 (90 days, approximately 1 million records) from the `daily_visits` table in the MySQL production database `stat_db` to the AWS S3 archive storage area, with the storage path `s3: / / archive-bucket / statistics / 2024 / Q1 / `. After the migration is complete, delete these 1 million migrated records from the `daily_visits` table in `stat_db`, freeing up approximately 2GB of storage space. When a user needs to query historical data for the first quarter of 2024, the query parameters (time range: 2024-01-01 to 2024-03-31, statistical dimension: daily visits) are passed through a separate query interface. The system locates and reads the corresponding data from the S storage and returns formatted query results, such as displaying 12,500 visits on January 1, 2024, 11,800 visits on January 2, 2024, and 15,600 visits up to March 31, 2024.

[0092] Some of these applications also include using a multi-threaded parallel strategy to execute multiple statistical tasks during batch processing; caching statistical results from multiple accesses during data querying; performing breakpoint resume and exception retry operations during archiving; and performing data drill-down, linked analysis, and conditional filtering operations in visualization.

[0093] Understandably, during batch processing, a multi-threaded parallel strategy is employed to execute multiple statistical tasks simultaneously to improve processing efficiency; during data querying, frequently accessed statistical results are cached to reduce redundant calculations and database access pressure; during archiving, a breakpoint resume mechanism is implemented to ensure the integrity of large-scale data migration, and automatic retry operations are implemented to ensure the reliability of archiving tasks; in the visualization stage, data drill-down operations are performed to achieve layer-by-layer drill-down analysis from summary data to detailed data, linked analysis operations are performed to achieve interactive responses between multiple related charts, and conditional filtering operations are performed to support users to filter and focus on specific data ranges as needed.

[0094] For example, a multi-threaded parallel strategy is used in batch processing, starting five threads simultaneously to execute five different statistical tasks. Each thread handles one statistical task, reducing the original 50-minute single-threaded serial processing time to 12 minutes. During data querying, a statistical result (such as the query result for "daily visits in the last 30 days") is cached with a 300-second validity period. During this period, the same query request directly returns the cached result, reducing the average query response time from 800 milliseconds to 150 milliseconds. A breakpoint resumption mechanism is implemented during archiving. When migrating 50GB of historical data, if a network interruption causes the transfer to stop at 35GB, it resumes from 35GB upon recovery, continuing the transfer of the remaining 15GB of data. An automatic retry mechanism is also set up for transmission anomalies, retrying a maximum of three times with a 5-minute interval between each attempt. In the visualization, users can click on the "Education Products" sector in the pie chart to perform data drill-down, drilling down from the product category to the specific course subject's access volume distribution; perform linked analysis operations, when users select "June 2024" data in the bar chart, the associated line chart automatically displays the detailed trend for that month; perform conditional filtering operations, users can set the time range from January 1, 2024 to June 30, 2024 and select the "Mobile Users" condition to filter and display only the data that meets the conditions.

[0095] The following is combined with Figures 2-7 Another embodiment of the statistical report generation method of the present invention is described below:

[0096] like Figure 2 , Figure 3 , Figure 4 , Figure 5 , Figure 6 and Figure 7As shown, this embodiment is based on a Lambda-like architecture and uses statistical template management, mainly including definitions of statistical statements, statistical results, dynamic parameter placeholders, and related source dataset information. Statistical template management achieves template reuse and efficient management through flexible binding of configuration information. The dynamic parameter placeholder function can dynamically replace key variables in statistical statements according to different tenants and business needs, enabling a single template to be reused in multiple scenarios, significantly improving configuration flexibility and development efficiency. Structured descriptions of statistical dimensions, indicators, formats, and other elements provide standardized output basis for subsequent data display and analysis. Related dataset information ensures that statistical statements accurately point to the required data source, while supporting dynamic switching between multiple dataset versions. Statistical tasks mainly include definitions of name, description, priority, related template information, type, and statistical engine. Statistical tasks achieve dynamic binding between tasks and templates through related template information, supporting parallel execution of multiple tasks and dynamic resource scheduling, improving the flexibility and execution efficiency of statistical tasks.

[0097] In the batch processing process, statistical tasks are triggered by a timed executor, which calls the corresponding batch processing engine to perform periodic batch calculations of statistical data. The calculation results are then persistently stored in a designated batch processing result database for easy subsequent data querying and display. Multiple thread task instances are converted and initialized based on the various statistical tasks. These thread task instances are compiled into an execution batch and submitted to a multi-threaded task execution tool for unified processing. The batch processing engine uses a multi-threaded parallel strategy to batch process statistical tasks, improving data processing efficiency and system throughput.

[0098] The persistence process specifically involves determining the type of total and grouped statistical results, differentiating between different data parsing and storage strategies, and thus marking the statistical dimensions and populating the statistical content of the stored records. For total results, a single record is created, with the statistical dimension set to "count," and the result is the total value. For grouped statistical results, multiple records are generated based on the grouping dimensions, with each record's statistical dimension field populated with the corresponding grouping type value, and the result is the specific grouped statistical value. Simultaneously, the uniqueness of the grouping dimensions is checked. If multiple grouping dimensions exist, they are combined to generate a composite dimension identifier, which is then set as the statistical dimension. The type and result of the overall grouping dimension are stored in the dimension content field, ensuring the uniqueness and traceability of the statistical dimension for each record. This supports rapid location and efficient retrieval of result data, significantly shortening the response time for data display and report generation, and improving the overall stability and scalability of the statistical system. A caching optimization strategy is introduced, caching frequently queried statistical results in memory, effectively reducing the pressure of repeated database accesses and improving data response speed. Automatic refresh and invalidation mechanisms for cached data are supported to ensure data real-time performance and accuracy.

[0099] The visualization view primarily includes definitions for name, identifier, page modules, display groups, display type, associated statistical task information, and data rendering and calculation engine. By binding corresponding statistical task identifiers, the visualization view dynamically associates with statistical results, supporting various display types such as bar charts, line charts, pie charts, and tables to meet diverse data presentation needs. Simultaneously, the view configuration information defines the execution logic of the data rendering and calculation engine, ensuring that charts are dynamically generated based on statistical results and displayed in the front-end page modules according to display groups. It also supports personalized view configuration, allowing users with different roles to flexibly call and customize statistical views according to business needs. Interactive operation functions are provided for the visualization view, including data drill-down, linked analysis, filtering, and sorting, enhancing users' data exploration capabilities and operational flexibility. Through interactive operations, users can delve deeper into the business patterns behind the data, achieving layer-by-layer analysis from macro-statistics to micro-details. Based on the quantitative analysis results of the visualization view, it can assist business decision-makers in quickly identifying key issues and potential trends, providing strong data support for strategy formulation and execution. By using a visualized early warning mechanism, abnormal indicators are promptly fed back to relevant personnel, helping the team to respond and adjust quickly in the early stages of problems, thereby improving the agility and accuracy of business operations.

[0100] Data queries, when specifying a view, differentiate query modes based on the user-selected time range. These include real-time statistical queries of the source dataset, batch processing result statistical queries, and queries combining both modes. Real-time statistical queries of the source dataset are suitable for incremental data scenarios, dynamically calculating statistical indicators and displaying the latest data dynamics. Batch processing result statistical queries rely on pre-calculated and stored statistical results, suitable for scenarios with large data volumes, long time spans, and high computational complexity, improving query efficiency by reading pre-generated statistical results. Queries combining both modes automatically switch between real-time calculations and batch processing results, or combine both query modes, depending on the time range. During the query process, intelligent scheduling algorithms optimize data requests, rationally allocate computing resources, reduce response latency, and ensure users can quickly obtain the required statistical information. Multi-dimensional data filtering and sorting functions are provided, allowing users to flexibly adjust query conditions according to business needs, further improving the accuracy and practicality of data retrieval. For frequently accessed data views, a caching mechanism is also supported to temporarily store frequently used query results to improve subsequent access efficiency. Data query results can be presented intuitively through various visualization formats, helping users quickly understand data characteristics and trends.

[0101] Data computation requires personalized calculations for different display types. For example, bar charts require dimensional classification and statistical summarization, while line charts require time series trend analysis. Data results returned for different query patterns need to be standardized in format, merged, calculated, and summarized according to statistical dimensions to form the final dataset for visualization.

[0102] The archiving engine categorizes and identifies historical statistical data, migrating cold data to another storage area according to preset strategies. Simultaneously, archived data in the original database is cleaned up, freeing up storage space and optimizing query efficiency. The archiving strategy supports flexible configuration, allowing for setting archiving cycles and retention periods for different data types based on business characteristics. This ensures long-term traceability of critical data, preventing system burden from data accumulation. The archiving engine features breakpoint resumption and exception retry mechanisms during execution, ensuring the integrity and stability of data migration. Archived data retains a separate query interface for long-term traceability. This avoids the degradation of query efficiency caused by mixing archived data with other data after it directly enters the archiving area. Users can access archived data through a separate interface, automatically locating the archive storage location in the background and returning the results to the front-end interface, maintaining a consistent query experience.

[0103] For the purpose of simplicity, the method steps disclosed in the above embodiments are described as a series of actions. However, those skilled in the art should understand that the embodiments of the present invention are not limited to the described order of actions, because according to the embodiments of the present invention, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily essential to the embodiments of the present invention.

[0104] like Figure 8 As shown, the present invention also provides a statistical report generation system, comprising:

[0105] The statistics request configuration module 201 is configured to respond to a statistics configuration request, formulate a statistics statement template, and create a statistics task based on the statistics statement template. The statistics statement template includes statistics logic, result definition, dynamic parameter placeholders, and associated data source information.

[0106] The statistical result generation module 202 is configured to trigger a batch processing procedure according to a timer configuration, execute the statistical task, perform batch calculations on the source dataset associated with the statistical statement template, generate statistical results, and persistently store the statistical results in a specified database.

[0107] The visualization data generation module 203 is configured to respond to a visualization query request, select a query mode based on the identifier corresponding to the statistical task and the time range selected by the user, perform real-time statistical queries on the source dataset and / or query the batch processing results, and format and merge the query results to generate a visualization dataset.

[0108] The chart data display module 204 is configured to render and calculate the visualization dataset according to the visualization template configuration, generate chart data, and display it on the front-end page;

[0109] The archiving processing module 205 is configured to identify and migrate historical statistical data according to a preset archiving strategy, transfer data that meets the archiving conditions to the archiving storage area, clean up archived data in the database, and receive query requests for archived data through an independent query interface and return the corresponding results.

[0110] It is worth noting that although only some basic functional modules are disclosed in the embodiments of this invention, it does not mean that the composition of this system is limited to the above-mentioned basic functional modules. On the contrary, what this embodiment intends to express is that, based on the above-mentioned basic functional modules, those skilled in the art can arbitrarily add one or more functional modules in combination with existing technology to form an infinite number of embodiments or technical solutions. That is to say, this system is open rather than closed. The fact that this embodiment only discloses a few basic functional modules should not be considered as the scope of protection of the claims of this invention being limited to the disclosed basic functional modules. At the same time, for the convenience of description, the above device is described separately according to its functions as various units and modules. Of course, in implementing this invention, the functions of each unit and module can be implemented in one or more software and / or hardware.

[0111] like Figure 9 As shown, the present invention also provides an electronic device, including: a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; the memory stores a computer program, and when the computer program is executed by the processor, the processor performs the steps of a statistical report generation method.

[0112] Figure 9 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. For example... Figure 9 The structure shown in this embodiment of the invention includes an electronic device comprising one or more processors 710 and a memory 720; the processors 710 in this electronic device may be one or more. Figure 9Taking a processor 710 as an example; the memory 720 is used to store one or more programs; the one or more programs are executed by the one or more processors 710, so that the one or more processors 710 implement a statistical report generation method as described in any one of the embodiments of the present invention.

[0113] The electronic device may also include an input device 730 and an output device 740.

[0114] The processor 710, memory 720, input device 730, and output device 740 in this electronic device can be connected via a bus or other means. Figure 9 Taking the example of a connection between China and Israel via a bus.

[0115] The memory 720 in this electronic device serves as a computer-readable storage medium, capable of storing one or more programs. These programs can be software programs, computer-executable programs, or modules, such as the program instructions / modules corresponding to the statistical report generation method provided in this embodiment of the invention. The processor 710 executes various functional applications and data processing of the electronic device by running the software programs, instructions, and modules stored in the memory 720, thereby implementing the statistical report generation method described in the above embodiment.

[0116] The memory 720 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created based on the use of the electronic device. Furthermore, the memory 720 may include high-speed random access memory and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some instances, the memory 720 may further include memory remotely located relative to the processor 710, which can be connected to the device via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0117] Input device 730 can be used to receive input digital or character information, and to generate key signal inputs related to user settings and function control of the electronic device. Output device 740 may include display devices such as a display screen.

[0118] The present invention also provides a computer-readable storage medium storing a computer program executable by an electronic device, which, when run on the electronic device, causes the electronic device to perform the steps of a statistical report generation method.

[0119] Specifically, the computer storage medium in this embodiment of the invention can be any combination of one or more computer-readable media. The computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. For example, a computer-readable storage medium can be—but is not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of computer-readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this embodiment, the computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0120] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for generating statistical reports, characterized in that, include: In response to a statistical configuration request, a statistical statement template is defined, and a statistical task is created based on the statistical statement template. The statistical statement template includes statistical logic, result definition, dynamic parameter placeholders, and associated data source information. The batch processing procedure is triggered according to the timed configuration, the statistical task is executed, the source dataset associated with the statistical statement template is calculated in batches, the statistical results are generated, and the statistical results are persistently stored in the specified database. In response to a visualization query request, based on the identifier corresponding to the statistical task and the time range selected by the user, a query mode is selected, and real-time statistical queries are performed on the source dataset and / or the batch processing results are queried. The query results are then formatted and merged to generate a visualization dataset. Based on the visualization template configuration, the visualization dataset is rendered and calculated to generate chart data, which is then displayed on the front-end page. Based on the preset archiving strategy, historical statistical data is identified and migrated, data that meets the archiving conditions is transferred to the archiving storage area, archived data is cleaned up in the database, and query requests for archived data are received through an independent query interface and the corresponding results are returned.

2. The statistical report generation method according to claim 1, characterized in that, In response to a statistics configuration request, a statistics statement template is defined, and a statistics task is created based on the statistics statement template, further including: Define a statistical statement template, including statistical logic, statistical dimensions, indicator types, dynamic parameter placeholders, and associated data source identifiers; Create a statistics task, and configure its name, description, execution priority, associated statistics statement template identifier, and statistics engine type; Dynamically replace parameter placeholders in the statistical statement template according to business needs to generate executable statistical statements; Associate the statistics task with the timer executor and configure the execution cycle and trigger conditions.

3. The statistical report generation method according to claim 2, characterized in that, The batch processing procedure is triggered according to the timed configuration, the statistical task is executed, the source dataset associated with the statistical statement template is batch-calculated, statistical results are generated, and the statistical results are persistently stored in a specified database, further including: Statistical tasks are triggered by a timer to perform statistical calculations. Perform batch queries and calculations on the source dataset to generate statistical results. The statistical results data are classified to distinguish between total results data and grouped statistical results data, and different parsing and storage strategies are adopted accordingly. The statistical results are persistently stored in the batch processing results database, and the statistical dimensions and values ​​are recorded.

4. The statistical report generation method according to claim 1, characterized in that, In response to a visualization query request, based on the identifier corresponding to the statistical task and the time range selected by the user, a query mode is selected, real-time statistical queries are performed on the source dataset and / or queries are performed on the batch processing results, and the query results are formatted and merged to generate a visualization dataset, further including: Receive query requests and extract statistical task identifiers and time range parameters; Determine the query mode based on the time range parameter, and select real-time query, batch result query, or mixed query; The data returned by the query is standardized in format and merged and classified according to statistical dimensions to generate a visualization dataset that meets the requirements of the visualization template.

5. The statistical report generation method according to claim 1, characterized in that, Based on the visualization template configuration, the visualization dataset is rendered and calculated to generate chart data, which is then displayed on the front-end page, further including: Define the display type, chart style, and data rendering engine type based on the visualization template; Perform personalized calculations on visualized datasets, including dimensional classification and summarization, time series analysis, and proportion calculation; The calculated visualization data is organized into display groups to generate chart plotting data; Charts and data are rendered and displayed via the front-end page according to the configuration.

6. The statistical report generation method according to claim 1, characterized in that, Based on a preset archiving strategy, historical statistical data is identified and migrated, data that meets the archiving criteria is transferred to the archiving storage area, archived data is cleaned up in the database, and query requests for archived data are received and returned through an independent query interface. Further steps include: Execute data migration tasks according to the preset archiving cycle and rules; Identify historical statistical data to be archived and migrate it to the designated archive storage area; Clean up migrated data records in the database to free up storage space; The system receives query requests for archived data through an independent query interface and returns the corresponding results.

7. The statistical report generation method according to claim 1, characterized in that, The method further includes: A multi-threaded parallel strategy is used to execute multiple statistical tasks during batch processing; Cache the statistical results of multiple accesses during the data query process; Perform breakpoint resume and exception retry operations during the archiving process; Perform data drill-down, linked analysis, and conditional filtering operations within the visualization.

8. A statistical report generation system, characterized in that, include: The statistics request configuration module is configured to respond to a statistics configuration request, specify a statistics statement template, and create a statistics task based on the statistics statement template. The statistics statement template includes statistics logic, result definition, dynamic parameter placeholders, and associated data source information. The statistical result generation module is configured to trigger a batch processing procedure according to a timer configuration, execute the statistical task, perform batch calculations on the source dataset associated with the statistical statement template, generate statistical results, and persistently store the statistical results in a specified database. The visualization data generation module is configured to respond to visualization query requests, select a query mode based on the identifier corresponding to the statistical task and the time range selected by the user, perform real-time statistical queries on the source dataset and / or query the batch processing results, and format and merge the query results to generate a visualization dataset. The chart data display module is configured to render and calculate the visualization dataset according to the visualization template configuration, generate chart data, and display it on the front-end page; The archiving module is configured to identify and migrate historical statistical data according to a preset archiving strategy, transfer data that meets the archiving conditions to the archiving storage area, clean up archived data in the database, and receive query requests for archived data through an independent query interface and return the corresponding results.

9. An electronic device, characterized in that, include: The system includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other via the communication bus; the memory stores a computer program, which, when executed by the processor, causes the processor to perform the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, It stores a computer program executable by an electronic device, which, when run on the electronic device, causes the electronic device to perform the steps of the method according to any one of claims 1 to 7.

Images