Configurable real-time super-large data report processing method, system and equipment

Abstract

The embodiment of the invention provides a configurable real-time super-large data report processing method, system and equipment, and the method comprises the steps: receiving report template customization operation of a user, and generating a customized target report template; analyzing the target report template to convert the target report template into a data structure which can be identified by a system and storing the data structure; and reading target data associated with the data structure from a database in a streaming mode, processing the target data, and writing the target data and a processed result into the target report template to obtain a target report. By utilizing the method, the report template style can be realized by a user through configuration, the template style can be modified and newly added at any time, development resources and repeated development and modification are not needed, and massive report data downloading is supported without influencing business transaction processing; the business requirements of system users can be greatly met, and the report flexibility and service capability of a finance and accounting system are improved.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a configurable real-time ultra-large data report processing method, system and device. Background Technology

[0002] With the deep application of computer technology in the accounting field, a large amount of accounting ledger data is generated and stored by the system, and this data contains enormous value. As a crucial data asset for enterprises, it is extremely important that accounting data can be easily and flexibly downloaded, analyzed, and used by users. Because accounting systems process and store massive amounts of ledger data, users and administrators often need to download, analyze, and archive certain types of massive amounts of data. Therefore, supporting flexible configuration of report styles and real-time download of massive amounts of data is a vital part of system design. Designing a configurable solution for processing ultra-large data reports in accounting systems is one of the urgent problems that needs to be solved.

[0003] Currently, accounting systems offer solutions for using pre-defined reports to support user report downloads. However, this requires pre-defining report templates and coding the report data logic, making modifications inconvenient. Furthermore, to ensure system availability, the amount of data in a single report needs to be limited, which is extremely inconvenient when users require flexible reports and have massive amounts of data. Another solution involves using the accounting system to generate reports in batches on a scheduled basis, but this faces problems such as the inability to process data in real time and the inability to meet users' urgent reporting needs. Summary of the Invention

[0004] This invention provides a configurable real-time ultra-large data report processing method, system, and device, which achieves processing without the need for investment in development resources and repeated development modifications, and supports the download of massive report data without affecting business transaction processing. It can greatly meet the business needs of system users and improve the flexibility and service capabilities of accounting system reports.

[0005] Firstly, this embodiment provides a configurable real-time ultra-large data report processing method, which includes:

[0006] Receive user requests to customize report templates and generate the customized target report template.

[0007] The target report template is parsed to convert it into a system-recognizable data structure and then stored.

[0008] The target data associated with the data structure is read from the database, the target data is processed, and the target data and the processed results are written into the target report template to obtain the target report.

[0009] Secondly, this embodiment provides a configurable real-time ultra-large data reporting system, which includes:

[0010] The template customization module is used to receive users' report template customization requests and generate the customized target report template.

[0011] The template parsing module is used to parse the target report template, convert the target report template into a data structure that the system can recognize and store it;

[0012] The report generation module is used to read target data associated with the data structure from the database, process the target data, and write the target data and the processed results into the target report template to obtain the target report.

[0013] Thirdly, this embodiment provides an electronic device, including:

[0014] At least one processor; and

[0015] A memory communicatively connected to the at least one processor; wherein,

[0016] The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the configurable real-time ultra-large data report processing method according to any embodiment of the present invention.

[0017] This invention provides a configurable real-time large-scale data report processing method, system, and device. The method includes: first, receiving a user's report template customization operation and generating a customized target report template; then, parsing the target report template to convert it into a system-recognizable data structure and storing it; finally, reading target data associated with the data structure from the database using a streaming method, processing the target data, and writing the target data and the processed results into the target report template to obtain the target report. This technical solution essentially provides users with a configurable method for processing large-scale data reports in an accounting system. Users can customize report templates, then analyze, verify, and statistically analyze the database tables and data conditions related to the target data based on the user-customized report template, store the table and field mapping relationships as a data structure, and finally read data from the database, perform logical processing, and write data into the template, thus generating the target report. By providing users with simple and convenient template customization, the system development workload is low, and it does not affect the original business functions of the system. The above functions can be completed with low investment costs, requiring no investment in hardware or software resources. It is user-friendly and has high practicality and widespread applicability. Furthermore, employing streaming processing technology, it can process massive data streams in real time and generate reports that meet user needs, ensuring strong real-time performance. Report template styles can be configured by users, and can be modified and added at any time without investing development resources or repeated development modifications. It also supports the download of massive amounts of report data without affecting business transaction processing, greatly meeting the business needs of system users and improving the flexibility and service capabilities of the accounting system's reports.

[0018] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This is a flowchart illustrating a configurable real-time ultra-large data report processing method provided in Embodiment 1 of the present invention.

[0021] Figure 2This is a flowchart illustrating another configurable real-time ultra-large data report processing method provided in Embodiment 2 of the present invention.

[0022] Figure 3 This is an example diagram illustrating the parsing of a report template provided in Embodiment 2 of the present invention;

[0023] Figure 4 This is a flowchart illustrating a data processing procedure provided in Embodiment 2 of the present invention;

[0024] Figure 5 This is a schematic diagram of the structure of a configurable real-time ultra-large data report processing system provided in Embodiment 3 of the present invention;

[0025] Figure 6 This is a structural example diagram of a configurable real-time ultra-large data report processing system provided in Embodiment 3 of the present invention;

[0026] Figure 7 This is a schematic diagram of the structure of an electronic device provided in Embodiment 4 of the present invention. Detailed Implementation

[0027] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0028] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0029] Existing technologies, specifically for accounting systems, cannot meet the requirements of business scenarios involving extremely large reports, lack a complete management solution, and require significant investment in manpower, hardware, and other resources for implementation. Traditional accounting system data report downloads, limited by system resources, often require data pagination or scheduled batch generation, leading to various problems such as cumbersome operations increasing user workload or compromising data real-time performance. Insufficient flexibility: using user interface controls to edit report templates results in a WYSIWYG (What the User Sees Is Not What They Get) experience; Insufficient management capabilities: lacking user and template management functions, unable to manage all report-related resources; Performance bottlenecks: performance bottlenecks exist when processing extremely large amounts of data, causing slow system operation, affecting normal system transactions, and even triggering system anomalies and production accidents.

[0030] To more clearly describe the methods provided in the embodiments of this invention, some terms are introduced below. Database: A repository on a computer storage device that organizes, stores, and manages data according to a data structure. Persistence Layer Framework: A technology for operating databases, making communication between developers and the database more efficient, effectively combining business logic and database processing, responsible for storing and managing data information, and responding to database queries. Streaming Read: An efficient big data processing method that reads data block by block or line by line as needed, instead of loading the entire dataset into memory at once. Accounting System: A computer system used to process corporate financial data and generate financial statements. Report Template: A user-defined report format, including report structure, style, data source, and calculation formulas. Massive Data: Refers to datasets with a huge volume of data that cannot be queried and processed in a short time using traditional methods.

[0031] Example 1

[0032] Figure 1 This is a flowchart illustrating a configurable real-time ultra-large data report processing method provided in Embodiment 1 of the present invention. This method is applicable to the processing of ultra-large data reports and can be executed by a configurable real-time ultra-large data report processing system. This configurable real-time ultra-large data report processing system can be implemented in hardware and / or software and is generally integrated into an electronic device.

[0033] like Figure 1 As shown, the configurable real-time ultra-large data report processing method provided in this embodiment may specifically include the following steps:

[0034] S101: Receive the user's report template customization operation and generate the customized target report template.

[0035] It should be noted that the method provided in this embodiment can be integrated into electronic devices as a report processing tool, enabling real-time customization of business data report templates for accounting systems, supporting massive data processing and downloading of user-customized reports. This step is used to complete the report template customization.

[0036] In a specific application scenario, when a user wants to generate a report with a certain style and containing certain data according to their needs, they can open the report processing tool provided by the electronic device. This tool offers a report template customization function. The user can perform a report template customization operation on the electronic device to generate a customized report template, denoted as the target report template. The report template customization operation can be understood as the operation of customizing the report style.

[0037] This embodiment provides a configurable real-time large-scale data reporting method, offering functions such as report template user information management, historical report template information management, basic report template information management, and report template editing rule guidance. The report template user information management function primarily involves identifying which users have the authority to create reports and how to associate user-edited report templates with specific users. For example, a first-level user with the highest administrator privileges can create and distribute report templates to second-level users.

[0038] The historical report template information management function mainly stores user-defined report templates. Users can later call upon these historical report templates and customize reports based on actual needs. The basic report template information management function mainly creates basic report templates based on some common functions included in reports. Users can later add special reports to the basic report templates according to actual needs.

[0039] For example, a report processing tool provides users with an Excel template. Users can download the template, fill it out, and then upload the completed Excel file to the system. The system recognizes it as a valid template and, when downloading data using that template, it retrieves data from the system based on the user-uploaded headers—that is, the template's style. The final downloaded data is the portion the user edited, along with the corresponding data.

[0040] The report processing tool manages all user information related to template definitions, historical report templates, and basic template styles within the system. For example, if a user has defined report templates A, B, and C, the user can manage these historically defined templates. Furthermore, these three report templates can be shared with other users.

[0041] Specifically, users can flexibly customize report styles according to their actual needs. Customization includes elements such as report title, header content, row and column headers, cell styles, worksheet titles, and maximum data limit per worksheet. Users can create blank report templates, receive customization requests for these templates, and generate the customized target report template. Furthermore, the report processing tool provides a historical report style storage and query function, allowing users to copy new template styles from historical report style information for easy secondary customization or to directly use existing styles to generate reports. It also provides a basic report style storage and query function, allowing users to copy new template styles from basic report style information for easy secondary customization or to directly use existing styles to generate reports.

[0042] For accounting systems, there is often a need to handle ad-hoc and urgent accounting audits. Due to the suddenness of audit work and the uncertainty of audit content, the format of the report data to be extracted often cannot be determined in advance. In the past, in such cases, business managers had to send the audit content to system maintenance personnel, and then the data extraction was completed through a backend database script query. This resulted in the overall reporting cycle not being able to respond in a timely manner, and the data feedback format might not meet the audit requirements. Based on this scenario, the report template style configuration table information is introduced as follows. The application number and template number are uniquely generated by the system when the template is customized, and they correspond uniquely to the drawn template. The template verification result field is updated according to the parsing result. The report template style configuration information table can be used to represent the statistics of configuration-related information of the user-customized report templates. Each report template contains corresponding customized content. The customized report templates can be stored, and the corresponding templates can be called according to the needs when used later. Table 1 is the report template style configuration table provided in Embodiment 1 of the present invention. As shown in Table 1, the report template style configuration table can record the application number, template number, template name, maintenance employee number, maintenance time, and template verification result when the template is customized. Each type of configuration information corresponds to a data type. The template validation result refers to whether the customized template passes the validation.

[0043] Table 1

[0044]

[0045] For example, Table 2 is an example of a custom report template provided in this embodiment. As shown in Table 2, the column headers of this custom report template include field 1, specifically TableA.filedX.UN, field 2, specifically TableA.filedY, field 3, specifically TableB.filedU, field 4, specifically TableB.filedV, and the operation relationship includes field 5, specifically Calc:TableA.filedY.add(TableB.filedV).

[0046] Table 2

[0047]

[0048] S102. The target report template is parsed to convert it into a data structure that the system can recognize and store it.

[0049] This step parses the user-created target report template, converts it into a system-recognizable format, and stores it. Essentially, it maintains a mapping relationship between the template and tables and fields in the database. Knowing this mapping relationship, during data retrieval, the system uses this mapping to parse each row and field individually, determining which table the required data resides in, whether that table is in the maintained database, and whether the field exists. Specifically, it identifies various user-defined elements in the target report template, such as titles, column names, data sources, and calculation formulas, and converts them into corresponding data structures and stores them.

[0050] For example, a title parser is used to parse and validate the column title information in the target report template, obtaining successfully validated title data, and storing the title data in a title structure table format. Similarly, a relation parser is used to parse and validate the relational operations in the title fields of the target report template, obtaining successfully validated relational data, and storing the relational data in a relational structure table format.

[0051] S103. Read the target data associated with the data structure from the database using a streaming method, process the target data, and write the target data and the processed results into the target report template to obtain the target report.

[0052] After parsing the target report template and obtaining the parsed information, a streaming approach is used to complete operations such as reading data from the database, logical processing, and writing data into the template. Once the data structure is obtained, it records which table and field in the database the data to be read belongs to. Based on this, the data associated with the data structure can be read from the database; this data is the desired data, denoted as the target data.

[0053] Following the above description, after reading the target data, it needs to be processed, such as displaying fields and processing report fields based on the parsing results of field operations. It should be noted that this embodiment uses a streaming approach to read the target data, which is equivalent to summarizing the target data in stages. Streaming is an efficient big data processing method, reading data block by block or line by line as needed, rather than loading the entire dataset into memory at once. After all the staged data summarization is completed, it is written to the user-selected report template using a streaming method. According to the field mapping relationship, the corresponding data and calculation results are written to the corresponding positions in the report template, thereby realizing the automatic generation of the report. The report after all data has been fully processed is called the target report. Based on this, the user can download the completed target report.

[0054] The above technical solution provides users with a configurable method for processing massive amounts of data in accounting systems. Users can customize report templates, then analyze, verify, and statistically analyze the target data's database tables and data conditions based on these templates. The table and field mapping relationships are stored in a data structure format. Finally, data is read from the database, logically processed, and written to the template to generate the target report. By providing users with simple and convenient template customization, the system development workload is low, and it does not affect the original business functions of the system. The above functions are completed at a low cost, requiring no investment in hardware or software resources. It is user-friendly and has high practicality and widespread applicability. Furthermore, the use of streaming processing technology enables real-time processing of massive data streams and the generation of reports that meet user needs, ensuring strong real-time performance. Report template styles can be configured by users and can be modified and added at any time without requiring development resources or repeated development modifications. It also supports downloading massive amounts of report data without affecting business transaction processing, greatly meeting the business needs of system users and improving the flexibility and service capabilities of accounting system reports.

[0055] Example 2

[0056] Figure 2This is a flowchart illustrating another configurable real-time large-scale data report processing method provided in Embodiment 2 of the present invention. This embodiment is a further optimization of the above embodiment. In this embodiment, the limitations of "receiving the user's report template customization operation and generating the customized target report template" are further optimized, as are the limitations of "parseting the target report template to convert the target report template into a data structure that the system can recognize and store" and "reading the target data associated with the data structure from the database in a streaming manner, processing the target data, and writing the target data and the processed result into the target report template to obtain the target report" are further optimized.

[0057] like Figure 2 As shown in the figure, this embodiment 2 provides a configurable real-time ultra-large data report processing method, which specifically includes the following steps:

[0058] S201. Create a blank report template, receive a user's report template customization operation for the blank report template, and generate a customized target report template; or, query historical report template style information / basic report template information management to obtain any historical report template style / basic report template style; use the historical report template style / basic report template style as the target report template; or, receive a user's report template customization operation for the historical report template style / basic report template style, and generate a customized target report template.

[0059] In this embodiment, the report processing tool integrated with the method provides users with a report template customization function, allowing users to customize report templates according to business needs. One approach is for users to download a blank report template and then customize it. Users can flexibly customize the report style according to actual needs, including elements such as report title, header content, row and column titles, cell styles, worksheet titles, and maximum data limit per worksheet. Customizing the blank report template generates the final target report template.

[0060] In addition, this embodiment also provides a historical report style storage and query function, allowing users to copy new template styles from historical report style information for easy secondary customization or to directly use existing styles to initiate report generation. The historical report template style information stores multiple report templates customized within a historical period. Users can query the historical report template style information to obtain any historical report template style; use the historical report template style as the target report template; or receive user customization operations for historical report template styles to generate the customized target report template.

[0061] Additionally, it provides a basic report style storage and query function, allowing users to copy new template styles from the basic report style information for easy customization or to directly use existing styles to generate reports. The basic report template information management system stores multiple report templates customized within a basic time period. Users can query the basic report template information management system to retrieve any basic report template style; use a basic report template style as the target report template; or receive user customization operations on basic report template styles to generate the customized target report template.

[0062] This step is equivalent to completing the customization of the report template.

[0063] S202. The column header information in the target report template is parsed and verified by the header parser to obtain the successfully verified header data, and the header data is stored in the header structure table format.

[0064] In this embodiment, after the user customizes the report template, it is necessary to analyze, verify, and statistically analyze the database tables and data conditions involved in the data to be processed based on the user-customized template. That is, to verify and statistically analyze the data tables, fields, and operation relationships.

[0065] This step involves the title parser parsing the column header information of the user-defined target report template. The title parser sequentially reads the header information from the target report template, collecting, parsing, and storing the header data using a header structure table. Based on a specific header formula in the target report template, the title parser performs processing, including verifying the existence of tables and fields in the database. Parsing errors are returned to the user for re-editing. Successfully parsed headers are registered according to the column header structure table format. This registration result supports subsequent data querying, processing, and formatted storage, reducing the parsing cost of reusing templates. This step essentially determines which tables and fields in the database the target report template needs to read, establishing the mapping relationship between column headers and tables / fields in the database.

[0066] As a specific implementation method, the step of parsing and validating the column header information in the target report template through a header parser to obtain successfully validated header data can be optimized, including:

[0067] a1) Using a title parser, the column title information in the target report template is parsed based on a set parsing formula to obtain the parsed title data.

[0068] Specifically, the title parser reads the title information in the target report template sequentially, uses a title structure table to collect, parse and store the title data, and records the parsed data as title data. For example, when reading the template, if the parsing formula for a title on a certain page of the template is tableA.FieldX, that is, the content of this column represents the X field of table A, the title parser reads and processes according to this formula to obtain the parsed title data.

[0069] b1) Verify the mapping relationship between the parsed title data and the tables and fields in the database.

[0070] This step is used to verify the mapping relationship between the parsed title data and the tables and fields in the database, and to complete related verification processing actions such as confirming the existence of tables and fields in the database.

[0071] c1) If the verification fails, a verification exception is generated, and the receiving user's re-editing operation is returned.

[0072] In this embodiment, if a certain title-type data does not have a corresponding table or field in the database, it indicates that the title-type data validation fails. The validation exception is then returned to the user so that the user can re-edit the data.

[0073] d1) If the verification passes, the parsed title data will be used as the title data that has been successfully verified.

[0074] Specifically, title data that has been completely parsed and successfully verified is considered successfully verified title data. Table 3 is a column header structure table provided in Embodiment 2 of the present invention. In Table 3, the column header structure includes template number, worksheet number, title name, parsing formula, database table mapping result, data table field mapping result, and related query flag, etc. The data corresponding to each type will not be described in detail here. Among them, the related query flag is used to record the verification result.

[0075] Table 3

[0076]

[0077] The above technical solution specifies the steps of parsing and verifying the column header information in the target report template through a header parser, thereby determining the mapping relationship between column headers and tables and fields in the database, and providing a basis for subsequent data reading.

[0078] S203. The operation relationship in the title field of the target report template is parsed and verified by the operation relationship parser to obtain the successfully verified operation relationship data, and the operation relationship data is stored in the operation relationship structure table format.

[0079] This step is used to parse and validate the operational relationships in the title fields. Specifically, the operational relationships in the title fields can be understood as the existence of operational relationships between the title fields. The operational relationship parser for title fields parses the result fields of operational relationships in the report template and stores the parsing results in the title operational relationship table. The operational relationship parser sequentially reads the operational relationship data in the title of the target report template, performing validation of the fields participating in the operational relationships and operator validation. This step is equivalent to determining which table and which field in the database has a mapping relationship between the operational relationships in the title fields of the target report template, that is, which table and which field in the database needs to perform the operational relationship operation.

[0080] As a specific implementation method, the step of parsing and validating the operation relationships of the title fields in the target report template through the operation relationship parser to obtain the successfully validated operation relationship data can be optimized, including:

[0081] a2) The operation relationship in the title field of the target report template is parsed by the operation relationship parser to obtain the parsed operation relationship data.

[0082] Specifically, the operation relationship parser for the title fields parses the result fields of operation relationships in the report template and stores the parsing results in the title operation relationship table. The operation relationship parser reads the operation relationship data in the template title in turn. For example, when reading the template, if the operation formula of a title in a certain sheet of the template is Calc:TableA.filedY.add(TableB.filedV), then the content of this column represents the sum of the Y field of table A and the V field of table B.

[0083] b2) Verify the mapping relationship between the parsed operational relation data and the tables and fields in the database.

[0084] This step is used to verify the mapping relationship between the parsed operational relational data and the tables and fields in the database, and to complete related verification processing actions such as confirming the existence of tables and fields in the database.

[0085] c2) If the verification fails, a verification exception is generated, and the receiving user's re-editing operation is returned.

[0086] In this embodiment, if the operation relationship data does not have a corresponding table or field in the database, it indicates that the operation relationship data fails the validation. The validation exception item is then returned to the user so that the user can re-edit the data.

[0087] d2) If the verification passes, the parsed operation relation data will be used as the successfully verified operation relation data.

[0088] Specifically, operational relationship data that has been completely parsed and successfully verified is considered as successfully verified operational relationship data. Table 4 is an operational relationship structure table provided in Embodiment 2 of the present invention. In Table 4, the operational relationship structure table includes template number, worksheet number, title name, operation formula, database table mapping result 1, data table field mapping result 1, database table mapping result 2, data table field mapping result 2, etc. The data corresponding to each type will not be described in detail here.

[0089] Table 4

[0090]

[0091]

[0092] The above technical solution specifies the steps of parsing and validating the operational relationships of the title fields in the target report template through an operational relationship parser. This realizes the determination of the mapping relationship between the operational relationships of the title fields and the tables and fields in the database, providing a basis for subsequent data reading and calculation.

[0093] For example, to more clearly describe the process of parsing the report template, let's take a specific application scenario as an example. Figure 3 This is a flowchart illustrating the process of parsing a report template according to Embodiment 2 of the present invention. Figure 3 As shown, the user first uploads a customized report template to the system, and this execution entity reads the uploaded template in pairs. Then, the report template is parsed to obtain the target table structure in the database, map field relationships, and confirm field types, which is then stored as a column header structure record. Next, the report field operation relationships are parsed to resolve template operation operators and map field relationships, which is then stored as a header operation relationship record. Finally, the report template parsing results are processed to update the template style configuration validation results and perform report pre-generation validation, thus responding to the analysis results. Updating the template style configuration validation results involves updating the template validation in Table 1. Report pre-generation validation is equivalent to verifying whether a report can be pre-generated in the background.

[0094] S204. Determine the data access layer strategy based on the data structure.

[0095] In this embodiment, the system initiates a response based on the user-defined and selected target report template and set query conditions, completing the processing of the large report and supporting user download operations. For example, a headquarters user configures a report template of type XX. A branch office will select this report template configured by the headquarters user as a download condition. The query conditions might include selecting a specific year, a specific province, and other conditions such as the transaction amount exceeding a certain threshold. When this template is applied, the data populated in the template is the data that the user can see, filtered by adding certain conditions.

[0096] This step is used to confirm the single-table data access layer. It reads the template parsing information of the target report template, which is to obtain the stored data structure. The data structure records the mapping relationships between column headers and header field operations, and between these mappings and the tables and fields in the database to be accessed. Based on this, it can be determined which data tables and fields need to be accessed. To support custom template reports, a data access mode for calling different database single-table data access layers is designed using a strategy pattern. The template information parsing has been completed above, and the database table mapping results and field mapping results involved in the parsing have been stored. The FindbyWhereStreamStrategy is defined as the unified system query entry point.<T1,T2> The interface is implemented by each individual table data access layer to define the query method. When a query occurs, the specific data access layer method strategy is determined based on the type of the query object.

[0097] S205. Establish a query connection from the streaming query connection pool to the newly configured streaming read data source and execute the streaming query.

[0098] This step involves obtaining a connection from the streaming query connection pool and establishing a query connection. To support potentially large data reports, a data source streaming read processing strategy is adopted. A new streaming read data source configuration is added to the original system data source; report data processing uses this independent data source to reduce the impact on original transactional transactions. A query connection is established with the newly configured streaming read data source, and a streaming query is executed.

[0099] S206. Based on the system resource memory usage, use a cursor to read the target data.

[0100] This step involves reading data. After the streaming query connection to the data source is established, a streaming query cursor is returned for the program to process. To balance processing efficiency and system load, the system summarizes the cursor data in stages based on the current memory usage. In this embodiment, the data read by the cursor is recorded as the target data.

[0101] As a specific implementation method, the step of reading target data using a cursor based on system resource memory usage can be optimized, including:

[0102] a3) Determine the allowable resource memory usage for report processing based on the system's maximum allowed resource memory usage and basic resource memory usage.

[0103] Specifically, the maximum allowed resource memory usage of the system can be understood as the upper limit of resource memory usage that cannot be exceeded, while the basic resource memory usage can be understood as the memory usage of other resources besides report processing. The difference between the maximum allowed resource usage and the basic resource memory usage is the resource memory usage available for report processing, denoted as the allowed resource memory usage for report processing.

[0104] b3) Determine the end position of the cursor reading stage based on the allowed resource memory usage of the report processing.

[0105] In this embodiment, the amount of data that the cursor can read is determined based on the allowed resource memory usage for report processing. This allows us to determine the cutoff point for each cursor phase read, which is denoted as the phase cutoff point. For example, suppose the rule defines the system's maximum allowed resource memory usage as 70%. Data aggregation occurs when memory pressure is below 70%. In a certain scenario, the cursor will return a total of 5 million data entries. Currently, during peak system business hours, the basic resource memory usage is 60%. When the phase aggregated data reaches 50,000 entries, the memory reaches a critical value, so cursor data reading stops, and processing begins.

[0106] c3) Use a cursor to read the target data until the stage cutoff position is reached.

[0107] Specifically, a cursor is used to read the target data until the stage's cutoff position is reached. Continuing with the example above, when the stage's summary data reaches 50,000 records and the memory reaches a critical value, cursor data reading stops, and processing begins.

[0108] The above technical solution specifies the steps for reading target data using a cursor based on system resource memory usage, thus balancing processing efficiency and system pressure.

[0109] S207. Process the target data to obtain the processed result.

[0110] Specifically, the phased summary processing includes operations such as displaying field data and processing report fields by reading the results of field operation parsing. By performing these processing steps on the target data, the processed results are obtained.

[0111] S208. Write the target data and the processed results into the target report template, and return to continue executing the streaming query step.

[0112] In this embodiment, after all the phase summary data has been processed, it is written to the user report using a streaming method. First, the report header information is generated based on the template selected by the user and written to the system user data buffer while returning a writable stream. Then, a field mapping relationship cache is generated based on the generated results. Based on this mapping relationship, the field position, style, and other information are confirmed and streamed into the writable stream to complete the processing of all data in one batch. The above process is repeated according to the cursor position until all data is written.

[0113] As a specific approach, the step of writing the target data and the processed result into the target report template can be optimized, including:

[0114] a4) Generate the target report header information based on the target report template.

[0115] Specifically, report header information is generated based on the template selected by the user.

[0116] b4) Based on the data structure, the target data and the processed results are written to the position corresponding to the header information of the target report using a streaming write method to obtain the target report.

[0117] In this embodiment, while writing to the system user data buffer, a writable stream is returned. Then, a field mapping relationship cache is generated based on the generated results. Based on the mapping relationship, the field position, style, and other information are confirmed and written to the writable stream in a streaming manner to complete the processing of a batch of data. The above process is repeated according to the cursor position until all data is written.

[0118] The above technical solution specifies that the target data and the processed results are written into the target report template, thus realizing the writing of the target report.

[0119] S209. Until the cursor reads empty, obtain the target report.

[0120] Specifically, once the processing is complete, i.e. when the cursor is empty, the cursor resources are released, the writable stream resources are closed, the target report is obtained, and the user is provided with the report to download and complete the report task.

[0121] For example, Figure 4 This is a flowchart illustrating a data processing procedure provided in Embodiment 2 of the present invention, as shown below. Figure 4As shown, the data processing procedure includes: reading template parsing information to confirm the data access layer method, i.e., reading and analyzing the stored data structure. Then, a connection is obtained from the streaming query connection pool; a streaming query is executed to return a CURSOR cursor; it is determined whether the CURSOR is empty; if it is not empty, the data stage summary is completed through the CURSOR, and the stage summary data is displayed, processed, etc. The template field mapping is completed to complete the streaming data writing, and the process returns to continue executing the streaming query to realize the next stage of data summary, processing, and writing; if it is empty, the process ends.

[0122] The above technical solution provides a configurable real-time ultra-large data reporting solution for accounting systems. It allows for real-time customization of report templates through flexible user configuration. The system parses the customized report templates to perform conditional analysis of the data to be downloaded, queries the raw data through streaming data processing, and writes the template data stream through data processing, fulfilling the user's report data processing needs. Users can create report templates, which the system can convert into an executable format and use streaming processing technology to query, process, and generate reports from ultra-large data volumes in real time. The use of streaming processing technology enables real-time processing of ultra-large data streams and generates reports that meet user needs, offering strong real-time performance. Users can customize report templates according to business requirements, and the system can automatically parse and generate reports that meet those requirements, offering high flexibility. The system development workload is low, does not affect the original business functions of the system, and completes the above functions with low investment costs, requiring no investment in hardware or software resources. Furthermore, it uses an independent data source for download management, does not occupy the original system's connection pool resources, and employs a memory limit control mechanism for data processing to ensure smooth transaction execution in the original system, minimizing system intrusion.

[0123] Example 3

[0124] Figure 5 This is a schematic diagram of a configurable real-time ultra-large data report processing system provided in Embodiment 3 of the present invention. This system is applicable to situations requiring real-time ultra-large data report processing. The configurable real-time ultra-large data report processing system can be implemented in hardware and / or software, and is generally integrated into electronic devices. For example... Figure 5 As shown, the system includes: a template customization module 31, a template parsing module 32, and a data processing module 33, wherein...

[0125] The template customization module 31 is used to receive the user's report template customization operation and generate the customized target report template.

[0126] The template parsing module 32 is used to parse the target report template, so as to convert the target report template into a data structure that the system can recognize and store it;

[0127] The data processing module 33 is used to read target data associated with the data structure from the database in a streaming manner, process the target data, and write the target data and the processed results into the target report template to obtain the target report.

[0128] It should be noted that this embodiment provides a configurable real-time ultra-large data report processing method. This method can be executed by a configurable real-time ultra-large data report processing system. This system can adopt a modular design, including a template customization module for configuring report template styles, a template parsing module for parsing user-customized report templates, and a data processing module for ultra-large data querying, processing, and downloading based on streaming processing technology. The system adopts a modular design, is easy to expand and upgrade, and can form pluggable components to meet the same type of needs of different application systems, exhibiting good scalability.

[0129] For example, Figure 6 This is a structural example diagram of a configurable real-time ultra-large data reporting system provided in Embodiment 3 of the present invention, as shown below. Figure 6 As shown, the template customization module implements functions including: report template user information management, historical report template information management, basic report template information management, and report template editing rule guidance. The template parsing module implements functions including: report template column header parsing, report template field operation relationship parsing, and report template result analysis and storage management. The data processing module implements functions including: obtaining the required data access object layer interface, establishing query connections, completing intermediate data processing, and completing data writing.

[0130] The above technical solution provides users with a configurable method for processing massive amounts of data in accounting systems. Users can customize report templates, then analyze, verify, and statistically analyze the target data's database tables and data conditions based on these templates. The table and field mapping relationships are stored in a data structure format. Finally, data is read from the database, logically processed, and written to the template to generate the target report. By providing users with simple and convenient template customization, the system development workload is low, and it does not affect the original business functions of the system. The above functions are completed at a low cost, requiring no investment in hardware or software resources. It is user-friendly and has high practicality and widespread applicability. Furthermore, the use of streaming processing technology enables real-time processing of massive data streams and the generation of reports that meet user needs, ensuring strong real-time performance. Report template styles can be configured by users and can be modified and added at any time without requiring development resources or repeated development modifications. It also supports downloading massive amounts of report data without affecting business transaction processing, greatly meeting the business needs of system users and improving the flexibility and service capabilities of accounting system reports.

[0131] Optionally, the template customization module 31 is specifically used for:

[0132] Create a blank report template, receive user customization requests for the blank report template, and generate the customized target report template; or,

[0133] Query historical report template style information / basic report template information management, and retrieve any historical report template style / basic report template style;

[0134] Use the historical report template style / basic report template style as the target report template; or...

[0135] The system receives user requests for customization of the historical report template style / basic report template style, and generates the customized target report template.

[0136] Optionally, the template parsing module 32 includes:

[0137] The column header parsing unit is used to parse and verify the column header information in the target report template through the header parser, obtain the successfully verified header data, and store the header data in a header structure table format;

[0138] The operation relationship parsing unit is used to parse and verify the operation relationship of the title field in the target report template through the operation relationship parser, obtain the successfully verified operation relationship data, and store the operation relationship data in the operation relationship structure table format.

[0139] Optionally, the column header parsing unit is specifically used for:

[0140] The title parser parses the column title information in the target report template based on a set parsing formula to obtain the parsed title data.

[0141] The mapping relationship between the parsed title data and the tables and fields in the database is validated.

[0142] If the verification fails, a verification exception is generated, and the user's re-editing operation is returned.

[0143] If the verification passes, the parsed title data will be used as the title data that has been successfully verified.

[0144] Optionally, the operation relation parsing unit is specifically used for:

[0145] The operation relationship parser is used to parse the operation relationship of the title field in the target report template to obtain the parsed operation relationship class data;

[0146] The parsed operational relation data is then used to verify the mapping relationship between the data and the tables and fields in the database.

[0147] If the verification fails, a verification exception is generated, and the user's re-editing operation is returned.

[0148] If the verification passes, the parsed operation relation data will be used as the successfully verified operation relation data.

[0149] Optionally, the data processing module 33 is specifically used for:

[0150] The strategy determination unit is used to determine the data access layer strategy based on the data structure.

[0151] The query connection unit is used to establish a query connection with the newly configured streaming data source from the streaming query connection pool and execute the streaming query;

[0152] The data reading unit is used to read target data using a cursor based on the system resource memory usage.

[0153] The processing unit is used to process the target data to obtain the processed result;

[0154] The data writing unit is used to write the target data and the processed results into the target report template and return to the step of continuing to execute the streaming query;

[0155] The report acquisition unit is used to acquire the target report until the cursor reads empty.

[0156] Optionally, the data reading unit is specifically used for:

[0157] The allowable resource memory usage for report processing is determined based on the system's maximum allowed resource memory usage and basic resource memory usage.

[0158] Based on the allowed resource memory usage for the report processing, determine the cutoff position for the cursor reading stage;

[0159] The target data is read using a cursor until the stage cutoff position is reached.

[0160] Optionally, the data writing unit is specifically used for:

[0161] Generate the target report header information based on the target report template;

[0162] According to the data structure, the target data and the processed results are written to the position corresponding to the header information of the target report using a streaming write method to obtain the target report.

[0163] The configurable real-time ultra-large data report processing system provided in this embodiment of the invention can execute the configurable real-time ultra-large data report processing method provided in any embodiment of the invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0164] Example 4

[0165] Figure 7 This is a schematic diagram of an electronic device according to Embodiment 4 of the present invention. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (such as helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0166] like Figure 7 As shown, the electronic device 40 includes at least one processor 41 and a memory, such as a read-only memory (ROM) 42 or a random access memory (RAM) 43, communicatively connected to the at least one processor 41. The memory stores computer programs executable by the at least one processor. The processor 41 can perform various appropriate actions and processes based on the computer program stored in the ROM 42 or loaded into the RAM 43 from storage unit 48. The RAM 43 may also store various programs and data required for the operation of the electronic device 40. The processor 41, ROM 42, and RAM 43 are interconnected via a bus 44. An input / output (I / O) interface 45 is also connected to the bus 44.

[0167] Multiple components in electronic device 40 are connected to I / O interface 45, including: input unit 46, such as keyboard, mouse, etc.; output unit 47, such as various types of monitors, speakers, etc.; storage unit 48, such as disk, optical disk, etc.; and communication unit 49, such as network card, modem, wireless transceiver, etc. Communication unit 49 allows electronic device 40 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0168] Processor 41 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 41 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 41 performs the various methods and processes described above, such as configurable real-time large-scale data reporting methods.

[0169] In some embodiments, the configurable real-time large-scale data reporting method can be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 48. In some embodiments, part or all of the computer program can be loaded and / or installed on electronic device 40 via ROM 42 and / or communication unit 49. When the computer program is loaded into RAM 43 and executed by processor 41, one or more steps of the configurable real-time large-scale data reporting method described above can be performed. Alternatively, in other embodiments, processor 41 can be configured to perform the configurable real-time large-scale data reporting method by any other suitable means (e.g., by means of firmware).

[0170] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0171] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0172] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0173] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0174] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or computing systems that include middleware components (e.g., application servers), or computing systems that include frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0175] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0176] This invention also provides a computer program product, including a computer program that, when executed by a processor, implements a configurable real-time large data report processing method as provided in any embodiment of this invention.

[0177] In implementing a computer program product, computer program code for performing the operations of this disclosure can be written in one or more programming languages ​​or a combination thereof. These programming languages ​​include, but are not limited to, object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0178] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0179] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A configurable real-time ultra-large data report processing method, characterized in that, include: Receive user requests to customize report templates and generate the customized target report template. The target report template is parsed to convert it into a system-recognizable data structure and then stored. The target data associated with the data structure is read from the database in a streaming manner, the target data is processed, and the target data and the processed results are written into the target report template to obtain the target report.

2. The method according to claim 1, characterized in that, The process of receiving the user's report template customization operation and generating the customized target report template includes: Create a blank report template, receive user customization requests for the blank report template, and generate the customized target report template; or, Query historical report template style information / basic report template information management, and retrieve any historical report template style / basic report template style; Use the historical report template style / basic report template style as the target report template; or... The system receives user requests for customization of the historical report template style / basic report template style, and generates the customized target report template.

3. The method according to claim 1, characterized in that, The step of parsing the target report template to convert it into a system-recognizable data structure and storing it includes: The column header information in the target report template is parsed and verified by the header parser to obtain the successfully verified header data, and the header data is stored in a header structure table format. The operation relationship parser parses and verifies the operation relationship of the title field in the target report template, obtains the successfully verified operation relationship data, and stores the operation relationship data in the operation relationship structure table format.

4. The method according to claim 3, characterized in that, The step of parsing and validating the column header information in the target report template using a header parser to obtain successfully validated header data includes: The title parser parses the column title information in the target report template based on a set parsing formula to obtain the parsed title data. The mapping relationship between the parsed title data and the tables and fields in the database is validated. If the verification fails, a verification exception is generated, and the user's re-editing operation is returned. If the verification passes, the parsed title data will be used as the title data that has been successfully verified.

5. The method according to claim 3, characterized in that, The step of parsing and validating the operational relationships of the title fields in the target report template using an operational relationship parser to obtain successfully validated operational relationship data includes: The operation relationship parser is used to parse the operation relationship of the title field in the target report template to obtain the parsed operation relationship class data; The parsed operational relation data is then used to verify the mapping relationship between the data and the tables and fields in the database. If the verification fails, a verification exception is generated, and the user's re-editing operation is returned. If the verification passes, the parsed operation relation data will be used as the successfully verified operation relation data.

6. The method according to claim 1, characterized in that, The process involves reading target data associated with the data structure from the database using a streaming method, processing the target data, and writing the target data and the processed results into the target report template to obtain the target report. This includes: Based on the data structure, determine the data access layer strategy; Establish a query connection from the streaming query connection pool to the newly configured streaming read data source and execute the streaming query; Based on the system resource memory usage, a cursor is used to read the target data; The target data is processed to obtain the processed result; Write the target data and the processed results into the target report template, and return to the step of continuing to execute the streaming query; The target report is obtained when the cursor reads an empty value.

7. The method according to claim 6, characterized in that, The step of reading target data using a cursor based on system resource memory usage includes: The allowable resource memory usage for report processing is determined based on the system's maximum allowed resource memory usage and basic resource memory usage. Based on the allowed resource memory usage for the report processing, determine the cutoff position for the cursor reading stage; The target data is read using a cursor until the stage cutoff position is reached.

8. The method according to claim 6, characterized in that, The step of writing the target data and the processed results into the target report template includes: Generate the target report header information based on the target report template; According to the data structure, the target data and the processed results are written to the position corresponding to the header information of the target report using a streaming write method to obtain the target report.

9. A configurable real-time ultra-large data reporting system, characterized in that, include: The template customization module is used to receive users' report template customization requests and generate the customized target report template. The template parsing module is used to parse the target report template, convert the target report template into a data structure that the system can recognize and store it; The data processing module is used to read target data associated with the data structure from the database in a streaming manner, process the target data, and write the target data and the processed results into the target report template to obtain the target report.

10. An electronic device, characterized in that, include: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the configurable real-time ultra-large data reporting method as described in any one of claims 1-8.

Images