Report data processing method, system, device, equipment, medium and program product

Abstract

The invention provides a report data processing method, system, device and equipment, a medium and a program product, relates to the technical field of computers, and can solve the problems of poor real-time performance, poor precision and poor service continuity of statistical report data. According to the specific technical scheme, a data updating request from a service system is received; converting the data updating request into a predefined general key value pair format to obtain converted key value pair data; sending the key value pair data to a report service node; according to the key value pair data, updating data in a local statistical report data table of the report service node; calculating a data item calibration value of a target data dimension based on the total business data at a predetermined time point, and sending the data item calibration value to a report service node to update a daily end reference value; and in response to a report query request, calculating to obtain a real-time statistical result based on the daily end reference value in the statistical report data table and the daily change value of the data item. The method is used for processing report data in a distributed scene.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a report data processing method, system, device, equipment, medium, and program product. Background Technology

[0002] In the information system architecture of banks and other financial institutions, statistical reporting is a core module supporting business decision-making and compliance monitoring. Traditionally, statistical reporting in centralized systems is achieved through direct querying or pre-summarizing data tables. However, with the expansion of banking operations and the evolution of system architecture towards a distributed model, data is scattered across multiple functional modules or deployment units. In a distributed computing environment, business systems typically consist of multiple functional modules deployed on different nodes. Under this architecture, data generation and storage are also correspondingly dispersed across various nodes. When it is necessary to generate comprehensive statistical reports across systems and nodes, traditional data aggregation solutions face significant challenges.

[0003] In existing technologies, the following solutions are mainly used to generate statistical reports: Solution 1: Detailed data is retrieved from various data sources in real time for statistical aggregation when querying reports. Solution 2: A centralized statistical report data table is updated in real time when business occurs, and the table is directly read during queries. However, these solutions all have significant drawbacks. Solution 1 becomes extremely time-consuming in real-time statistical operations when the data volume reaches the TB level, failing to meet the stringent response time requirements of high-concurrency online transactions. Solution 2, in high-concurrency scenarios, the centralized statistical report data table becomes a "data hotspot," triggering intense database lock contention, consuming a large amount of server resources, leading to prolonged system response time and decreased processing success rate. Furthermore, none of the above solutions fully consider the system's requirement for 24 / 7 business continuity, which may affect service availability during data updates and maintenance.

[0004] Therefore, existing technologies cannot guarantee the real-time performance and high accuracy of statistical report data, nor can they ensure the business continuity of the system. Summary of the Invention

[0005] This application provides a report data processing method, system, device, equipment, medium, and program product, which can ensure the real-time performance and high accuracy of statistical report data and improve the business continuity of the system.

[0006] To achieve the above objectives, the embodiments of this application adopt the following technical solutions: Firstly, a report data processing method is provided, comprising: receiving a data update request from a business system, the data update request carrying data change values ​​of a target data dimension; converting the data update request into a predefined general key-value pair format to obtain converted key-value pair data, wherein the target data dimension and its corresponding data item are combined as a key, and the data change value is recorded as a value; sending the key-value pair data to a report service node via an asynchronous message component; updating data in a local statistical report data table of the report service node based on the key-value pair data, the statistical report data table containing data dimensions, data items, end-of-day baseline values, daily change values ​​of data items, and data dates; calculating data item calibration values ​​of the target data dimension based on full business data at a predetermined time point, and sending the data item calibration values ​​to the report service node to update the end-of-day baseline values; and responding to a received report query request, calculating the current statistical value as a real-time statistical result based on the end-of-day baseline values ​​and daily change values ​​of data items in the statistical report data table.

[0007] In the report data processing method of this application embodiment, data transmission between the business system and the report service node is achieved through an asynchronous message component. This separates the high-concurrency online transactions and the background report statistics process into two relatively independent stages, effectively avoiding data hotspots and database contention issues caused by directly updating centralized statistical tables, thereby ensuring the processing performance and stability of the core business system. Furthermore, a data collaboration mechanism is constructed through a specific statistical report data table structure that includes a daily baseline value and the daily change value of data items, combined with a periodic batch calibration process. On the one hand, lightweight online asynchronous updates of the daily change value of data items ensure near real-time report data; on the other hand, batch full calculations are performed during off-peak business periods to obtain data item calibration values ​​to cover the daily baseline value, thereby correcting minor errors that may occur throughout the day due to message loss or other reasons, ensuring long-term data accuracy. In addition, this data separation structure allows the real-time update and batch calibration processes to be executed concurrently without blocking each other, achieving true 24 / 7 business continuity. Thus, this solution ensures the real-time performance and high accuracy of statistical report data, guaranteeing the business continuity of the system.

[0008] In one possible implementation of the first aspect, updating the data in the local statistical report data table of the report service node based on key-value pair data includes: determining the relationship between the received data date and the data date in the statistical report data table; if the data dates are the same, then accumulating the received data change value to the daily change value of the data item in the statistical report data table; if the received data date is greater than the data date in the statistical report data table, then using the sum of the daily baseline value and the daily change value of the data item in the statistical report data table as the new daily baseline value, clearing the daily change value of the data item in the statistical report data table, and updating the data date in the statistical report data table to the received data date.

[0009] Thus, by setting clear date judgment and data merging rules, precise operational logic is provided for updating online data. When dates are the same, only the changed value is accumulated, which allows high-concurrency update operations to be performed lock-free or with fine locking, improving concurrency performance. When a date change is detected, a day-switching operation is automatically executed: the sum of the end-of-day baseline value and the daily changed value of the data item is used as the new baseline, and the changed value is cleared to zero, ensuring the continuity of data in the time dimension, while enabling the system to automatically complete the initialization of daily data without interrupting service.

[0010] In another possible implementation of the first aspect, sending the data item calibration value to the report service node to update the end-of-day baseline value includes: determining the relationship between the data date of the data item calibration value and the data date in the statistical report data table; if the data dates are the same, updating the end-of-day baseline value to the data item calibration value; if the data date of the data item calibration value is greater than the data date in the statistical report data table, updating the end-of-day baseline value to the data item calibration value, clearing the daily change value of the data item in the statistical report data table, and updating the data date in the statistical report data table to the data date of the data item calibration value.

[0011] Thus, this implementation sets a date determination rule similar to but independent of online updates for the batch calibration process, ensuring that batch calibration data is correctly aligned with online update data on the timeline. Whether overwriting the baseline value of the current day or performing a day-cut overwrite, this rule guarantees the validity of the full batch calculation results and can effectively correct for accumulated errors that may be caused by asynchronous online updates.

[0012] In another possible implementation of the first aspect, before converting the data update request into a predefined general key-value pair format to obtain the converted key-value pair data, the method further includes: establishing a report configuration table, which stores the mapping relationship between the unique identifiers of different business reports and the data dimension definitions and data item definitions contained therein. The data dimension definition includes dimension hierarchy and dimension encoding rules, and the data item definition includes data item name, data type, and calculation relationship. The conversion of the data update request into a predefined general key-value pair format to obtain the converted key-value pair data includes: querying the report configuration table according to the business scenario to which the data update request belongs, and obtaining the corresponding data dimension definition and data item definition; performing dimension combination on the original business data contained in the data update request according to the dimension encoding rules in the obtained data dimension definition, and generating a dimension combination string as the key; calculating the data change value that conforms to the data type from the original business data contained in the data update request according to the calculation relationship in the obtained data item definition, and recording it as a value to obtain the converted key-value pair data.

[0013] In this way, by using the report configuration table and the dimension coding rules and calculation relationships defined therein, the data transformation process is transformed into a configuration-driven and highly flexible data processing process. This allows for dynamic querying of configurations based on business scenarios and the generation of standardized keys and values ​​from the original business data according to the configuration rules. New business reports or data formats can be adapted without modifying the program code, reducing maintenance costs and improving adaptability and real-time performance to changes in business requirements.

[0014] In another possible implementation of the first aspect, the above-mentioned response to the received report query request, based on the daily baseline value and the daily change value of the data item in the statistical report data table, calculates the current statistical value as the real-time statistical result, including: parsing the report query request to obtain the target report identifier and query conditions; obtaining the corresponding format template from a predefined template library according to the target report identifier; wherein the above-mentioned format template is used to map the data dimension to the row identifier of the output report and to map the data item to the column identifier of the output report; and obtaining all matching data items from the statistical report data table according to the query conditions. The key-value pair data is processed as follows: For each matching key-value pair, the daily baseline value and the daily change value of the data item are added together to obtain the current statistical value corresponding to the matching key-value pair; using the format template as the mapping rule, multiple current statistical values ​​calculated from multiple key-value pairs with the same data dimension but different data items are aggregated into the same row of data; where the data dimension is written into the identifier field of the same row, and the current statistical values ​​corresponding to different data items are filled into different data columns of the same row; the aggregated row-based data set is output as the real-time statistical result.

[0015] In this way, the internal general storage format is reverse-converted into the specific report format required by the business. Data dimensions and data items are mapped to rows and columns of a two-dimensional table through format templates, and column-to-row aggregation operations are performed. This efficiently completes the conversion from a storage model optimized for calculation to a report model optimized for display. This process shields the complexity of the underlying data structure and provides a simple and well-organized data view to upper-layer applications, enabling the query service to quickly and flexibly generate user-visualized reports that meet various customized needs.

[0016] In another possible implementation of the first aspect, the method further includes: periodically scanning the statistical report data table; identifying data dimensions and data item combinations in the statistical report data table that have not been accessed by any report query request within a preset time range; and marking the data corresponding to the identified data dimensions and data item combinations as invalid or deleting them from the statistical report data table.

[0017] In this way, by periodically scanning and cleaning up cold data that has not been accessed for a long time, an automated data lifecycle management mechanism is introduced. This effectively solves the problem of wasted storage space and decreased query performance caused by the accumulation of a large amount of invalid or expired data in statistical report data tables after long-term operation. It can automatically reclaim storage resources and improve the system's self-maintenance capability and long-term operational stability.

[0018] Secondly, a report data processing system is provided, comprising: a business layer and a framework layer. The business layer includes an online data collection module, a batch data collection module, and a statistical report query service module. The framework layer includes a data processing module, a data synchronization module, and a data merging module. The online data collection module receives data update requests from the business system, the data update requests carrying data change values ​​for the target data dimension. The data processing module converts the data update requests into a predefined general key-value pair format to obtain converted key-value pair data, wherein the target data dimension and its corresponding data item are combined as a key, and the data change values ​​are recorded as values. The data synchronization module sends the key-value pair data to the report service node via an asynchronous message component. The data merging module... The data merging module is used to update the data in the local statistical report data table of the report service node based on key-value pairs. This statistical report data table includes data dimensions, data items, daily baseline values, daily changes in data items, and the data date. The batch data collection module is used to calculate the calibration values ​​of data items for the target data dimensions based on the full amount of business data at a predetermined time point. The data synchronization module is also used to send the calibration values ​​of data items to the report service node via file transfer. The data merging module is also used to update the daily baseline values ​​in the statistical report data table based on the calibration values ​​of data items. The statistical report query service module is used to respond to the received report query request and calculate the current statistical value as the real-time statistical result based on the daily baseline values ​​and daily changes in data items in the statistical report data table.

[0019] Thirdly, a report data processing device is provided, comprising: a receiving module for receiving a data update request from a business system, the data update request carrying data change values ​​of a target data dimension; a processing module for converting the data update request into a predefined general key-value pair format to obtain converted key-value pair data, wherein the target data dimension and the corresponding data item are combined into a key, and the data change value is recorded as a value; a sending module for sending the key-value pair data to a report service node via an asynchronous message component; the processing module is further configured to update data in a local statistical report data table of the report service node based on the key-value pair data, the statistical report data table including data dimension, data item, end-of-day baseline value, daily change value of data item, and data date; the processing module is further configured to calculate a data item calibration value of the target data dimension based on full business data at a predetermined time point, and send the data item calibration value to the report service node to update the end-of-day baseline value; the processing module is further configured to respond to a received report query request, and calculate the current statistical value as a real-time statistical result based on the end-of-day baseline value and daily change value of data item in the statistical report data table.

[0020] It should be noted that the beneficial effects of the report data processing device can be found in the relevant description of the beneficial effects of the report data processing method mentioned above, and will not be repeated here to avoid repetition.

[0021] Fourthly, an electronic device is provided, the method comprising: a memory and at least one processor. The memory is communicatively connected to the processor. The memory is used to store computer program code, the computer program code including computer instructions. When the processor executes the computer instructions, it causes the electronic device to perform the method as described in the first aspect and any possible implementation thereof.

[0022] Fifthly, embodiments of this application provide a computer-readable storage medium storing computer instructions. When executed by a processor, these computer instructions are used to implement the method described in the first aspect and any possible implementation thereof.

[0023] Sixthly, embodiments of this application provide a computer program product that, when run on a computer / executed by a computer's processor, implements the method described in the first aspect and any possible design thereof. The computer may be the electronic device described in the fourth aspect and any possible implementation thereof.

[0024] Understandably, the beneficial effects achieved by the system of the second aspect, the apparatus of the third aspect, the electronic equipment of the fourth aspect, the computer-readable storage medium of the fifth aspect, and the computer program product of the sixth aspect provided above can be referred to as the beneficial effects of the first aspect and any possible implementation thereof, which will not be repeated here. Attached Figure Description

[0025] Figure 1 A schematic diagram of a system architecture applied to a report data processing method provided in an embodiment of this application; Figure 2 A flowchart illustrating a report data processing method provided in an embodiment of this application; Figure 3 A schematic diagram illustrating the processing flow of a report data processing method provided in an embodiment of this application; Figure 4 This is a schematic diagram of the structure of a report data processing system provided in an embodiment of this application; Figure 5 This is a schematic diagram of the structure of a report data processing device provided in an embodiment of this application; Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0026] Hereinafter, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this embodiment, unless otherwise stated, "a plurality of" means two or more.

[0027] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0028] The technical solutions provided in this application, including the collection, storage, use, processing, transmission, provision, and disclosure of financial data or user data, comply with relevant laws and regulations and do not violate public order and good morals.

[0029] It should be noted that in the embodiments of this application, certain software, components, models and other existing solutions in the industry may be mentioned. These should be regarded as exemplary and are only intended to illustrate the feasibility of implementing the technical solution of this application. However, they do not mean that the applicant has used or necessarily used the solution.

[0030] The terms / concepts involved in the embodiments of this application are explained below.

[0031] Centralized system: A central node consisting of one or more host computers, where data is centrally stored and all business units of the entire system are centrally deployed, and all functions of the system are centrally processed by it.

[0032] Distributed system: A system consisting of multiple deployment units that work together to complete the same business logic.

[0033] KAFKA: A high-throughput distributed publish-subscribe messaging system.

[0034] NFT: A tool for transferring files over a network.

[0035] Online transactions: Transactions submitted directly by the system, characterized by high real-time performance, high concurrency, and atomicity. Examples include cash withdrawals and balance inquiries.

[0036] Batch processing: The system performs a certain type of operation at fixed intervals, characterized by large data volume processing, controllable processing flow, and support for resuming from breakpoints.

[0037] Data hotspots: Some data in the system is accessed by a large number of threads at the same time, which leads to lock contention and a decrease in system concurrency.

[0038] SQL: A language for manipulating databases, including creating databases, deleting databases, querying records, modifying records, adding fields, etc.

[0039] The report data processing apparatus and the report data processing method applicable to the report data processing apparatus provided in this application will be explained and described below with reference to the accompanying drawings and through specific embodiments and application scenarios.

[0040] The report data processing methods, systems, devices, equipment, media, and program products provided in this application can be applied to any scenario that requires near real-time data statistics and report generation in a distributed, high-concurrency environment, such as scenarios that require data accuracy, system performance, and business continuity.

[0041] For example, in the field of fintech, it can be specifically applied to: Core Bank Business Management: Real-time statistics on key operating indicators such as deposit / loan volume, transaction volume, and sales of wealth management products for the entire bank and its branches, supporting management decision-making.

[0042] Insurance business reports: Generates reports such as policy sales, premium income, and claims statistics in near real-time to meet management reporting and internal risk control needs.

[0043] Securities transaction clearing: In a distributed trading system, the transaction volume and fund changes of each node are aggregated in real time, and a full clearing and data calibration are performed at the end of the day to ensure absolute accounting accuracy.

[0044] Furthermore, in the fields of e-commerce and the internet, it can be specifically applied to: Real-time dashboard for major promotional events: During certain promotional events, real-time statistics and displays of total transaction volume, order volume, user visits, and other data across the entire platform, various categories, and merchants are provided, while ensuring that the backend statistics system does not affect the frontend transaction and order placement process.

[0045] User behavior analysis reports: Generates near real-time analysis reports on user activity, product click-through rate, and ad conversion rate, supporting rapid adjustments to operational strategies.

[0046] For example, in the fields of network operations or the Internet of Things, it can be specifically applied to: Call charges and data usage statistics: Real-time aggregation of user call details and data usage data, generating near real-time call charge bills and data usage reports, and providing an accurate calibration basis for monthly bills.

[0047] Multi-tenant resource metering and billing: In the distributed cloud platform, the computing, storage, and network resource consumption data of each node are collected in real time to generate near real-time cost reports for each tenant and ensure the long-term accuracy of billing data.

[0048] Data aggregation from IoT devices: Data such as operating status and sensor readings reported by a large number of IoT devices need to be aggregated in real time to generate statistical reports for regions or device types for monitoring and early warning.

[0049] In the daily operations of banks, management requires the system to be able to organize, classify, and summarize detailed data to reflect the bank's business activities and financial income and expenditure. However, as the banking system migrates from a centralized system to a distributed system, the following problems have arisen.

[0050] First, in a distributed environment, because the system functions are divided into multiple deployment units according to functional modules, the data is split into multiple deployment units, which means that reports need to summarize data from multiple systems, increasing system complexity.

[0051] Secondly, as business grows, the system needs to maintain extremely high capacity. Real-time reporting and statistics solutions consume a great deal of system resources, reducing system capacity; at the same time, high-concurrency system updates also bring about the problem of hot data.

[0052] Then, business development brings a large number of statistical reporting scenarios, and the development of functions for specific businesses wastes a lot of human resources.

[0053] Finally, the bank's core system requires 24 / 7 continuous operation, and the original solution could not meet these requirements.

[0054] To achieve this function, the following solutions are generally adopted: Option 1 involves performing data statistics in real time when a report query occurs.

[0055] Option 2 involves updating the statistical report data table in real time when relevant business transactions occur, while statistical report querying only requires querying the statistical report data table.

[0056] Option 3 involves the system periodically compiling and summarizing statistical reports in batches.

[0057] First, Option 1, which performs report data statistics only when the system's data volume is small, is only suitable for systems with small data volumes. When the system's data volume reaches the TB level, the detailed data summary operation often takes several minutes, exceeding the online transaction response time target. Option 2 requires real-time updates to statistical report data when related business occurs. However, under high system concurrency, the report data becomes hot data, leading to a large amount of lock contention in the database (updating a piece of data in the database will result in a row lock on that record on the index; if there is no index, it will result in a table lock), consuming server resources. Lock waiting and failed lock requests lead to longer system response times and a lower success rate. Option 3, which uses a periodic statistical method, suffers from low timeliness of statistical data and errors during synchronization. Finally, none of the above options consider the bank system's requirement for system continuity and do not implement a 24 / 7 design.

[0058] Therefore, existing technologies cannot guarantee the real-time performance and high accuracy of statistical report data, nor can they ensure the business continuity of the system.

[0059] In the technical solution provided in this application embodiment, for the problem of updating statistical data in high-concurrency scenarios: this solution uses an asynchronous message mechanism to asynchronously update statistical data to solve the impact of data hotspots on online transactions. At the same time, in order to solve the problem of message blocking in the message middleware or message loss causing delays or errors in statistical report data, a mechanism of online near real-time data update and batch daily calibration is adopted to ensure the near real-time performance and accuracy of the data.

[0060] For 24 / 7 business continuity design: This solution employs a combination of online asynchronous processing and end-of-day aggregation to synchronize data and address statistical data update issues in high-concurrency scenarios. To address the problem of data merging during simultaneous online asynchronous processing and end-of-day aggregation, this solution expands the data definition. The system changes from simply storing and transmitting the current data value to storing and transmitting the data period, the initial data value for the current period, and the data change values. It performs batch updates of the initial data value for the current period, online updates of data change values, and resets the initial data value and data change values ​​for the current period when the data period changes.

[0061] Thus, this solution addresses the impact of data hotspots on the system by using an asynchronous message synchronization mechanism; the unified data transmission format ensures compatibility with different business scenarios; and the 24 / 7 design guarantees business continuity.

[0062] This application provides a system architecture for a report data processing method. In this application, the business layer and framework layer are designed separately to build a high-concurrency, near-real-time, and highly available statistical report data processing platform. This system architecture describes a distributed data synchronization and processing platform. For example... Figure 1As shown, the system architecture can be divided into three parts: the data sending end, the data synchronization middleware, and the data receiving end, supporting both real-time and batch data processing modes. Through the collaboration between the business layer and the framework layer, the system achieves the collection, synchronization, merging, and querying of statistical report data. The business layer directly addresses specific business scenarios, realizing data collection and final service provision; the framework layer provides standardized, decoupled technical capabilities to ensure the reliability and efficiency of data transmission and processing.

[0063] Specifically, the entire data flow comprises two main threads: Online near real-time streaming: Processes high-concurrency incremental data to ensure near real-time reporting; Batch calibration stream: Process all data during off-peak hours to ensure long-term report accuracy.

[0064] The components of the business layer and framework layer are described below: 1. Data sending end The data sending end is responsible for collecting data from the business source, performing preliminary processing, and sending it.

[0065] The business layer module includes: Online data collection module: This module is embedded in various online transactions and is the starting point for business data generation. It is triggered when a business operation (such as a transaction or repayment) is completed, and synchronously collects the statistical report data changes brought about by this operation. Batch Data Collection Module: This module is independent of online transactions and is mainly used for data correction. It starts at a predetermined time (such as at night) and recalculates the accurate values ​​of statistical indicators based on the full amount of business data to correct the cumulative errors that may be generated by the online stream.

[0066] Framework layer modules, including: Data Processing Unit: This unit is key to decoupling the business layer from the framework layer. It receives raw business data of different formats from the two collection modules in the upper layer and converts it into a predefined general key-value pair format so that it can be uniformly transmitted and stored within the framework layer. Online data sending unit: responsible for submitting the processed online key-value pair data to the data synchronization middleware; Batch data sending unit: responsible for submitting the batch-calculated calibration data to the data synchronization middleware.

[0067] Data storage, including: Distributed database: Used to store the final statistical data after processing by the data merging module. This database stores key fields including data dimensions, data items, daily baseline values, and daily changes in data items to support efficient real-time queries.

[0068] 2. Data synchronization middleware Data synchronization middleware acts as a bridge connecting the sender and receiver, employing two different synchronization mechanisms to adapt to the needs of different scenarios.

[0069] Data synchronization middleware includes: Asynchronous messaging component: Used to transmit online incremental data sent by the online data sending unit. This component (such as Kafka) solves the data hotspot problem caused by the strong coupling between online transaction and statistical report data processing through an asynchronous decoupling mechanism, and meets the near real-time requirements. File transfer component: Used to transfer batch calibration data sent by the batch data sending unit. To ensure 100% arrival of calibration data and avoid long-term error expansion due to message loss, this component adopts a more reliable file transfer method.

[0070] 3. Data receiving end The data receiving end is responsible for the final merging, storage, and query services of the data.

[0071] Framework layer modules, including: The online data receiving unit receives data from the asynchronous message component; The batch data receiving unit receives data from the file transfer component; Data processing unit: This unit plays the role of reverse data format conversion at the receiving end, restoring the general key-value pair format data circulating in the framework layer into the specific format required by the business layer or storage layer; Data merging module: This module is the core of ensuring 24 / 7 service continuity and eventual data consistency. It is responsible for merging received online incremental updates and batch calibration data into local storage. Its functions include: Collaborative updates: Ensure that online dynamic updates and batch static data corrections can be performed concurrently without blocking each other; Date switching processing: Through intelligent date judgment and merging rules, it ensures data continuity and accuracy when switching dates; Data cleanup: Clean up the statistical report data that has been taken offline to prevent dirty data from affecting system stability.

[0072] Data storage, including: Distributed database: Used to store the final statistical data after processing by the data merging module. This database stores key fields including data dimensions, data items, daily baseline values, and daily changes in data items to support efficient real-time queries.

[0073] The business layer module includes: Statistical report query service: This is the final service provider for users. It responds to the front-end report query requests, calculates the current statistical values ​​in real time based on the daily baseline value and the daily change value of data items in the distributed database, and displays the results to users after customized processing.

[0074] This architecture achieves a balance between business variability and technical stability by separating the business layer from the framework layer. Through a dual mechanism of online asynchronous updates and batch end-of-day calibration, it ensures near real-time data performance and long-term accuracy while maintaining system processing performance and decoupling.

[0075] It is understandable that the business layer of this solution mainly completes the collection of statistical reports for various business scenarios and the implementation of various statistical report query services. The business layer is used to adapt to various business scenarios and isolate the business from the underlying layer. The framework layer mainly performs basic functions such as data formatting and processing, cross-system data synchronization, and online transaction data merging.

[0076] The business layer consists of three modules: an online data collection module, a batch data collection module, and an online statistical report query service. The online data collection module is primarily embedded within various online transactions to collect data related to statistical reports. The batch data collection module is designed to address the possibility of data loss in asynchronous message components, which could lead to errors in statistical reports, and these errors might gradually increase as the system runs. To solve this problem, this solution is designed to perform a full data aggregation at night to correct existing data. The online statistical report query service is the final service provider for statistical reports, customizing and processing statistical data for different business scenarios before presenting it to users.

[0077] The framework layer consists of three modules: a data processing module, an online batch data synchronization module, and a data merging module. The data processing module formats data collected from different business scenarios and then transmits and stores it within the framework layer, facilitating decoupling between the framework layer and business logic. The online batch data synchronization module primarily synchronizes data from various distributed nodes to the node hosting the online query service for statistical reports. The online synchronization utilizes asynchronous message components, such as Kafka. This asynchronous message mechanism addresses the data hotspot problem caused by the strong coupling between online transaction and statistical report data, and it also meets near real-time performance requirements. The batch synchronization module addresses the issue that online asynchronous messages cannot always arrive 100%, leading to increasingly larger errors in data reports over time. Therefore, it introduces batch processing to perform statistical analysis on all data, correcting the data reports, and using file transfer to synchronize data and ensure 100% arrival. Data merging ensures that statistical reports can be provided 24 / 7, preventing congestion during online and batch data synchronization. It also ensures that online static data can be updated concurrently and dynamically for correction. Furthermore, the data merging module incorporates a data cleanup function to clean up offline statistical report data, preventing dirty data from affecting system stability.

[0078] This application provides a report data processing method, which can be applied to a report data processing device or an electronic device. The following illustration uses a report data processing device executing the report data processing method as an example. Figure 2 As shown, the report data processing method may include the following steps 201 to 206.

[0079] Step 201: The report data processing device receives a data update request from the business system.

[0080] In some embodiments of this application, the aforementioned data update request carries the data change value of the target data dimension.

[0081] In some embodiments of this application, the aforementioned target data dimension refers to the perspective or grouping conditions used when performing data statistics. For example, in a banking system, it could be branch A_product 1, and in an e-commerce system, it could be a region_mobile phone category.

[0082] In some embodiments of this application, the aforementioned data change value refers to the numerical change of a certain statistical indicator under this data dimension during a single business operation.

[0083] In some embodiments of this application, the aforementioned data update request is typically generated synchronously within a business transaction, but its subsequent processing will be asynchronous.

[0084] In some embodiments of this application, the report data processing device can receive data update requests from the business system through an online data collection module at the business layer. This online data collection module is used to collect statistical report data for various business scenarios. It is designed to be embedded in various online transactions to collect data related to statistical reports. For example, when a user performs a repayment transaction, this module is triggered and receives the data changes brought about by this transaction.

[0085] Step 202: The report data processing device converts the data update request into a predefined general key-value pair format to obtain the converted key-value pair data.

[0086] In some embodiments of this application, the aforementioned target data dimension and corresponding data item are combined into a key, and the data change value is recorded as a value.

[0087] In some embodiments of this application, the aforementioned data items refer to specific indicators that need to be statistically analyzed, such as loan balance and overdue amount.

[0088] In some embodiments of this application, the aforementioned general key-value pair format is a standardized data intermediate format that combines dimensions and items into a key and uses the change value as the value, for example, Key=Branch A_Product 1_Loan Balance, Value=+10. The purpose of this transformation is to unify diverse business data into a format that is easy for the framework layer to transmit, store, and process.

[0089] In some embodiments of this application, the report data processing device can convert the update request into a predefined general key-value pair format through the data processing module of the framework layer to obtain the converted key-value pair data. The data processing module is used to format the data collected from different business scenarios and then transmit and store it in the framework layer so as to decouple the framework layer from the business. Its purpose is to convert the data into KEY-VALUE.

[0090] In some embodiments of this application, before step 202 above, the report data processing method provided in this application embodiment further includes step 301 below, and step 202 above can be specifically implemented by steps 202a to 202c below.

[0091] Step 301: The report data processing device establishes a report configuration table.

[0092] In some embodiments of this application, the above-mentioned report configuration table is used to store the mapping relationship between the unique identifiers of different business reports and the data dimension definitions and data item definitions contained therein. The data dimension definition includes dimension hierarchy and dimension encoding rules, and the data item definition includes data item name, data type and calculation relationship.

[0093] In some embodiments of this application, the above-mentioned dimension encoding rules define how to concatenate original business data fields (such as branch number, product number) into a unique dimension string (such as branch A_product 1).

[0094] In some embodiments of this application, the above calculation relationship defines how to derive standardized data change values ​​from the original business data. For example, for loan balance, the calculation relationship may be to directly take the transaction amount, while for overdue amount, the calculation relationship may need to make conditional judgments based on business status.

[0095] Step 202a: The report data processing device queries the report configuration table according to the business scenario to which the data update request belongs, and obtains the corresponding data dimension definition and data item definition.

[0096] In some embodiments of this application, the report data processing device can search the configuration table based on the source of the data update request or the built-in business type identifier to locate all the rules required to process the business, that is, to obtain the corresponding data dimension definition and data item definition.

[0097] Step 202b: The report data processing device performs dimension combination on the original business data contained in the data update request according to the dimension encoding rules in the obtained data dimension definition, and generates a dimension combination string as the key.

[0098] For example, the report data processing device extracts the branch number = branch A and the product number = product 1 from the original request, and then concatenates them according to the rule {branch number}_{product number} to generate branch A_product 1.

[0099] Step 202c: The report data processing device calculates the data change value that conforms to the data type from the original business data contained in the data update request according to the calculation relationship in the obtained data item definition, and records it as a value to obtain the converted key-value pair data.

[0100] For example, the report data processing device extracts the repayment amount from the data update request based on the calculation relationship of the loan balance, uses it as the change value, and finally generates a complete key-value pair record. See Tables 4 to 6 below for details.

[0101] In this way, by using the report configuration table and the dimension coding rules and calculation relationships defined therein, the data transformation process is transformed into a configuration-driven and highly flexible data processing process. This allows for dynamic querying of configurations based on business scenarios and the generation of standardized keys and values ​​from the original business data according to the configuration rules. New business reports or data formats can be adapted without modifying the program code, reducing maintenance costs and improving adaptability and real-time performance to changes in business requirements.

[0102] Step 203: The report data processing device sends key-value pair data to the report service node through the asynchronous message component.

[0103] In some embodiments of this application, the report data processing device executes through the online batch data synchronization module of the framework layer: key-value pair data is sent to the report service node via an asynchronous message component. The online synchronization method uses an asynchronous message component, such as Kafka. The asynchronous message mechanism solves the data hotspot problem caused by the strong coupling between online transaction and statistical report data, and also meets near real-time requirements.

[0104] In some embodiments of this application, the aforementioned asynchronous messaging component (such as Kafka) is a high-throughput distributed publish-subscribe messaging system. Specifically, the sender publishes messages to a specified topic, and the consumer (report service node) subscribes to and retrieves messages from the topic. The sender can return without waiting for the receiver to finish processing, thus avoiding blocking high-concurrency online transactions. Even if the receiver is temporarily unable to process the messages, the messages will accumulate in the message queue without affecting the business system. The asynchronous messaging mechanism solves the data hotspot problem caused by the strong coupling between online transactions and statistical report data, and it also meets near real-time requirements.

[0105] Step 204: The report data processing device updates the data in the local statistical report data table of the report service node based on the key-value pair data.

[0106] In some embodiments of this application, the above-mentioned statistical report data table includes data dimensions, data items, end-of-day baseline values, daily changes in data items, and data dates.

[0107] In some embodiments of this application, the end-of-day baseline value can be the end-of-day value of the previous day.

[0108] In some embodiments of this application, the aforementioned end-of-day baseline value (i.e., the previous day's end-of-day value) is a batch-calibrated, accurate value at the start of each day. The aforementioned daily change value of the data item is the cumulative value of the changes generated by all online asynchronous updates within the day. The aforementioned data date is used to identify the business date to which the current data belongs.

[0109] It is understood that in this application embodiment, for the problem of updating statistical data in high-concurrency scenarios: this solution uses an asynchronous message mechanism to asynchronously update statistical data to solve the impact of data hotspots on online transactions. At the same time, in order to solve the problem of message blocking in the message middleware or message loss causing delays or errors in statistical report data, a mechanism of online near real-time data update and batch daily calibration is adopted to ensure the near real-time performance and accuracy of the data.

[0110] For 24 / 7 business continuity design: To address the issue of data merging during simultaneous online asynchronous processing and end-of-day aggregation, this solution expands the data definition. The system changes from simply storing and transmitting the current data value to storing and transmitting the data period, the initial data value for the current period, and the data change value. It batch updates the initial data value for the current period, updates the data change value online, and resets the initial data value and data change value for the current period when the data period changes. The specific principle is as follows: The online data synchronization interfaces are shown in Table 1 below: Table 1

[0111] The batch data synchronization interface is shown in Table 2 below: Table 2

[0112] The database storage format is shown in Table 3 below: Table 3

[0113] The specific merging rules are as follows: When the date of the data stored in the database is equal to the date of the data in the data synchronization interface, the online synchronization interface accumulates the changes of the day, and the batch synchronization interface replaces the end value of the previous day.

[0114] When the date of the data stored in the database is less than the date of the data in the data synchronization interface (this scenario is generally sent after 23:59 for the first synchronization operation), the online synchronization interface will add the change value of the day to the end value of the previous day, and then start accumulating the change value of the day from zero, while adding the data date. The batch synchronization interface replaces the end value of the previous day, clears the change value of the day, and adds the data date.

[0115] When the date of the data stored in the database is later than the date of the data in the data synchronization interface (this scenario is generally a case where the data synchronization from the previous day is congested until 00:00 the next day): the data expires and the synchronized data is discarded.

[0116] General data format: To isolate the impact of different business scenario data formats on the overall solution, the data undergoes format conversion upon entering the framework layer, transforming it into KEY-VALUE. The conversion rules are as follows, using personal loan scale statistics reports and micro-loan scale statistics reports as examples.

[0117] The data format for the individual loan scale statistical report is shown in Table 4 below (the following report represents the loan balance and overdue amount of a certain product of a certain branch): Table 4

[0118] The data format for the statistical report on the scale of micro and small loans is shown in Table 5 below (the following reports represent the number of loan accounts and the amount of write-offs for a certain institution): Table 5

[0119] The results of the column-to-row transformation of the data by the data processing module are shown in Table 6 below: Table 6

[0120] Through the above conversion, data from different business scenarios can be transmitted and stored in the same data format. When a data report query occurs, the data processing module will reverse the data to restore it.

[0121] In some embodiments of this application, step 204 can be specifically implemented by steps 204a to 204c as described below.

[0122] Step 204a: The report data processing device determines the relationship between the received data date and the data date in the statistical report data table.

[0123] Step 204b: If the data dates are the same, the report data processing device will add the received data change values ​​to the daily change values ​​of the data items in the statistical report data table.

[0124] Step 204c: If the date of the received data is greater than the date of the data in the statistical report data table, the report data processing device will use the sum of the end-of-day baseline value and the daily change value of the data item in the statistical report data table as the new end-of-day baseline value, clear the daily change value of the data item in the statistical report data table, and update the date of the data in the statistical report data table to the date of the received data.

[0125] In some embodiments of this application, the report data processing device executes the following through the data merging module of the framework layer: updating the data in the local statistical report data table of the report service node based on key-value pair data. This data merging module is responsible for ensuring that statistical reports can be provided 24 / 7, preventing congestion during online and batch data synchronization, and ensuring concurrent dynamic updates and corrections of statistical data during batch static data online processing. Its processing logic, i.e., the specific merging rules, is as follows: When the date of data stored in the database equals the date of data in the data synchronization interface, the online synchronization interface accumulates the daily change value. When the date of the data stored in the database is less than the date of the data in the data synchronization interface (this scenario is generally sent after the first synchronization operation at 23:59), the online synchronization interface will add the change value of the day to the end value of the previous day, and then start accumulating the change value of the day from zero, while adding the data date.

[0126] Thus, by setting clear date judgment and data merging rules, precise operational logic is provided for updating online data. When dates are the same, only the changed value is accumulated, which allows high-concurrency update operations to be performed lock-free or with fine locking, improving concurrency performance. When a date change is detected, a day-switching operation is automatically executed: the sum of the end-of-day baseline value and the daily changed value of the data item is used as the new baseline, and the changed value is cleared to zero, ensuring the continuity of data in the time dimension, while enabling the system to automatically complete the initialization of daily data without interrupting service.

[0127] Step 205: At a predetermined time point, the report data processing device calculates the data item calibration value of the target data dimension based on the full amount of business data, and sends the data item calibration value to the report service node to update the end-of-day baseline value.

[0128] In some embodiments of this application, the report data processing device, through a batch data collection module at the business layer and a data synchronization module at the framework layer, calculates the data item calibration value for the target data dimension based on the full amount of business data at a predetermined time point, and sends the data item calibration value to the report service node to update the end-of-day baseline value. The batch data collection module addresses the possibility of data loss in asynchronous message components, which could lead to errors in statistical reports, and these errors could gradually increase as the system runs. To solve this problem, this solution is designed to perform a statistical summary of the full amount of data at night to correct existing data. The data synchronization module synchronizes data via file transfer methods (such as NFTs) to ensure 100% data arrival.

[0129] In some embodiments of this application, the aforementioned predetermined time point is typically selected during off-peak business hours, such as nighttime.

[0130] In some embodiments of this application, the aforementioned calculation based on full business data refers to bypassing the incremental data that may be lost due to asynchronous messages, and directly recalculating the precise values ​​of statistical indicators from the source database or backup of the business system by executing complex query statements. The aforementioned data item calibration value represents the absolutely accurate value of the data item at the time of calculation.

[0131] In some embodiments of this application, the step 205 above, "sending the data item calibration value to the report service node to update the end-of-day baseline value", can be specifically implemented through the following steps 205a to 205c.

[0132] Step 205a: The report data processing device determines the relationship between the data date of the data item calibration value and the data date in the statistical report data table.

[0133] Step 205b: If the data dates are the same, the report data processing device will update the end-of-day baseline value to the data item calibration value.

[0134] Step 205c: If the data date of the data item calibration value is greater than the data date in the statistical report data table, the report data processing device will update the end-of-day baseline value to the data item calibration value, clear the daily change value of the data item in the statistical report data table, and update the data date in the statistical report data table to the data date of the data item calibration value.

[0135] In some embodiments of this application, the merging rule for the batch calibration process executed by the report data processing device is as follows: When the date of the data stored in the database is equal to the date of the data in the data synchronization interface, the online synchronization interface accumulates the changes of the day, and the batch synchronization interface replaces the end value of the previous day. When the date of the data stored in the database is less than the date of the data in the data synchronization interface (this scenario is generally sent after 23:59 for the first synchronization operation), the online synchronization interface will add the change value of the day to the end value of the previous day, and then start accumulating the change value of the day from zero, while adding the data date. The batch synchronization interface replaces the end value of the previous day, clears the change value of the day, and adds the data date. When the date of the data stored in the database is later than the date of the data in the data synchronization interface (this scenario is generally a case where the data synchronization from the previous day is congested until 00:00 the next day): the data expires and the synchronized data is discarded.

[0136] Thus, this implementation sets a date determination rule similar to but independent of online updates for the batch calibration process, ensuring that batch calibration data is correctly aligned with online update data on the timeline. Whether overwriting the baseline value of the current day or performing a day-cut overwrite, this rule guarantees the validity of the full batch calculation results and can effectively correct for accumulated errors that may be caused by asynchronous online updates.

[0137] Step 206: The report data processing device responds to the received report query request and calculates the current statistical value as the real-time statistical result based on the daily baseline value and the daily change value of the data item in the statistical report data table.

[0138] In some embodiments of this application, the report data processing device executes the statistical report online query service through the business layer: based on the daily baseline value and the daily change value of data items in the statistical report data table, the current statistical value is calculated as the real-time statistical result. This service is the final service provider of statistical reports, which customizes the statistical data for different business scenarios and displays it to the user.

[0139] In some embodiments of this application, the current statistical value = the end-of-day baseline value + the daily change value of the data item, which distributes the complex statistical calculations to daily updates and periodic calibrations, making the query itself fast and lightweight, and able to meet the requirements of real-time response.

[0140] In some embodiments of this application, step 206 can be specifically implemented by steps 206a to 206f as described below.

[0141] Step 206a: The report data processing device parses the report query request and obtains the target report identifier and query conditions.

[0142] In some embodiments of this application, the report data processing device can parse Hypertext Transfer Protocol Request (HTTP) or Remote Procedure Call Parameters (RPC) call parameters to obtain the report ID (such as the daily personal loan report) and filtering conditions (such as date range, institution, etc.) that the user wants to query.

[0143] Step 206b: The report data processing device retrieves the corresponding format template from the predefined template library based on the target report identifier.

[0144] In some embodiments of this application, the above format template is used to map data dimensions to row identifiers of the output report and to map data items to column identifiers of the output report.

[0145] In some embodiments of this application, the above-mentioned format template may be an XML, JSON file or database configuration that defines which dimensions constitute rows and which data items constitute columns in the final report.

[0146] Step 206c: The report data processing device retrieves all matching key-value pairs from the statistical report data table according to the query conditions.

[0147] Step 206d: For each matching key-value pair, the report data processing device adds the end-of-day baseline value and the daily change value of the data item in the matching key-value pair to obtain the current statistical value corresponding to the matching key-value pair.

[0148] In some embodiments of this application, the query result set is traversed in memory, and the current statistical value = daily baseline value + daily change value of data item is calculated for each row of data.

[0149] Step 206e: The report data processing device uses the format template as the mapping rule to aggregate multiple current statistical values ​​calculated from multiple key-value pairs with the same data dimension but different data items into the same row of data.

[0150] In some embodiments of this application, the aforementioned data dimensions are written into the identifier field of the same row, and the current statistical values ​​corresponding to different data items are filled into different data columns of the same row.

[0151] In some embodiments of this application, the report data processing device groups data according to data dimensions, and then fills the calculation results of different data items within the same group into the corresponding columns defined by the format template. For example, the loan balance and overdue amount of branch A_product 1 are placed in two different columns of the same row record.

[0152] Step 206f: The report data processing device outputs the aggregated row data set as the real-time statistical result.

[0153] In some embodiments of this application, the report data processing device can return aggregated data conforming to a two-dimensional table structure to the front end or the caller in the form of JSON, HTML tables or files.

[0154] In some embodiments of this application, the report data processing device performs a reverse operation on the data processing module. That is, when a data report query occurs, the data processing module reverses the data to restore the unified key-value pair format into the specific report format required by the business party for display.

[0155] In this way, the internal general storage format is reverse-converted into the specific report format required by the business. Data dimensions and data items are mapped to rows and columns of a two-dimensional table through format templates, and column-to-row aggregation operations are performed. This efficiently completes the conversion from a storage model optimized for calculation to a report model optimized for display. This process shields the complexity of the underlying data structure and provides a simple and well-organized data view to upper-layer applications, enabling the query service to quickly and flexibly generate user-visualized reports that meet various customized needs.

[0156] In the report data processing method of this application embodiment, data transmission between the business system and the report service node is achieved through an asynchronous message component. This separates the high-concurrency online transactions and the background report statistics process into two relatively independent stages, effectively avoiding data hotspots and database contention issues caused by directly updating centralized statistical tables, thereby ensuring the processing performance and stability of the core business system. Furthermore, a data collaboration mechanism is constructed through a specific statistical report data table structure that includes a daily baseline value and the daily change value of data items, combined with a periodic batch calibration process. On the one hand, lightweight online asynchronous updates of the daily change value of data items ensure near real-time report data; on the other hand, batch full calculations are performed during off-peak business periods to obtain data item calibration values ​​to cover the daily baseline value, thereby correcting minor errors that may occur throughout the day due to message loss or other reasons, ensuring long-term data accuracy. In addition, this data separation structure allows the real-time update and batch calibration processes to be executed concurrently without blocking each other, achieving true 24 / 7 business continuity. Thus, this solution ensures the real-time performance and high accuracy of statistical report data, guaranteeing the business continuity of the system.

[0157] In some embodiments of this application, the above-described report data processing method further includes steps 302 to 304.

[0158] Step 302: The report data processing device periodically scans the statistical report data table.

[0159] In some embodiments of this application, the report data processing device performs a scanning task periodically (e.g., weekly) by means of a database timed task or an external scheduling system to scan the statistical report data table.

[0160] Step 303: The report data processing device identifies the data dimensions and data item combinations in the statistical report data table that have not been accessed by any report query request within a preset time range.

[0161] In some embodiments of this application, the report data processing device can identify cold data by adding a last access time field to the statistical report data table and updating the field during querying. The scanning task can identify cold data by checking whether the field exceeds a preset threshold (such as 90 days).

[0162] Step 304: The report data processing device marks the data corresponding to the identified data dimension and data item combination as invalid or deletes it from the statistical report data table.

[0163] In some embodiments of this application, the report data processing device performs the following through the data merging module of the framework layer: marking the data corresponding to the identified data dimension and data item combination as invalid or deleting it from the statistical report data table. This is the data cleaning function introduced by the device, which is used to clean up offline statistical report data to prevent invalid or expired data from affecting system stability.

[0164] In some embodiments of this application, the report data processing device can execute a query statement to mark data as soft-deleted, or directly execute a deletion statement to physically delete it.

[0165] In this way, by periodically scanning and cleaning up cold data that has not been accessed for a long time, an automated data lifecycle management mechanism is introduced. This effectively solves the problem of wasted storage space and decreased query performance caused by the accumulation of a large amount of invalid or expired data in statistical report data tables after long-term operation. It can automatically reclaim storage resources and improve the system's self-maintenance capability and long-term operational stability.

[0166] In some embodiments of this application, such as Figure 3 The diagram shown is a schematic of the processing flow of a report data processing method provided in an embodiment of this application. The processing flow shows the entire process of statistical report data from generation, transmission, processing to query, and is mainly divided into three parts: online data stream, batch data stream and query data stream.

[0167] Part 1: Online Data Stream. This process is triggered by business transactions and is used to update statistical report data in near real-time. It includes: online data sender and online data receiver.

[0168] The online data sending end is used for: Statistical report related business: It starts with the online transaction that needs to be statistically analyzed (such as loan disbursement or repayment) that occurs; Transaction data collection: During business processing, the transaction data collection module is triggered to collect statistical data changes related to the transaction. Transaction data transmission for statistical reports: The collected transaction data is transmitted to the transaction data transmission module for statistical reports; Data format conversion: Before being sent, the data undergoes a data format conversion, which transforms the business data into a standard format agreed upon by the system for transmission. The converted data is then sent out. Online data receiving end, used for: The system receives transaction data from statistical reports. After receiving the data, it undergoes a data format conversion to an internal format that facilitates subsequent merging processing. The converted data is then sent to the statistical data merging unit.

[0169] Part Two: Batch Data Stream: This process is triggered by a scheduled task to periodically perform full data calibration to ensure long-term accuracy. It includes a batch data sender and a batch data receiver.

[0170] The batch data sending end is used for: Periodic tasks: The process is initiated by periodic tasks (such as scheduled tasks every day or night); Summary of statistical report data: Recalculate the precise values ​​of statistical indicators based on all business data; Data format conversion; Statistical report data retention database: The summarized data, after being converted into a data format, is stored in the statistical report data retention database. This database may be a temporary storage or a data snapshot library. Subsequently, periodic tasks are performed to collect statistical report data and retain it in the database: for example, collecting statistical report data and retaining calibration data in the database in preparation for sending.

[0171] Batch data receiving unit, used for: Receiving batch data for statistical reports: After the batch calibration data is sent, it is received by the receiving unit of receiving batch data for statistical reports at the receiving end; the received data is sent to the statistical data merging unit.

[0172] Part Three: Data Merging and Querying: This part is responsible for integrating online and batch data and providing services for query requests, including: statistical data merging unit and data report query module.

[0173] The statistical data merging unit is used to: receive online incremental data from transaction data of the received statistical reports and batch calibration data from batch data of the received statistical reports; merge the two types of data according to predefined rules (such as date judgment) (for example, add the online change value to the end-of-day baseline value) to ensure the near real-time performance and final accuracy of the data, and update the final result after merging to the statistical report data retention database.

[0174] The data report query module is used for: Statistical report query service: The process begins with a user initiating a statistical report query service to request and retrieve statistical report data; the query request triggers the statistical report data retrieval module, which retrieves the latest, merged statistical data from the statistical report data retention database; Data format conversion: The raw data obtained from the database will undergo a data format conversion to generate the format required by the front-end report (such as two-dimensional table, chart data, etc.). Data Query and Statistical Report Retrieval: The final converted report data is returned to the user, completing the statistical report query and ending the entire query process.

[0175] As is understandable, this solution divides the processing flow into three parts: online processing, batch processing, and online report query service processing. The following example uses the retail loan system's institutional loan scale statistics business. Node A represents the business end where scale data changes occur, and node B represents the loan scale query service provider.

[0176] The online processing flow is as follows: The user performs a repayment transaction at node A; Calculate the loan amount based on the repayment amount; Transmit the loan amount to the data synchronization module; The data synchronization module formats the data; The loan amount is sent to node B via KAFKA; Node B obtains the loan amount from KAFKA and passes it to the data merging module to merge the statistical report data table.

[0177] The batch processing flow is as follows: Every night, the balances of all accounts across the bank are summarized to obtain the institutional loan balance. The data synchronization module formats the data and saves it in the local database; Wait for all batch processing of statistical report business to complete; Node A generates a file from the statistical report data and sends it to Node B via NFT; Node B parses the file and passes the data to the data merging module to be merged into the statistical report data table.

[0178] The online report query service processing flow is as follows: Users can inquire about loan amounts at node B, the executing agency. The online transaction calls the data processing module to query data and perform reverse operations on the formatted data.

[0179] This solution addresses the impact of data hotspots on the system by using an asynchronous message synchronization mechanism; a unified data transmission format ensures compatibility with different business scenarios; and a 24 / 7 design guarantees business continuity.

[0180] Figure 4 This is a schematic diagram of the structure of a report data processing system provided in an embodiment of this application, such as... Figure 4 As shown, the report data processing system 80 includes a business layer 81 and a framework layer 82. The business layer 81 includes an online data collection module 810, a batch data collection module 811, and a statistical report query service module 812. The framework layer 82 includes a data processing module 820, a data synchronization module 821, and a data merging module 822.

[0181] The online data collection module 810 is used to receive data update requests from the business system, which carry data change values ​​for the target data dimension.

[0182] The aforementioned data processing module 820 is used to convert data update requests into a predefined general key-value pair format to obtain converted key-value pair data, wherein the aforementioned target data dimension and the corresponding data item are combined into a key, and the aforementioned data change value is recorded as a value.

[0183] The aforementioned data synchronization module 821 is used to send key-value pair data to the report service node via an asynchronous message component.

[0184] The aforementioned data merging module 822 is used to update the data in the local statistical report data table of the report service node based on key-value pair data. The statistical report data table includes data dimensions, data items, end-of-day baseline values, daily change values ​​of data items, and data dates.

[0185] The aforementioned batch data collection module 811 is used to calculate the data item calibration value of the target data dimension based on the full amount of business data at a predetermined time point.

[0186] The aforementioned data synchronization module 821 is also used to send the data item calibration value to the report service node via file transfer.

[0187] The aforementioned data merging module 822 is also used to update the end-of-day baseline value in the statistical report data table based on the calibration value of the data item.

[0188] The aforementioned statistical report query service module 812 is used to respond to received report query requests and calculate the current statistical value as a real-time statistical result based on the daily baseline value and the daily change value of the data items in the statistical report data table.

[0189] In some embodiments of this application, the above-mentioned report data processing system further includes a configuration management module, which is used to process a report configuration table. This report configuration table stores the mapping relationship between unique identifiers of different business reports and the data dimension definitions and data item definitions they contain. The data dimension definitions include dimension levels and dimension encoding rules, and the data item definitions include data item names, data types, and calculation relationships. Specifically, the data processing module 820 is used to: query the report configuration table according to the business scenario to which the data update request belongs, and obtain the corresponding data dimension definitions and data item definitions; perform dimension combination on the original business data contained in the data update request according to the dimension encoding rules in the obtained data dimension definitions, generating a dimension combination string as the key; and calculate the data change value conforming to the data type from the original business data contained in the data update request according to the calculation relationship in the obtained data item definitions, and record it as a value, obtaining the converted key-value pair data.

[0190] In some embodiments of this application, the above-mentioned report data processing system further includes a resource cleanup module, which is used to periodically scan the statistical report data table; identify data dimensions and data item combinations in the statistical report data table that have not been accessed by any report query request within a preset time range; and mark the data corresponding to the identified data dimensions and data item combinations as invalid or delete them from the statistical report data table.

[0191] It should be noted that the explanation of the solution involved in the report data processing system 80 can be found in the description of the report data processing method in the above embodiments. To avoid repetition, it will not be repeated here.

[0192] Figure 5 This is a schematic diagram of the structure of a report data processing device provided in an embodiment of this application, as shown below. Figure 5 As shown, the report data processing device 100 includes: a receiving module 101, a processing module 102, and a sending module 103.

[0193] The receiving module 101 is used to receive a data update request from the business system, which carries the data change value of the target data dimension. The processing module 102 is used to convert the data update request into a predefined general key-value pair format to obtain the converted key-value pair data, wherein the target data dimension and the corresponding data item are combined into a key, and the data change value is recorded as a value. The sending module 103 is used to send the key-value pair data to the report service node through an asynchronous message component. The processing module 102 is also used to update the report service based on the key-value pair data. The data in the local statistical report table of the node includes data dimensions, data items, daily baseline values, daily changes in data items, and data dates. The aforementioned processing module 102 is also used to calculate the calibration value of the data item for the target data dimension based on the full amount of business data at a predetermined time point, and send the calibration value of the data item to the report service node to update the daily baseline value. The aforementioned processing module 102 is also used to respond to the received report query request and calculate the current statistical value as the real-time statistical result based on the daily baseline value and daily changes in data items in the statistical report table.

[0194] In some embodiments of this application, the processing module 102 is specifically used to: determine the relationship between the received data date and the data date in the statistical report data table; if the data dates are the same, the received data change value is added to the daily change value of the data item in the statistical report data table; if the received data date is greater than the data date in the statistical report data table, the sum of the daily baseline value and the daily change value of the data item in the statistical report data table is used as the new daily baseline value, and the daily change value of the data item in the statistical report data table is cleared to zero, and the data date in the statistical report data table is updated to the received data date.

[0195] In some embodiments of this application, the processing module 102 is specifically used to: determine the relationship between the data date of the data item calibration value and the data date in the statistical report data table; if the data dates are the same, update the end-of-day baseline value to the data item calibration value; if the data date of the data item calibration value is greater than the data date in the statistical report data table, update the end-of-day baseline value to the data item calibration value, clear the daily change value of the data item in the statistical report data table, and update the data date in the statistical report data table to the data date of the data item calibration value.

[0196] In some embodiments of this application, the processing module 102 is further configured to establish a report configuration table before converting the data update request into a predefined general key-value pair format to obtain the converted key-value pair data. This report configuration table stores the mapping relationship between the unique identifiers of different business reports and the included data dimension definitions and data item definitions. The data dimension definitions include dimension levels and dimension encoding rules, and the data item definitions include data item names, data types, and calculation relationships. Specifically, the processing module 102 is configured to: query the report configuration table according to the business scenario to which the data update request belongs, and obtain the corresponding data dimension definitions and data item definitions; perform dimension combination on the original business data included in the data update request according to the dimension encoding rules in the obtained data dimension definitions, generating a dimension combination string as the key; calculate the data change value conforming to the data type from the original business data included in the data update request according to the calculation relationships in the obtained data item definitions, and record it as a value to obtain the converted key-value pair data.

[0197] In some embodiments of this application, the processing module 102 is specifically used for: parsing the report query request to obtain the target report identifier and query conditions; obtaining the corresponding format template from a predefined template library according to the target report identifier; wherein the format template is used to map the data dimension to the row identifier of the output report and to map the data item to the column identifier of the output report; obtaining all matching key-value pairs from the statistical report data table according to the query conditions; for each matching key-value pair, adding the daily baseline value and the daily change value of the data item in the matching key-value pair to obtain the current statistical value corresponding to the matching key-value pair; using the format template as the mapping rule, aggregating multiple current statistical values ​​calculated from multiple key-value pairs with the same data dimension but different data items into the same row of data; wherein the data dimension is written into the identifier field of the same row, and the current statistical values ​​corresponding to different data items are filled into different data columns of the same row; and outputting the aggregated row data set as the real-time statistical result.

[0198] In some embodiments of this application, the processing module 102 described above is further configured to: periodically scan the statistical report data table; identify data dimensions and data item combinations in the statistical report data table that have not been accessed by any report query request within a preset time range; and mark the data corresponding to the identified data dimensions and data item combinations as invalid or delete them from the statistical report data table.

[0199] It should be noted that the explanation of the report data processing device 100 can be found in the relevant description of the report data processing device in the above embodiments, and will not be repeated here to avoid repetition. The report data processing device provided in this application embodiment can achieve the same effect as the above-described report data processing method. The effect of the report data processing method can be found in the relevant description of the effect of the above-described report data processing method embodiments, and will not be repeated here to avoid repetition.

[0200] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Figure 6 As shown, the electronic device 500 includes: a memory 501, a transceiver 502, and at least one processor 503.

[0201] Transceiver 502 is used to interact with other devices to send and receive data.

[0202] The memory 501 is used to store computer program code, which includes computer instructions. These computer instructions run in the aforementioned electronic device to implement the method shown in the above-described method embodiments. For example, the memory may include high-speed random access memory (RAM), and may also include non-volatile memory (NVM), such as at least one disk storage device, or a USB flash drive, portable hard drive, read-only memory, magnetic disk, or optical disk, etc.

[0203] Processor 503 can be a general-purpose processor, including a Central Processing Unit (CPU), a network processor (NP), etc.; it can also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. Processor 503 can also be other general-purpose processors. The general-purpose processor can be a microprocessor or any conventional processor.

[0204] The memory 501, transceiver 502, and processor 503 are communicatively connected. For example, the memory 501 and transceiver 502 can be connected to the processor 503 via a system bus and communicate with each other. The system bus can be a peripheral component interconnect (PCI) bus, an extended industry standard architecture (EISA) bus, an industry standard architecture (ISA) bus, etc. The system bus can be divided into address bus, data bus, control bus, etc. For ease of representation, only one thick line is used in the figure, but this does not mean that there is only one bus or one type of bus.

[0205] Optionally, the memory 501 can be either standalone or integrated with the processor 503. When the memory 501 is set up independently, it is connected to the processor 503 via a system bus.

[0206] This application also provides a chip for executing instructions, which is used to execute the technical solution of the report data processing method in the above embodiments.

[0207] This application also provides a computer-readable storage medium storing computer instructions. When these computer instructions are executed by a processor, they are used to implement the technical solution of the report data processing method described in the above embodiments. Specifically, when the computer instructions are executed by a processor, the electronic device can perform the technical solution of the report data processing method described in the above embodiments.

[0208] This application also provides a computer program product, which includes a computer program stored in a computer-readable storage medium. At least one processor can read the computer program from the computer-readable storage medium, and when the at least one processor executes the computer program, it can implement the technical solution of the report data processing method in the above embodiments.

[0209] The aforementioned computer-readable storage media can be implemented from any type of volatile or non-volatile storage device or a combination thereof, such as Static Random-Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read-Only Memory (EPROM), Programmable Read-Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The computer-readable storage media can be any available medium accessible to a general-purpose or special-purpose computer.

[0210] An exemplary computer-readable storage medium is coupled to a processor, enabling the processor to read information from and write information to the storage medium. Of course, the computer-readable storage medium can also be a component of the processor. The processor and the computer-readable storage medium can reside in an application-specific integrated circuit (ASIC). Alternatively, the processor and the computer-readable storage medium can exist as discrete components in an electronic control unit or main control device; this application does not limit this.

[0211] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices, or modules, and may be electrical, mechanical, or other forms.

[0212] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to implement the solution of this embodiment according to actual needs.

[0213] Furthermore, the functional modules in the various embodiments of this application can be integrated into one processing unit, or each module can exist physically separately, or two or more modules can be integrated into one unit. The unit composed of the above modules can be implemented in hardware or in the form of hardware plus software functional units.

[0214] The integrated modules described above, implemented as software functional modules, can be stored in a computer-readable storage medium. These software functional modules, stored in a storage medium, include several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute some steps of the methods of the various embodiments of this application.

[0215] It should be understood that the steps of the method disclosed in the embodiments of this application can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules in the processor.

[0216] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0217] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

Claims

1. A method for processing report data, characterized in that, include: Receive a data update request from the business system, the data update request carrying the data change value of the target data dimension; The data update request is converted into a predefined general key-value pair format to obtain the converted key-value pair data, wherein the target data dimension and the corresponding data item are combined into a key, and the data change value is recorded as a value; The key-value pair data is sent to the report service node via an asynchronous message component; Based on the key-value pair data, update the data in the local statistical report data table of the report service node. The statistical report data table includes data dimensions, data items, end-of-day baseline values, daily change values ​​of data items, and data dates. At a predetermined time point, based on the full amount of business data, the calibration value of the data item for the target data dimension is calculated, and the calibration value of the data item is sent to the report service node to update the end-of-day baseline value; In response to a received report query request, the current statistical value is calculated as a real-time statistical result based on the daily baseline value and the daily change value of the data item in the statistical report data table.

2. The report data processing method according to claim 1, characterized in that, The step of updating the data in the local statistical report data table of the report service node based on the key-value pair data includes: Determine the relationship between the received data date and the data date in the statistical report data table; If the data dates are the same, the received data change values ​​will be added to the daily change values ​​of the data items in the statistical report data table; If the date of the received data is greater than the date of the data in the statistical report data table, then the sum of the daily baseline value and the daily change value of the data item in the statistical report data table is used as the new daily baseline value, the daily change value of the data item in the statistical report data table is cleared to zero, and the date of the data in the statistical report data table is updated to the date of the received data.

3. The report data processing method according to claim 1, characterized in that, Sending the data item calibration value to the report service node to update the end-of-day baseline value includes: Determine the relationship between the data date of the calibration value of the data item and the data date in the statistical report data table; If the data dates are the same, then the end-of-day baseline value is updated to the calibration value of the data item; If the data date of the data item calibration value is greater than the data date in the statistical report data table, then the end-of-day baseline value is updated to the data item calibration value, the daily change value of the data item in the statistical report data table is cleared, and the data date in the statistical report data table is updated to the data date of the data item calibration value.

4. The report data processing method according to any one of claims 1 to 3, characterized in that, Before converting the data update request into a predefined general key-value pair format to obtain the converted key-value pair data, the method further includes: Establish a report configuration table, which is used to store the mapping relationship between the unique identifiers of different business reports and the data dimension definitions and data item definitions contained therein. The data dimension definitions include dimension hierarchy and dimension encoding rules, and the data item definitions include data item name, data type and calculation relationship. The step of converting the data update request into a predefined general key-value pair format to obtain the converted key-value pair data includes: Based on the business scenario to which the data update request belongs, query the report configuration table to obtain the corresponding data dimension definition and data item definition; Based on the dimension encoding rules in the obtained data dimension definition, the original business data contained in the data update request is combined by dimension to generate a dimension combination string as the key; Based on the calculation relationship in the obtained data item definition, the data change value that conforms to the data type is calculated from the original business data included in the data update request, and recorded as a value to obtain the converted key-value pair data.

5. The report data processing method according to claim 1, characterized in that, In response to a received report query request, the current statistical value is calculated as a real-time statistical result based on the end-of-day baseline value and the daily change value of the data items in the statistical report data table, including: Parse the report query request to obtain the target report identifier and query conditions; Based on the target report identifier, the corresponding format template is obtained from the predefined template library; wherein, the format template is used to map data dimensions to row identifiers of the output report and to map data items to column identifiers of the output report; Based on the query conditions, retrieve all matching key-value pairs from the statistical report data table; For each matching key-value pair, the daily baseline value and the daily change value of the data item in the matching key-value pair are added together to obtain the current statistical value corresponding to the matching key-value pair. Using the format template as a mapping rule, multiple current statistical values ​​calculated from multiple key-value pairs with the same data dimension but different data items are aggregated into the same row of data; wherein, the data dimension is written into the identifier field of the same row, and the current statistical values ​​corresponding to different data items are filled into different data columns of the same row; The aggregated row data set is output as the real-time statistical result.

6. The report data processing method according to claim 1, characterized in that, The method further includes: The statistical report data table is periodically scanned; Identify the data dimensions and data item combinations in the statistical report data table that have not been accessed by any report query request within a preset time range; The data corresponding to the identified data dimension and data item combination is marked as invalid or deleted from the statistical report data table.

7. A report data processing system, characterized in that, include: The system comprises a business layer and a framework layer. The business layer includes an online data collection module, a batch data collection module, and a statistical report query service module. The framework layer includes a data processing module, a data synchronization module, and a data merging module. The online data collection module is used to receive data update requests from the business system, the data update requests carrying data change values ​​of the target data dimension; The data processing module is used to convert the data update request into a predefined general key-value pair format to obtain the converted key-value pair data, wherein the target data dimension and the corresponding data item are combined into a key, and the data change value is recorded as a value; The data synchronization module is used to send the key-value pair data to the report service node through an asynchronous message component; The data merging module is used to update the data in the local statistical report data table of the report service node according to the key-value pair data. The statistical report data table includes data dimensions, data items, end-of-day baseline values, daily change values ​​of data items, and data dates. The batch data collection module is used to calculate the data item calibration value of the target data dimension based on the full amount of business data at a predetermined time point; The data synchronization module is also used to send the calibration value of the data item to the report service node via file transfer; The data merging module is also used to update the end-of-day baseline value in the statistical report data table according to the calibration value of the data item; The statistical report query service module is used to respond to the received report query request and calculate the current statistical value as the real-time statistical result based on the daily baseline value and the daily change value of the data item in the statistical report data table.

8. The report data processing system according to claim 7, characterized in that, The report data processing system also includes a configuration management module; The configuration management module is used to process the report configuration table. The report configuration table is used to store the mapping relationship between the unique identifiers of different business reports and the data dimension definitions and data item definitions contained therein. The data dimension definitions include dimension hierarchy and dimension encoding rules, and the data item definitions include data item name, data type and calculation relationship. The data processing module is specifically used for: querying the report configuration table according to the business scenario to which the data update request belongs, and obtaining the corresponding data dimension definition and data item definition; combining the dimensions of the original business data contained in the data update request according to the dimension encoding rules in the obtained data dimension definition, and generating a dimension combination string as the key; calculating the data change value that conforms to the data type from the original business data contained in the data update request according to the calculation relationship in the obtained data item definition, and recording it as a value, to obtain the converted key-value pair data.

9. The report data processing system according to claim 7, characterized in that, The report data processing system also includes a resource cleanup module; The resource cleanup module is used to periodically scan the statistical report data table; identify data dimensions and data item combinations in the statistical report data table that have not been accessed by any report query request within a preset time range; and mark the data corresponding to the identified data dimensions and data item combinations as invalid or delete them from the statistical report data table.

10. A report data processing device, characterized in that, include: The receiving module is used to receive data update requests from the business system, wherein the data update requests carry data change values ​​for the target data dimension; The processing module is used to convert the data update request into a predefined general key-value pair format to obtain the converted key-value pair data, wherein the target data dimension and the corresponding data item are combined into a key, and the data change value is recorded as a value; The sending module is used to send the key-value pair data to the report service node via an asynchronous message component; The processing module is also used to update the data in the local statistical report data table of the report service node according to the key-value pair data. The statistical report data table includes data dimensions, data items, end-of-day baseline values, daily change values ​​of data items, and data dates. The processing module is also used to calculate the data item calibration value of the target data dimension based on the full amount of business data at a predetermined time point, and send the data item calibration value to the report service node to update the end-of-day baseline value; The processing module is also used to respond to the received report query request and calculate the current statistical value as a real-time statistical result based on the daily baseline value and the daily change value of the data item in the statistical report data table.

11. An electronic device, characterized in that, include: The electronic device includes a memory and at least one processor; the memory is communicatively connected to the processor; the memory is used to store computer program code, the computer program code including computer instructions; when the processor executes the computer instructions, the electronic device performs the report data processing method as described in any one of claims 1-6.

12. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions, which, when executed by a processor, are used to implement the report data processing method as described in any one of claims 1-6.

13. A computer program product, characterized in that, When the computer program product is run on a computer / executed by the computer's processor, it implements the report data processing method as described in any one of claims 1-6.

Images