Lightweight data model real-time statistical method based on memory
By setting up a statistical model in memory to process the data in real-time, the problem of insufficient I/O speed when traditional databases process large amounts of real-time data is solved, efficient real-time statistics and data analysis are achieved, and hardware resource requirements and maintenance costs are reduced.
Patent Information
- Application Number
- CN202411970814.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-30
- Publication Date
- 2025-04-25
AI Technical Summary
When traditional databases process large amounts of real-time data, the I/O speed of the hard disk cannot meet the requirements, resulting in an increase in data processing delay. In the long-term real-time use, the I/O occupancy status of the hard disk increases, and the I/O needs to be continuously expanded to meet the needs, thereby increasing the cost.
The real-time statistical method of the lightweight data model based on memory is adopted. By batch obtaining raw data and loading it into memory, a statistical model is set in memory to filter, group and aggregate the data, and quickly locate the records in the database using the in-memory index to achieve real-time statistics.
It significantly improves data processing speed, supports real-time data processing and analysis, reduces hardware resource requirements and maintenance costs, and improves the system's response speed and scalability.
Smart Images
Figure CN119884188A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of data statistics, and in particular to a real-time statistics method of a lightweight data model based on memory. Background Art
[0002] In today's digital age, many industries, such as financial monitoring, IoT data processing, and real-time recommendation systems, are facing the need for long-term real-time updates of certain statistical data. These environments usually require the system to respond quickly to data changes and provide accurate and timely statistical results to support key businesses such as decision-making, risk control, and user experience optimization.
[0003] Currently, many systems use traditional database storage technology to meet this demand. The core of this technology is to store data in the database first. When statistical data is needed, the original data of the query object is traversed and filtered in the database through query statements. Subsequently, the raw data output by the database is aggregated and calculated to obtain the required statistical results. This process mainly relies on the storage and processing capabilities of the hard disk.
[0004] Since traditional database technology needs to store and process data on the hard disk, its I / O speed becomes a key factor restricting performance. When a large amount of real-time data needs to be processed, the hard disk's I / O speed often cannot meet the requirements, resulting in increased data processing delays. In addition, when used in real time for a long time, the hard disk's I / O is always occupied. The more historical data that needs to be counted, the more I / O is occupied, and the I / O needs to be continuously expanded to meet the demand, which increases costs.
[0005] As the amount of data continues to grow, traditional databases face challenges in horizontal expansion. This may lead to a decline in statistical performance and fail to meet the business requirements for real-time and accuracy. For complex multi-dimensional analysis queries, traditional databases may require a long query planning and execution time. This not only reduces the system's response speed, but may also affect user experience and the accuracy of business decisions. Traditional databases usually rely on regularly updated materialized views or indexes to provide statistical analysis functions. This method can only be updated periodically, cannot achieve true real-time statistical analysis, and cannot reflect the latest changes in data in a timely manner.
[0006] As the amount of data increases, the maintenance cost of traditional databases (including backup, recovery, optimization, etc.) will also increase accordingly. This not only increases the operating costs of enterprises, but may also affect the stability and reliability of the system. Although mature big data analysis frameworks such as Apache Hadoop and Spark can process large-scale data sets and provide batch processing and complex computing task capabilities, their system architecture is highly complex and requires more hardware resources and complex cluster management. This not only increases maintenance costs, but may also limit the applicability of these frameworks in certain application scenarios. Summary of the invention
[0007] The present invention aims to provide a lightweight data model real-time statistical method based on memory to solve the problem that the existing technology cannot meet the requirements of low hardware resources and efficient real-time statistics when processing batch analysis of large amounts of data.
[0008] To achieve the above object, the present invention adopts the following technical solution: a real-time statistical method of a lightweight data model based on memory, comprising the following steps:
[0009] S1. Acquire raw data in batches and load them into memory, where the raw data is data based on a relational database;
[0010] S2. Setting a statistical model in memory, wherein the statistical model filters the original data, selects data that meets the statistical conditions, and groups the selected data according to the statistical dimensions, generating a unique rowKey for each group of data;
[0011] S3, the statistical model performs aggregation calculation on each group of data according to the statistical direction to obtain statistical results under the corresponding time window and data characteristics, and the statistical results are associated with the corresponding rowKey;
[0012] S4. Set a statistical table in the database to save the statistical results, and detect whether there is a record with the same rowKey in the statistical table; if so, load the statistical data corresponding to the same rowKey in the database into the memory, perform re-aggregation calculation with the current statistical data, and update the statistical data obtained by the re-aggregation calculation into the statistical table; if not, directly save the statistical data and rowKey into the statistical table;
[0013] S5. When statistical data needs to be queried, a rowKey is generated according to the query requirements, the rowKey is used as the unique index of the statistical table to query the statistical data, and the queried statistical data is fed back to the user.
[0014] The principle and advantages of this solution are as follows: in actual application, the original data is acquired in batches and loaded into memory. In memory, a statistical model is set to filter, group, and aggregate the data. This process is completed entirely in memory, so the high-speed access characteristics of memory can be fully utilized. The statistical model filters the original data and selects data that meets the statistical conditions. Then, the selected data is grouped according to the statistical dimensions, and a unique rowKey is generated for each group of data. This rowKey serves as the unique index for subsequent data queries and updates. When statistical data needs to be queried, a rowKey is generated according to the query requirements, and the rowKey is used as the unique index of the statistical table for query. Since the query process directly uses the index in the memory to quickly locate the records in the database, the query speed can be significantly improved.
[0015] Compared with big data framework and SQL, this solution has the following significant advantages:
[0016] High-speed processing: Since data is stored in memory, the limitations of hard disk I / O are avoided, which significantly improves the data processing speed. This is especially important for scenarios where large amounts of data need to be processed and analyzed in real time. Performing aggregate calculations directly in memory is usually much faster than querying the database through SQL, especially when processing large amounts of data. This is because memory access speed is much faster than disk access speed, and it reduces the overhead of database queries. Real-time analysis: It can process and analyze data streams in real time, and supports real-time monitoring and analysis of data streams. It can quickly respond to user query requests and provide real-time statistical results. Resource optimization: Compared with traditional databases and big data frameworks, memory-based solutions require fewer hardware resources. This reduces costs and makes the system easier to maintain and deploy. The system processing capacity can be easily expanded by increasing memory capacity.
[0017] Flexibility and scalability: The data model is designed to be flexible and can be customized according to statistical needs. In addition, the system is easy to expand and can meet the growing data needs. This makes this solution applicable to a variety of scenarios, such as financial monitoring, IoT data processing, and real-time recommendation systems. Consistency and real-time update: The data model design supports real-time update and consistency maintenance. This ensures the accuracy and timeliness of statistical results, allowing the system to provide users with reliable data support. Easy to maintain and deploy: The system architecture is relatively simple, easy to maintain and deploy; compared with the big data framework, the aggregation calculation is completed directly in memory through Java programs, and there is no need to configure and manage complex big data clusters. Especially in resource-limited environments, this solution can significantly reduce maintenance costs and complexity. High concurrency support: Since data processing and query processes are completed in memory, the system can handle high-concurrency requests. This meets the needs of multiple users accessing at the same time and improves the availability and stability of the system. And processing large data sets through in-memory databases or distributed architectures can break through the limitations of memory-based solutions on data scale, so that this solution still has certain advantages when processing large-scale data, and can effectively achieve the needs of low hardware resource requirements and efficient real-time statistics when processing batch analysis of large amounts of data.
[0018] Preferably, as an improvement, in S1, the method for acquiring the original data includes: extracting batches of original data from the relational database or other data source in a self-incrementing ID sequence, or pulling batches of original data from a distributed message queue.
[0019] The beneficial effect of this improvement is that batches of original data are extracted from the database or other data sources in the order of auto-increment IDs. This method is suitable for scenarios where statistical data reports are generated regularly. By writing SQL query statements, a large amount of historical data can be extracted from the database at one time to meet the needs of regular statistical analysis. This method is simple and direct and can ensure the integrity and consistency of the data.
[0020] Pulling batches of raw data from a distributed message queue is suitable for real-time data stream processing or high-concurrency data processing scenarios. Distributed message queues can receive and store data streams in real time. The system can pull data from the queue and process and analyze it in real time. This method can ensure the real-time and accuracy of data and improve the system's response speed and processing capabilities.
[0021] Preferably, as an improvement, the parameter settings of the statistical model include statistical conditions for screening and filtering data, grouping dimension values for data grouping, statistical directions for setting aggregation calculation directions, statistical window types, statistical window lengths, and window statistical granularity.
[0022] The beneficial effects of this improvement are: the statistical model can be flexibly set according to statistical needs, so that the system can perform customized statistical analysis according to different business needs. It is easy to change the configuration. When business needs change, the parameters of the statistical model can be adjusted to adapt to the new needs without large-scale modification and reconstruction of the system.
[0023] Preferably, as an improvement, the statistical window types include sliding time windows and fixed time windows.
[0024] The beneficial effects of this improvement are: Sliding time window: Applicable to application scenarios that require real-time tracking and updating of statistical results, such as real-time monitoring of system performance indicators. Sliding time window can reflect the latest changes in data in real time and provide accurate real-time statistical data. Fixed time window: Applicable to application scenarios that perform statistics and analysis on data within a specific time period, such as daily, weekly and monthly reports. Fixed time window can ensure the integrity and consistency of data and provide accurate statistical data.
[0025] Preferably, as an improvement, the rowKey includes a grouping dimension value and time.
[0026] The beneficial effect of this improvement is that by using the grouping dimension value and time as the rowKey, the uniqueness and queryability of the data can be ensured, which facilitates subsequent data storage and query, and improves the query efficiency and processing capacity of the system.
[0027] Preferably, as an improvement, in S2, the statistical model designs a corresponding data structure according to statistical requirements, and is implemented through a Java program and nested into any system as a component; the design of the statistical model is based on SQL statistical syntax, and includes one or more statistical indicators and logic for data filtering and grouping.
[0028] The beneficial effects of this improvement are: by expanding the memory space, the statistical capacity can be expanded without large-scale hardware upgrades or reconstruction of the system. By setting multiple models or multiple logics, different statistical needs can be flexibly responded to to meet the diversity and variability of the business. The expansion cost is low and the expansion method is simple, which reduces the maintenance cost of the system. The statistical speed is fast, and accurate statistical data can be provided in real time, which improves the response speed and processing capacity of the system. BRIEF DESCRIPTION OF THE DRAWINGS
[0029] Figure 1 It is a schematic structural diagram of an embodiment of the present invention. DETAILED DESCRIPTION
[0030] The following is further described in detail through specific implementation methods:
[0031] Example
[0032] Basically as attached Figure 1 As shown, a real-time statistical method of a lightweight data model based on memory includes:
[0033] S1. Obtain batch raw data, extract batch raw data from a database or other data source in auto-increment ID order, or pull batch raw data from a distributed message queue.
[0034] S2. A statistical model for processing raw data is set up in the memory. The statistical model filters the raw data, selects the data that meets the statistical conditions, and then groups the selected data according to the statistical dimensions to generate a unique rowKey for each group of data.
[0035] S3. The statistical model performs aggregation calculations on each group of data according to the statistical direction to obtain statistical results under the corresponding time window and data characteristics. The statistical results of each group of data correspond to its rowKey.
[0036] S4. A statistical table for storing statistical results is provided in the database to detect whether the same rowKey record already exists in the database; if so, the statistical data corresponding to the same rowKey in the database is loaded into the memory, and then the two statistical data of the same rowKey are aggregated again, and the statistical data obtained by the second aggregation calculation is updated to the statistical table as the corresponding statistical data of the unique rowKey; if there is no same rowKey record, the statistical data and rowKey are directly saved in the database.
[0037] S5. When statistical data needs to be queried, a rowKey is generated according to the grouping dimension and time, and the rowKey is used as the unique index of the statistical table to query the statistical data. The statistical data queried by the rowKey is fed back to the user to provide analysis and decision support.
[0038] Among them, in S1, the original data obtained is data based on a relational database (such as MYSQL); there are two ways to obtain original data: (1) extracting batches of original data from a database or other data source in an auto-incrementing ID sequence, which can be used to regularly generate statistical data reports. By writing SQL query statements, a large amount of historical data can be extracted from the database at one time; (2) pulling batches of original data from a distributed message queue, which can be used for real-time data stream processing or high-concurrency data processing.
[0039] In S2, the corresponding data structure is designed according to the statistical requirements to form a statistical model. The statistical model is implemented as a Java program and can be nested into any system as a component. The design of the statistical model is based on the SQL statistical syntax (e.g., select sum(amount)from table where class='xxx'group by name, time).
[0040] The statistical model includes one or more statistical indicators and logic for data filtering and grouping to meet different statistical needs. The parameter settings of the statistical model include statistical conditions for filtering data, grouping dimension values for data grouping, statistical direction for aggregate calculation, statistical window type, statistical window length, and statistical window granularity.
[0041] Among them, the sliding time window is suitable for application scenarios that require real-time tracking and updating of statistical results, such as real-time monitoring of system performance indicators; the fixed time window is used to count the data of the day. The fixed time window is suitable for application scenarios that require statistics and analysis of data within a specific time period, such as daily, weekly, and monthly reports.
[0042] Statistics window type: Select sliding time window or fixed time window.
[0043] Statistics window length: Set the length of the time window, such as 1 hour, 1 day, etc.
[0044] Window statistics granularity: Set the statistics granularity within the time window, such as by second, by minute, by hour, etc.
[0045] Statistical conditions: Set the conditions for filtering data, such as only counting data of a specific category.
[0046] Grouping dimension value: Set the grouping dimension of the data, such as by user ID, by product category, etc.
[0047] Statistical direction: Set the statistical direction, such as calculating the sum, average, maximum, minimum, etc.; set through aggregation fields or aggregation functions.
[0048] The rowKey generation rule is: grouping dimension value + current timestamp / number of milliseconds corresponding to the window unit granularity. When querying statistical data, the rowKey is generated based on the provided grouping dimension value + time as the database unique index query to efficiently query the relational database.
[0049] To more clearly illustrate the parameter setting and use of the statistical model, an example is given; assuming that the usage scenario is an e-commerce company, and it is necessary to count the purchase amount of each user in the past hour, the parameters of the statistical model can be set as follows:
[0050] Statistical window type: sliding time window.
[0051] Statistics window length: 1 hour.
[0052] Window statistics granularity: by minute (or finer time granularity, such as by second).
[0053] Statistical condition: User ID is within a specific range (or only specific users are counted).
[0054] Grouping dimension value: group by user ID.
[0055] Statistical direction: Calculate the total purchase amount of each user.
[0056] In this way, the system will track and update the purchase amount of each user in the past hour in real time, and store the statistical results in the database. When you need to query the purchase amount of a certain user, you only need to generate a rowKey based on the user ID and time, and find the corresponding statistical data in the database.
[0057] The above is only an embodiment of the present invention, and the common knowledge such as the known specific technical solutions and / or characteristics in the solution is not described in detail here. It should be pointed out that for those skilled in the art, without departing from the technical solution of the present invention, several modifications and improvements can be made, which should also be regarded as the protection scope of the present invention, and these will not affect the effect of the implementation of the present invention and the practicality of the patent. The scope of protection required by this application shall be based on the content of its claims, and the specific implementation methods and other records in the specification can be used to interpret the content of the claims.
Claims
1. A real-time statistical method for a lightweight data model based on memory, characterized in that: The following steps are involved: S1. Acquire raw data in batches and load them into memory, where the raw data is data based on a relational database; S2. Setting a statistical model in memory, wherein the statistical model filters the original data, selects data that meets the statistical conditions, and groups the selected data according to the statistical dimensions, generating a unique rowKey for each group of data; S3, the statistical model performs aggregation calculation on each group of data according to the statistical direction to obtain statistical results under the corresponding time window and data characteristics, and the statistical results are associated with the corresponding rowKey; S4. Set a statistical table in the database to save the statistical results, and detect whether there is a record with the same rowKey in the statistical table; If it already exists, the statistical data corresponding to the same rowKey in the database is loaded into the memory, and the statistical data is re-aggregated and calculated with the current statistical data, and the statistical data obtained by the re-aggregation calculation is updated into the statistical table; If it does not exist, the statistical data and rowKey are directly saved in the statistical table; S5. When statistical data needs to be queried, a rowKey is generated according to the query requirements, the rowKey is used as the unique index of the statistical table to query the statistical data, and the queried statistical data is fed back to the user.
2. The real-time statistical method of a lightweight data model based on memory according to claim 1, characterized in that: In S1, the original data is obtained by extracting batches of original data from the relational database or other data sources in a self-increasing ID sequence, or pulling batches of original data from a distributed message queue.
3. The real-time statistical method of a lightweight data model based on memory according to claim 2, characterized in that: The parameter settings of the statistical model include statistical conditions for screening and filtering data, grouping dimension values for data grouping, statistical directions for setting aggregation calculation directions, statistical window types, statistical window lengths, and window statistical granularity.
4. The real-time statistical method of a lightweight data model based on memory according to claim 3, characterized in that: The statistical window types include sliding time windows and fixed time windows.
5. The real-time statistical method of a lightweight data model based on memory according to claim 4, characterized in that: The rowKey includes a grouping dimension value and time.
6. The real-time statistical method of a lightweight data model based on memory according to claim 5, characterized in that: In S2, the statistical model designs a corresponding data structure according to statistical requirements, and is implemented through a Java program and embedded into any system as a component; the design of the statistical model is based on SQL statistical syntax, and includes one or more statistical indicators and logic for data filtering and grouping.