Method and system for accessing statistical data

Abstract

The application provides a method and system for accessing statistical data. The method comprises: a client aggregating and counting data details obtained in a preset period according to dimensions or data types to obtain statistical data, and sending the statistical data to a server; when the resource usage rate of the client is less than a preset threshold, the client asynchronously sends the data details to the server; the server stores the received statistical data in a first storage layer and stores the received data details in a second storage layer; and the server reads the statistical data or the data details according to a preset reading rule when receiving a query request for the statistical data or the data details. The application reduces the reporting pressure of the client and the storage cost of the server by differentiating the sending of the client and the cold and hot layered storage of the server, and improves the query efficiency of the statistical data.

Description

Technical Field

[0001] This invention relates to the field of data storage and data processing technology, and in particular to methods and systems for storing and retrieving statistical data. Background Technology

[0002] Most systems currently have data statistics functionality. The main implementation involves the client reporting detailed data to the server, either individually or in batches. The server receives the data, processes it into a database, and then aggregates the data through real-time queries or scheduled tasks before displaying it on the page. This approach has several problems: First, the client-side reporting method suffers from resource constraints due to the large volume of detailed data, leading to a backlog of uploaded records. Second, the server-side data storage method is problematic. Storing detailed data directly on a regular hard drive reduces costs but results in low read efficiency. Storing on a solid-state drive (SSD) improves read efficiency but increases costs due to the large data volume. Summary of the Invention

[0003] The technical problem to be solved by this invention is to provide a method and system for storing and retrieving statistical data, so as to reduce the data reporting pressure on the client in application scenarios of client data collection and server data storage in Internet products, and improve the efficiency of statistical query while reducing server storage costs.

[0004] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is as follows: A method for accessing statistical data, comprising: The client summarizes and statistically analyzes the data details obtained within a preset period by dimension or data type to obtain statistical data, and then sends the statistical data to the server. When the client's resource utilization rate is less than a preset threshold, the client asynchronously sends the data details to the server. The server stores the received statistical data in the first storage layer and the received data details in the second storage layer. When the server receives a query request for the statistical data or the data details, it reads the statistical data or the data details according to the preset reading rules.

[0005] To solve the above-mentioned technical problems, another technical solution adopted by the present invention is: a statistical data storage and retrieval system, including a client and a server, wherein the client implements the following steps: The data details obtained within a preset period are summarized and statistically analyzed to obtain statistical data, which is then sent to the server. When the resource utilization rate is less than a preset threshold, the data details are sent to the server asynchronously. The server implements the following steps: The received statistical data is stored in the first storage layer, and the received data details are stored in the second storage layer; when a query request for the statistical data or the data details is received, the statistical data or the data details are read according to a preset reading rule.

[0006] The beneficial effects of this invention are as follows: By prioritizing the sending of statistical data by the client and asynchronously sending of detailed data during resource-idle periods, the reporting pressure on the client is effectively reduced, avoiding the impact of accumulated detailed data uploads on client performance; by storing statistical data in the first storage layer and detailed data in the second storage layer by the server, physical separation of different types of data is achieved, reducing the server's dependence on high-performance storage media and lowering overall storage costs; by reading data according to preset reading rules when the server receives a query request, differentiated reading of statistical data and detailed data is achieved, improving the query response speed of statistical data while ensuring the integrity and traceability of detailed data. Building upon this foundation, end-to-end query efficiency optimization is achieved by combining resource-aware scheduling on the client side with data hot / cold tiering on the server side: the client dynamically adjusts its data sending strategy based on its own resource utilization, prioritizing the sending of statistical data and delaying the sending of detailed data when resources are scarce, ensuring that statistical data arrives at the server with the shortest possible delay, thereby improving the real-time queryability of statistical data; the server performs hot / cold tiered storage based on the access frequency of data, storing frequently accessed statistical data in the high-speed SSD storage layer and storing infrequently accessed detailed data in ordinary disks, and periodically migrating expired statistical data from the hot storage layer to the cold storage layer, maximizing the response speed of high-frequency queries while controlling storage costs. Attached Figure Description

[0007] Figure 1 This is a flowchart illustrating a method for accessing statistical data according to an embodiment of the present invention; Figure 2 This is a schematic diagram of a statistical data storage and retrieval system according to an embodiment of the present invention. Detailed Implementation

[0008] Definitions:

[0009] To explain in detail the technical content, objectives, and effects of the present invention, the following description is provided in conjunction with the embodiments and accompanying drawings.

[0010] In existing technologies, data statistics functions are widely used in various internet products, such as application page click statistics, user behavior analysis, and business operation monitoring. These scenarios typically require collecting detailed data generated by user operations on the client side, reporting the data to the server for storage and aggregation, and finally displaying the statistical results to operations or maintenance personnel through page charts and other forms. With the growth of user scale and business complexity, the amount of data that the client needs to collect and report increases dramatically, leading to problems such as client resource shortages and reporting delays. At the same time, the server needs to store massive amounts of detailed data to support subsequent statistical queries and problem tracing. If all storage media such as solid-state drives are used, storage costs will increase significantly; if all storage media are used, the response speed of statistical queries cannot meet real-time requirements. Therefore, how to achieve efficient access to statistical data under the constraints of limited client resources and high server storage costs has become a common technical challenge in the above-mentioned application scenarios.

[0011] To at least solve the above problems, this invention provides a method for storing and retrieving statistical data. The client first sends the statistical data, and then asynchronously sends the detailed data when resources are idle. The server stores the statistical data in a hot storage layer and the detailed data in a cold storage layer. When querying, the data is read from the corresponding storage layer according to the rules.

[0012] This approach reduces the data reporting pressure on the client side, lowers the storage costs on the server side, and improves the efficiency of querying statistical data.

[0013] The following describes in detail a method for storing and retrieving statistical data according to the present invention. Please refer to [link / reference]. Figure 1 The method 100 includes steps 101 to 104: Step 101: The client summarizes and statistically analyzes the data details obtained within a preset period by dimension or data type to obtain statistical data, and sends the statistical data to the server.

[0014] Specifically, taking page click data statistics as an example, the client collects detailed page click data for each preset period (e.g., 15 seconds). Each detailed data entry contains at least the following fields: device model (e.g., iPhone 14), application (e.g., App name or version), page module (e.g., homepage carousel, settings page button), timestamp, and user ID. The client maintains a hash table or dictionary structure in memory with "device model|application|page module" as the key. For each detailed data entry, the device model, application, and page module fields are extracted and concatenated to form a unique key, and then the counter corresponding to that key is incremented by 1. At the end of the period, the hash table is traversed, and each key and its count are treated as a statistical data entry. The structure of the statistical data can be represented as: {dimension combination: "iPhone 14|MyApp|homepage carousel", count value: 128}, which may also include the start and end timestamps of the period. Finally, the statistical data is serialized and sent to the server.

[0015] Here's an example of summarizing statistics by data type: Assume the data details collected by the client include an "Event Type" field, such as "Click," "Slide," "Long Press," and "Exit." Within a preset period, the client maintains a hash table with "Data Type" as the key, counting the occurrences of each data type. For example, if 150 click events, 80 slide events, 5 long press events, and 2 exit events are obtained within the period, the statistics would be: {"Click": 150, "Slide": 80, "Long Press": 5, "Exit": 2}. If the data type also combines other attributes (such as further distinguishing button IDs under the "Click" type), statistics can be performed using the composite key "Data Type|Button ID".

[0016] This approach avoids reporting large amounts of raw details in real time, reduces frequent interactions between the client and the server, and thus lowers the data reporting pressure on the client.

[0017] Step 102: When the client's resource utilization rate is less than a preset threshold, the client asynchronously sends the data details to the server.

[0018] Specifically, when the CPU utilization, memory utilization, and bandwidth utilization are all less than or equal to 50%, the data details are asynchronously sent to the server according to a preset first time interval; when any one of the CPU utilization, memory utilization, and bandwidth utilization is greater than 50%, the first time interval is expanded by a preset step size to obtain a second time interval, and the data details are asynchronously sent to the server according to the second time interval.

[0019] Specifically, when CPU utilization, memory utilization, and bandwidth utilization are all less than or equal to 50%, the client asynchronously sends data details to the server every 15 seconds; when any of the metrics is greater than 50%, the sending interval is extended to 30 seconds or 1 minute before sending the detailed data.

[0020] In this way, the reporting frequency of detailed data is automatically reduced when client resources are scarce, prioritizing the normal operation of the client's own functions, while utilizing idle periods to complete the final reporting of detailed data, thus balancing data integrity and client performance.

[0021] Step 103: The server stores the received statistical data in the first storage layer and the received data details in the second storage layer.

[0022] Specifically, the server stores the received statistical data to an SSD (first storage layer) and the received data details to a regular disk (second storage layer).

[0023] In this way, high-frequency query statistics can be read at high speed, while large amounts of data details that are accessed infrequently are stored in low-cost media, thereby improving the efficiency of statistical queries while reducing the overall storage cost.

[0024] After the server stores the received statistical data in the first storage layer, it further includes traversing the statistical data stored in the first storage layer according to a preset scanning cycle through a scheduled task; calculating the difference between the traversal time and the storage time of the statistical data, and determining whether the difference exceeds the preset maximum storage time; when the difference exceeds the preset maximum storage time, migrating the statistical data from the first storage layer to the second storage layer through an asynchronous task, and deleting the statistical data in the first storage layer.

[0025] Specifically, the calculation method for the preset maximum storage time is as follows: First, identify the business scenario of the statistical data and its data timeliness requirements. For each type of statistical data, set an "effective attention period" T (unit: days) according to its purpose. For example, the effective attention period for real-time ranking data is 1 day (rankings reset daily), for daily active user statistics on page clicks it is 30 days (operations typically focus on trends over the past month), for annual summary data it is 365 days (used for annual report analysis), and for anomaly monitoring statistics it is 7 days (mainly for troubleshooting recent faults); then set the preset maximum storage time M: M = T + D, Where D is the buffer time (e.g., 7 days), which allows users to still have a query window before the data expires and avoids premature data migration due to delays in the execution of scheduled tasks. The buffer time can be set to twice the scanning cycle of the scheduled task.

[0026] In specific configuration, a configuration file or database table is configured for each statistical business (or each dimension / data type). Fields include the statistical business ID, business name, effective monitoring period T (days), and buffer time D (default 7 days). Administrators can adjust T according to business needs. When the server starts or updates its configuration, it reads the configuration and calculates M. When the server stores statistical data in the first storage layer (hot storage layer), it simultaneously records the storage timestamp S. A scheduled task on the server (e.g., executed daily at 2 AM) iterates through the statistical data in the first storage layer and calculates the difference between the current time C and the storage time S, Diff. Diff = CS (in days) If Diff≥M, the statistical data is marked as to be migrated; then, the statistical data to be migrated is copied from the first storage layer to the second storage layer (cold storage layer) through an asynchronous task. After the copy is successful, the original data in the first storage layer is deleted; for statistical services that do not configure a specific M, the global default value (e.g., 60 days) is used.

[0027] Furthermore, the size of the statistical data can be dynamically adjusted based on its volume: if the statistical data for a certain dimension is too large, the value of M can be reduced to free up hot storage space; conversely, for statistical data that is small and frequently queried, M can be extended. For example, if the effective attention period for page click statistics is T=60 days and the buffer time is D=7 days, then M=67 days. This means that page click statistics stored for more than 67 days will be migrated from the SSD to a regular disk. The server scans the statistical data in the SSD daily using a scheduled task. If a statistical data item has been stored for more than 67 days, it is migrated from the SSD to a regular disk via an asynchronous task, and the original data in the SSD is deleted.

[0028] In this way, the hot storage layer (SSD) always retains the most recent and most frequently accessed statistics, effectively controlling the capacity usage of high-cost storage media, while expired statistics can still be queried through ordinary disks, ensuring the long-term availability of data.

[0029] Step 104: When the server receives a query request for the statistical data or the data details, it reads the statistical data or the data details according to the preset reading rules.

[0030] Specifically, when the server receives a query request for the statistical data from the client, it reads the statistical data from the first storage layer; if the statistical data is not found in the first storage layer, it reads the statistical data from the second storage layer; and then returns the statistical data to the client. When the server receives a query request for the data details from the client, it reads the data details from the second storage layer and returns the data details to the client.

[0031] Specifically, when a user queries statistical data, the server prioritizes reading from the SSD; if the data is not found on the SSD, it reads from the regular disk and returns it. When a user queries detailed data, the server reads directly from the regular disk and returns it.

[0032] In this way, the vast majority of statistical query requests are handled by high-performance SSDs, significantly improving the response speed of page data summary display; while detailed queries access ordinary disks, because they occur infrequently and are used in troubleshooting scenarios, they have minimal impact on user experience, achieving a reasonable balance between query efficiency and cost.

[0033] It also includes setting the corresponding preset maximum storage time based on the dimension or data type of the statistical data.

[0034] Specifically, for page click statistics, the attention period is usually 1-2 months, with a buffer time (e.g., 7 days). Therefore, the preset maximum storage time is the sum of the effective attention period and the buffer time (e.g., 67 days). For other statistical data with shorter or longer timeliness (e.g., the effective attention period for real-time ranking data is 1 day, and the annual summary data is 365 days), the preset maximum storage time can be calculated and set accordingly according to the rule M=T+D.

[0035] In this way, different statistical operations can independently configure expiration strategies according to their own data timeliness characteristics, avoiding the situation where the hot storage layer space is occupied by low-frequency data or data with high timeliness requirements is prematurely migrated to the cold storage layer due to the use of a uniform expiration time.

[0036] It also includes determining the preset period based on the timeliness of the statistical data and the resource utilization rate of the client.

[0037] Specifically, the calculation method for the preset period (i.e., the length of the time window for local aggregation and statistics on the client) is as follows: First, set the basic period P0 (unit: seconds) according to the business scenario. For example, set P0=15 seconds for page click statistics that require near real-time feedback, set P0=30 seconds for user behavior analysis, and set P0=5 seconds for server monitoring indicators. The client monitors its own resource utilization in real time, including CPU utilization Ucpu (percentage), memory utilization Umem (percentage), and network bandwidth utilization Ubw (percentage), and defines a resource stress threshold. For example, when any one of Ucpu, Umem, or Ubw is ≥70%, it is considered resource stressful. Calculate the resource load factor F. F=max(Ucpu / 70,Umem / 70,Ubw / 70), F=1 when all three items are less than 70%, and F>1 when any one item exceeds 70%. Preset period P: P = P0 × ceil(F), ceil(F) represents rounding up. For example, when F=1.2, ceil(F)=2, then P=15×2=30 seconds; when F=2.5, ceil(F)=3, then P=45 seconds. At the same time, a maximum period limit Pmax is set, such as 300 seconds, to prevent the statistical data from becoming outdated due to excessively long periods. The client recalculates P every few periods (e.g., every 5 periods) and dynamically adjusts the duration of subsequent periods. If the resource utilization drops below the threshold, P gradually recovers to P0.

[0038] Another implementation method is to preset multiple levels: define resource utilization levels as low (all three items ≤ 50%), medium (any item between 50% and 70%), and high (any item ≥ 70%). The corresponding period is P=P0 for low level, P=P0×2 for medium level, and P=P0×4 for high level, with a maximum period limit set. For example: the period is 15 seconds for normal client load (CPU=30%, memory=40%, bandwidth=20%, all below 50%); the period is extended to 30 seconds for medium load (CPU=60%, memory=45%, bandwidth=30%, one item greater than 50% but less than 70%); and the period is extended to 60 seconds (15 seconds multiplied by 4) for high load (CPU=85%, memory=70%, bandwidth=65%, any item ≥ 70%). Through the above dynamic adjustment, while ensuring the basic timeliness of statistical data, the frequency of statistics and reporting by the client under high load is reduced, thus reducing the impact on the client's own functions.

[0039] For page click statistics that require near real-time feedback, the calculation is performed dynamically at a preset period. If client resources are consistently strained, the period can be appropriately extended to reduce the frequency of statistics and reporting.

[0040] In this way, a dynamic balance is achieved between the timeliness of statistical data and the operating load of the client, ensuring that operators can see the statistical results in a timely manner while avoiding excessive pressure on client resources.

[0041] It also includes detecting the type of storage device connected to the server. If a solid-state drive (SSD) and a regular disk are detected, the SSD is configured as the first storage layer and the regular disk is configured as the second storage layer.

[0042] Specifically, when the server starts up, it detects the connected storage devices. If both solid-state drives (SSDs) and regular disks are present, the SSD is automatically configured as the first storage layer (hot storage layer), and the regular disk is configured as the second storage layer (cold storage layer). If only a regular disk is present, both the first and second storage layers are set on the same regular disk.

[0043] In this way, the method can adapt to different hardware environments, fully leverage the performance advantages of SSDs when available, and still degrade to run when SSDs are unavailable, thus improving the deployment flexibility and compatibility of the solution.

[0044] As described above, this embodiment achieves a significant reduction in the amount of data reported in real time without information loss by prioritizing local aggregation and statistical reporting of data details by the client within a preset period, thereby reducing the interaction frequency between the client and the server and effectively alleviating the data reporting pressure on the client. By having the client monitor its own resource utilization (CPU, memory, bandwidth) in real time and dynamically adjust the asynchronous sending interval of detailed data, automatically reducing the reporting frequency when resources are scarce and resuming normal reporting when idle, a dynamic balance between client load and data integrity is achieved, preventing detailed reporting from impacting the normal operation of the client. Furthermore, the server stores statistical data and detailed data in layers, with the hot storage layer (SSD) carrying frequently queried statistical data and the cold storage layer... The storage tier (ordinary disk) carries low-frequency access to detailed data, achieving an optimal balance between statistical query efficiency and storage cost, significantly reducing storage costs without sacrificing query performance. By periodically scanning statistical data in the hot storage tier and asynchronously migrating expired data to the cold storage tier, the hot storage tier's storage space is automatically released, ensuring that high-cost storage media always serves the most time-sensitive data, further improving resource utilization efficiency. The server prioritizes querying statistical data from the hot storage tier according to preset read rules, and only reads from the cold storage tier when the data is not found in the hot storage tier. At the same time, detailed queries directly access the cold storage tier, enabling rapid response to hot data, improving the experience of operations and maintenance personnel in viewing statistical reports, and ensuring the traceability of detailed data during troubleshooting.

[0045] Please refer to Figure 2 An embodiment of the present invention provides a statistical data storage and retrieval system 200, including a client 201 and a server 202, wherein the client implements the following steps: The data details obtained within a preset period are summarized and statistically analyzed to obtain statistical data, which is then sent to the server. When the resource utilization rate is less than a preset threshold, the data details are sent to the server asynchronously. The server implements the following steps: The received statistical data is stored in the first storage layer, and the received data details are stored in the second storage layer; when a query request for the statistical data or the data details is received, the statistical data or the data details are read according to a preset reading rule.

[0046] The above description is merely an embodiment of the present invention and does not limit the patent scope of the present invention. Any equivalent modifications made based on the content of the present invention's specification and drawings, or direct or indirect applications in related technical fields, are similarly included within the patent protection scope of the present invention.

Claims

1. A method for storing and retrieving statistical data, characterized in that, include: The client summarizes and statistically analyzes the data details obtained within a preset period by dimension or data type to obtain statistical data, and then sends the statistical data to the server. When the client's resource utilization rate is less than a preset threshold, the client asynchronously sends the data details to the server. The server stores the received statistical data in the first storage layer and the received data details in the second storage layer; When the server receives a query request for the statistical data or the data details, it reads the statistical data or the data details according to the preset reading rules.

2. The method for storing and retrieving statistical data according to claim 1, characterized in that, When the client's resource utilization rate is less than a preset threshold, the client asynchronously sends the data details to the server, including: When CPU utilization, memory utilization, and bandwidth utilization are all less than or equal to 50%, the data details are asynchronously sent to the server according to a preset first time interval.

3. The method for storing and retrieving statistical data according to claim 2, characterized in that, When the client's resource utilization rate is less than a preset threshold, the client asynchronously sends the data details to the server, which also includes: When any one of the CPU utilization, memory utilization, or bandwidth utilization exceeds 50%, the first time interval is expanded by a preset step size to obtain a second time interval, and the data details are asynchronously sent to the server according to the second time interval.

4. The method for storing and retrieving statistical data according to claim 1, characterized in that, After the server stores the received statistical data in the first storage layer, it also includes: The statistical data stored in the first storage layer is traversed by a scheduled task according to a preset scanning cycle. Calculate the difference between the time taken during traversal and the storage time of the statistical data, and determine whether the difference exceeds the preset maximum storage time; When the difference exceeds the preset maximum storage time, the statistical data is migrated from the first storage layer to the second storage layer via an asynchronous task, and the statistical data in the first storage layer is deleted.

5. A method for storing and retrieving statistical data according to claim 4, characterized in that, When the server receives a query request for the statistical data, it reads the statistical data according to preset reading rules, including: When the server receives a query request for the statistical data from the client, it reads the statistical data from the first storage layer; If the statistical data is not found in the first storage layer, then the statistical data is read from the second storage layer; The statistical data is returned to the client.

6. The method for storing and retrieving statistical data according to claim 1, characterized in that, When the server receives a query request for the data details, it reads the data details according to preset reading rules, including: When the server receives a query request for the data details from the client, it reads the data details from the second storage layer and returns the data details to the client.

7. The method for storing and retrieving statistical data according to claim 4, characterized in that, Also includes: Set the corresponding preset maximum storage time based on the dimension or data type of the statistical data.

8. The method for storing and retrieving statistical data according to claim 1, characterized in that, The preset period is determined based on the timeliness of the statistical data and the resource utilization rate of the client.

9. A method for storing and retrieving statistical data according to claim 1, characterized in that, Also includes: The system detects the type of storage device connected to the server. If both solid-state drives (SSDs) and regular disks are detected, the SSD is configured as the first storage layer, and the regular disk is configured as the second storage layer.

10. A statistical data storage and retrieval system, comprising a client and a server, characterized in that, The client implements the following steps: The data details obtained within a preset period are summarized and statistically analyzed to obtain statistical data, which is then sent to the server. When the resource utilization rate is less than a preset threshold, the data details are sent to the server asynchronously. The server implements the following steps: The received statistical data is stored in the first storage layer, and the received data details are stored in the second storage layer; when a query request for the statistical data or the data details is received, the statistical data or the data details are read according to a preset reading rule.

Images