Data aggregation optimization method and client

By dynamically orchestrating business statistics rule groups on the client asynchronous threads to perform data aggregation and transmitting the aggregated data to the server, the problem of low data aggregation efficiency in the existing technology is solved, and efficient data statistics and management are achieved.

CN119003575APending Publication Date: 2024-11-22FUJIAN TQ ONLINE INTERACTIVE INC
View PDF 3 Cites 0 Cited by

Patent Information

Application Number
CN202410995427.3
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-24
Publication Date
2024-11-22

AI Technical Summary

Technical Problem

The existing data aggregation method requires storing massive amounts of raw data information, resulting in large consumption of server resources and long-term statistical tasks, which require optimization to improve efficiency.

Method used

The client regularly pulls the original data through asynchronous threads, dynamically orchestrates the business statistics rule group for aggregation operations, and transmits the aggregated data to the server to store, reducing the amount of data storage and processing on the server.

Benefits of technology

It significantly improves the system's concurrent processing capabilities, ensures timely data aggregation, reduces server-side pressure, and improves the statistical efficiency and management efficiency of massive data.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119003575A_ABST
    Figure CN119003575A_ABST
Patent Text Reader

Abstract

The invention discloses a data aggregation optimization method and a client. The method comprises the following steps: starting an application instance and pulling a service statistical rule required by the application instance; regularly pulling a certain amount of original data from a local queue by adopting an asynchronous thread, dynamically arranging the service statistical rules and generating a service statistical rule group, and carrying out aggregation operation on the original data according to the service statistical rule group; and transmitting the aggregated data to a server, so that the server stores the aggregated data to a database. According to the invention, by dynamically arranging the business statistical rule, performing preliminary statistical aggregation on the data according to the business statistical rule corresponding to the application instance, and transmitting the aggregated data to the server, the data storage amount of the server is reduced, so that the processing data amount of the statistical task of the server is reduced, and the overall task efficiency is effectively improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of data aggregation, and in particular to an optimization method and client for data aggregation. Background Art

[0002] At present, whether it is a monitoring system or a statistical system, the data aggregation method they adopt is usually: each application instance collects monitoring information or raw data information on its own node, and passes this information to the server, which records and saves each piece of data in the database. Then, the stored data is aggregated and calculated by executing asynchronous tasks, and the statistics and calculations are summarized and recorded according to different business rules. The statistical results are queried and displayed by a special statistical management platform. However, this data aggregation method requires the storage of massive raw data information, and requires a lot of time and resources for statistical work. Therefore, there is an urgent need for an optimization method and client for data aggregation that can solve the above problems. Summary of the invention

[0003] The technical problem to be solved by the present invention is: to provide an optimization method and client for data aggregation, which can reduce the data storage volume of the server and the amount of data processed by its statistical tasks, and effectively improve the efficiency of the overall task.

[0004] In order to solve the above technical problems, the technical solution adopted by the present invention is:

[0005] A data aggregation optimization method comprises the following steps:

[0006] S1. Start the application instance and pull the business statistics rules required by the application instance;

[0007] S2, using an asynchronous thread to regularly pull a certain amount of raw data from a local queue, dynamically arranging the business statistics rules and generating a business statistics rule group, and performing aggregation operations on the raw data according to the business statistics rule group;

[0008] S3. Transmit the aggregated data to the server, so that the server stores the aggregated data in a database.

[0009] In order to solve the above technical problems, another technical solution adopted by the present invention is:

[0010] A data aggregation optimization client comprises a memory, a processor and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, each step of the above-mentioned data aggregation optimization method is implemented.

[0011] The beneficial effects of the present invention are as follows: the present invention provides an optimization method and client for data aggregation, which starts an application instance and pulls the business statistical rules required by the application instance, and by pulling the required business statistical rules, ensures that the application instance follows the established logic when processing data, thereby realizing accurate statistics and analysis of data information; uses an asynchronous thread to regularly pull a certain amount of original data from a local queue, dynamically arranges the business statistical rules and generates a business statistical rule group, and performs aggregation operations on the original data according to the business statistical rule group, significantly improving the concurrent processing capability of the system, ensuring that data can be aggregated in time, and reducing the amount of data stored on the server; transmits the aggregated data to the server, so that the server stores the aggregated data in a database, which is convenient for centralized and efficient management and statistics of data, and reduces the amount of data to be processed by the statistical tasks of the server by performing aggregation operations locally, effectively improving the statistical efficiency of massive data. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 A flowchart of a data aggregation optimization method according to an embodiment of the present invention;

[0013] Figure 2 A schematic diagram of an optimized client for data aggregation according to an embodiment of the present invention;

[0014] Description of labels:

[0015] 1. An optimized client for data aggregation; 2. Memory; 3. Processor. DETAILED DESCRIPTION

[0016] In order to explain the technical content, achieved objectives and effects of the present invention in detail, the following is an explanation in combination with the implementation modes and the accompanying drawings.

[0017] Please refer to Figure 1 , an embodiment of the present invention provides a method for optimizing data aggregation, comprising the steps of:

[0018] S1. Start the application instance and pull the business statistics rules required by the application instance;

[0019] S2, using an asynchronous thread to regularly pull a certain amount of raw data from a local queue, dynamically arranging the business statistics rules and generating a business statistics rule group, and performing aggregation operations on the raw data according to the business statistics rule group;

[0020] S3. Transmit the aggregated data to the server, so that the server stores the aggregated data in a database.

[0021] From the above description, it can be seen that the beneficial effects of the present invention are: the present invention provides an optimization method and client for data aggregation, which starts an application instance and pulls the business statistical rules required by the application instance, and by pulling the required business statistical rules, ensures that the application instance follows the established logic when processing data, thereby realizing accurate statistics and analysis of data information; uses an asynchronous thread to regularly pull a certain amount of original data from a local queue, dynamically arranges the business statistical rules and generates a business statistical rule group, and performs aggregation operations on the original data according to the business statistical rule group, significantly improving the concurrent processing capability of the system, ensuring that data can be aggregated in time, and reducing the amount of data stored on the server; transmits the aggregated data to the server, so that the server stores the aggregated data in a database, which is convenient for centralized and efficient management and statistics of data, and reduces the amount of data to be processed by the statistical tasks of the server by performing aggregation operations locally, effectively improving the statistical efficiency of massive data.

[0022] Furthermore, the business statistics rules are rules configured in the business statistics rule management platform according to various statistics tasks.

[0023] From the above description, it can be seen that by configuring specific business statistical rules through different statistical tasks, the statistical rule configuration is flexible, and can automatically execute statistical tasks, realize unified management of various statistical tasks, and improve the efficiency and quality of statistical work.

[0024] Furthermore, the business statistics rules are differentiated according to the identification field of the application instance, and the application instance can only pull the business statistics rules corresponding to the identification field.

[0025] From the above description, it can be seen that business statistics rules are distinguished according to the identification field of the application instance, which can realize the accurate extraction of business statistics rules for different application instances and improve the accuracy of business statistics. Different application instances can use different business statistics rules according to their own needs to meet the personalized needs of different application instances for business statistics rules and improve the work efficiency and flexibility of task statistics.

[0026] Furthermore, in step S2, the business statistics rules are dynamically arranged and a business statistics rule group is generated, including:

[0027] It is determined whether there are business statistical rules with the same statistical conditions. If so, the business statistical rules with the same statistical conditions are organized into a business statistical rule group.

[0028] From the above description, it can be seen that arranging business statistical rules with the same statistical conditions into a business statistical rule group can simplify statistical work, avoid unnecessary resource duplication, and improve data processing efficiency. At the same time, each business statistical rule in the business statistical group is independent and can be flexibly adjusted and optimized to meet new business needs.

[0029] Furthermore, the data aggregation operation in step S2 is specifically as follows:

[0030] The statistical result data of each business statistical rule in the same business statistical rule group are aggregated according to the corresponding statistical conditions, and the aggregated statistical result data are sent to the server.

[0031] From the above description, it can be seen that by aggregating the statistical result data of each business statistical rule in the same business statistical rule group according to the corresponding statistical conditions, data integration is achieved, the data storage volume and the number of statistical tasks processed on the server are reduced, thereby efficiently analyzing and processing the data and effectively improving the overall task efficiency.

[0032] Please refer to Figure 2 Another embodiment of the present invention provides a data aggregation optimization method and client, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned data aggregation optimization method steps when executing the computer program.

[0033] The above-mentioned data aggregation optimization method and client of the present invention are suitable for reducing the pressure on the server when counting massive data so as to improve the task processing efficiency. The following is an explanation through specific implementation methods:

[0034] Embodiment 1

[0035] Please refer to Figure 1 , a data aggregation optimization method, comprising the steps of:

[0036] S1. Start an application instance and pull business statistics rules required by the application instance.

[0037] In this embodiment, an application instance is started and the business statistical rules required by the application instance are pulled. By pulling the required business statistical rules, it is ensured that the application instance follows the established logic when processing data, thereby achieving accurate statistics and analysis of data information.

[0038] Furthermore, in this embodiment, the business statistics rules are rules configured according to various statistical tasks in the business statistics rule management platform, so as to realize unified management of various statistical tasks and improve the efficiency and quality of statistical work. The application instance is distinguished according to the identification field of the application instance, and the application instance can only pull the business statistics rules of the corresponding identification field, so as to realize accurate pulling of business statistics rules and improve the accuracy of business statistics. For example, when the project needs to start an application instance for collecting monitoring data or statistical data, the application instance is started by integrating the data aggregation software development kit, and the business statistics rules corresponding to the id are pulled from the business statistics rule management platform according to the id field of the application instance, and statistics are performed on the collected data.

[0039] S2. Use an asynchronous thread to regularly pull a certain amount of original data from the local queue, dynamically arrange the business statistics rules and generate a business statistics rule group, and perform aggregation operations on the original data according to the business statistics rule group.

[0040] In this embodiment, an asynchronous thread is used to periodically pull a certain amount of raw data from a local queue, dynamically arrange the business statistics rules and generate a business statistics rule group, and dynamically and personalizedly set various business statistics rules to achieve specific data statistics for different businesses; the raw data is aggregated according to the business statistics rule group, which significantly improves the concurrent processing capability of the system, ensures that data can be aggregated in a timely manner, and reduces the amount of data storage on the server side.

[0041] Furthermore, in this embodiment, the business statistics rules are dynamically arranged and a business statistics rule group is generated, including: judging whether there are business statistics rules with the same statistical conditions, if so, arranging the business statistics rules with the same statistics into a business statistics rule group, for example, rule 1: counting the number of positions of each person within a period of time; rule 2: counting the number of menus clicked by each person within a period of time, both business statistics rules need to perform data statistics based on the account id and have the same statistical conditions, therefore, the two business statistics rules are arranged into a business statistics rule group. In addition, the business statistics rules can also be automatically generated by a computer running a rule generation program, which is suitable for a server with a relatively unified statistical method, for example, counting the number of menu clicks, column clicks and other quantities of each person every day, etc. These statistical tasks are all achieved by querying a unified or relatively standardized library table, and the rule generation program can identify these library tables and derive statistical rules according to a preset established statistical method.

[0042] Furthermore, in the present embodiment, the data aggregation operation in the step S2 is specifically as follows: the statistical result data of each business statistical rule in the same business statistical rule group are aggregated according to the corresponding statistical conditions. For example, the above-mentioned rule 1 and rule 2 are arranged into a business statistical rule, and finally a plurality of result data are generated after aggregation according to the account id. The result data includes three parts: account id, number of locations, number of menus, and the aggregated statistical result data is sent to the server. The data aggregation operation is used to reduce the data storage capacity and the number of statistical tasks to be processed on the server, thereby efficiently realizing the analysis and processing of the data and effectively improving the overall task efficiency.

[0043] S3. Transmit the aggregated data to the server, so that the server stores the aggregated data in a database.

[0044] In this embodiment, the aggregated data is transmitted to the server so that the server stores the aggregated data in a database, which facilitates centralized and efficient data management and statistics. By performing aggregation operations locally, the amount of data to be processed by the server's statistical tasks is reduced, thereby effectively improving the statistical efficiency of massive data.

[0045] Embodiment 2

[0046] Please refer to Figure 2 A data aggregation optimization client 1 includes a memory 2, a processor 3, and a computer program stored in the memory 2 and executable on the processor 3. When the processor 3 executes the computer program, each step of a data aggregation optimization method of embodiment 1 is implemented.

[0047] In summary, the present invention provides a data aggregation optimization method and client, which starts an application instance, and pulls the business statistical rules of the corresponding identification field from the business statistical rule management platform according to the identification field of the application instance, and pulls accurate business statistical rules through the identification field to ensure that the application instance follows the established logic when processing data, thereby realizing accurate statistics and analysis of data information, and configuring different business statistical rules for different statistical tasks to realize unified management of various statistical tasks and improve the efficiency and quality of statistical work; using an asynchronous thread to regularly pull a certain amount of original data from a local queue, dynamically arrange the business statistical rules, and determine whether there are the same statistical conditions. If so, the business statistical rules with the same statistical conditions are arranged into a business statistical rule group, and the original data is aggregated according to the business statistical rule group. Through the data aggregation operation, the data storage volume of the server and the number of statistical tasks to be processed are reduced, thereby efficiently realizing the analysis and processing of the data and effectively improving the overall task efficiency; the aggregated data is transmitted to the server, so that the server stores the aggregated data in a database, which is convenient for centralized and efficient data management and statistics, and improves the overall task efficiency.

[0048] The above descriptions are merely embodiments of the present invention and are not intended to limit the patent scope of the present invention. Any equivalent transformations made using the contents of the present invention's specification and drawings, or directly or indirectly applied in related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A data aggregation optimization method, characterized in that: Includes steps: S1. Start the application instance and pull the business statistics rules required by the application instance; S2, using an asynchronous thread to regularly pull a certain amount of raw data from a local queue, dynamically arranging the business statistics rules and generating a business statistics rule group, and performing aggregation operations on the raw data according to the business statistics rule group; S3. Transmit the aggregated data to the server, so that the server stores the aggregated data in a database.

2. The data aggregation optimization method according to claim 1, characterized in that: The business statistics rules are rules configured in the business statistics rule management platform according to various statistics tasks.

3. The data aggregation optimization method according to claim 2, characterized in that: The business statistics rules are distinguished according to the identification field of the application instance, and the application instance can only pull the business statistics rules corresponding to the identification field.

4. The data aggregation optimization method according to claim 1, characterized in that: The step S2 dynamically arranges the business statistics rules and generates a business statistics rule group, including: It is determined whether there are business statistical rules with the same statistical conditions. If so, the business statistical rules with the same statistical conditions are organized into a business statistical rule group.

5. The data aggregation optimization method according to claim 1, characterized in that: The data aggregation operation in step S2 is specifically as follows: The statistical result data of each business statistical rule in the same business statistical rule group are aggregated according to the corresponding statistical conditions, and the aggregated statistical result data are sent to the server.

6. A data aggregation optimization client, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the following steps are implemented: S1. Start the application instance and pull the business statistics rules required by the application instance; S2, using an asynchronous thread to regularly pull a certain amount of raw data from a local queue, dynamically arranging the business statistics rules and generating a business statistics rule group, and performing aggregation operations on the raw data according to the business statistics rule group; S3. Transmit the aggregated data to the server, so that the server stores the aggregated data in a database.

7. The data aggregation optimization client according to claim 6, characterized in that: The business statistics rules are rules configured in the business statistics rule management platform according to various statistics tasks.

8. The data aggregation optimization client according to claim 7, characterized in that: The business statistics rules are distinguished according to the identification field of the application instance, and the application instance can only pull the business statistics rules corresponding to the identification field.

9. The data aggregation optimization client according to claim 6, characterized in that: The step S2 dynamically arranges the business statistics rules and generates a business statistics rule group, including: It is determined whether there are business statistical rules with the same statistical conditions. If so, the business statistical rules with the same statistical conditions are organized into a business statistical rule group.

10. The data aggregation optimization client according to claim 6, characterized in that: The data aggregation operation in step S2 is specifically as follows: The statistical result data of each business statistical rule in the same business statistical rule group are aggregated according to the corresponding statistical conditions, and the aggregated statistical result data are sent to the server.

Citation Information

Patent Citations

  • Method for strategy processing and network equipment

    CN103688489A

  • Data processing method and device, electronic equipment and storage medium

    CN113190411A

  • Method for Analyzing Transaction Traces to Enable Process Testing

    US20090019428A1