Method for calculating number of events capable of being handled per second for auditing products and related equipment
By designing independent event counters and central summary mechanisms in a multi-process environment, and using the synchronization mechanism to summarize the number of events in each process, the problem of inaccurate EPS statistics in a multi-process environment is solved, and efficient and accurate summary of event counts and system stability is achieved.
Patent Information
- Application Number
- CN202411842116.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-13
- Publication Date
- 2025-05-09
AI Technical Summary
In a multi-process environment, it is difficult for the prior art to accurately count the number of events per second (EPS), and data synchronization is difficult and fault tolerance is poor. Especially when dealing with burst traffic and process failures, data statistics may be inaccurate or even lost.
By designing an independent event counter and a central summary mechanism, each process independently calculates the number of events processed per second, and uses a synchronization mechanism to summarize the results of each process's calculations, and uses FIFO queues and polling to ensure the order and accuracy of the data.
It realizes the summary of the number of events across processes and nodes, ensures the accuracy and real-time statistics, has real-time summary, efficient synchronization and fault-tolerant recovery functions, and improves the system's concurrent processing capabilities and stability.
Smart Images
Figure CN119961015A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of computer technology, and in particular to a method and related equipment for calculating the number of events that can be processed per second for audit products. Background Art
[0002] With the rapid development of big data, network security, log analysis and other fields, the system has an increasing demand for real-time event processing. The traditional single-process event processing model can no longer meet the requirements in large-scale data scenarios. Currently, existing technologies at home and abroad mostly use single-process or multi-threaded methods for event processing. Although these methods can improve processing efficiency to a certain extent, in the face of high-concurrency scenarios, the processing capacity of a single machine is still bottlenecked, making it difficult to cope with the needs of large-volume data processing.
[0003] Internationally, systems such as Apache Kafka and Fluentd use distributed architecture to process and count large-scale event data, but these systems usually require complex cluster architectures, consume large resources, and accurately count the number of events processed per second under multiple processes is still a major technical difficulty. Some domestic monitoring and security systems also try to use multi-process processing mode, but there are still deficiencies in multi-process data synchronization, counter accuracy, system fault tolerance, etc., especially when dealing with sudden traffic and process failures, data statistics may be inaccurate or even lost. Summary of the invention
[0004] In order to effectively solve the problems of inaccurate statistics of the number of events per second (EPS), difficult data synchronization and poor fault tolerance in a multi-process environment, the present invention provides a method and related equipment for calculating the number of processable events per second for audit products.
[0005] In the first aspect, the present invention provides a method for calculating the number of events that can be processed per second for an audit product. When the audit product operates in a multi-process mode, each process independently calculates the number of events processed per second, and a synchronization mechanism is used to aggregate the number of events processed per second calculated by each process to obtain the number of events that can be processed per second for the audit product.
[0006] Furthermore, each process independently calculates the number of events processed per second, specifically including: A first-in-first-out FIFO queue is pre-set; Each process inserts the number of events processed within the preset time length and the current timestamp into the FIFO queue every preset time length; wherein the preset time length is less than 1 second.
[0007] Furthermore, the number of events processed per second calculated by each process is aggregated by the synchronization mechanism to obtain the number of events that can be processed per second by the audit product, which specifically includes: Create a statistics process in advance; The statistical process continuously extracts data from the FIFO queue in a polling manner; wherein the data extracted each time is the number of events at the same timestamp; The statistical process accumulates the number of events at the same timestamp to obtain the number of events that can be processed per second at the current timestamp of the audit product.
[0008] In a second aspect, the present invention provides a system for calculating the number of events that can be processed per second for an audit product, comprising: The calculation module is used for each process to independently calculate the number of events processed per second; The summary module is used to summarize the number of events processed per second calculated by each process using a synchronization mechanism to obtain the number of events that can be processed per second by the audit product.
[0009] Furthermore, the calculation module is specifically used to pre-set a first-in-first-out FIFO queue; each process inserts the number of events processed within the preset time length and the current timestamp into the FIFO queue every preset time length; wherein, the time length is assumed to be less than 1 second.
[0010] Furthermore, the summary module is specifically used to pre-create a statistical process, and use the statistical process to continuously extract data from the FIFO queue in a polling manner; wherein the data extracted each time is the number of events at the same timestamp; and the number of events at the same timestamp is accumulated to obtain the number of events that can be processed per second at the current timestamp of the audit product.
[0011] In a third aspect, the present invention provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method described in the first aspect when executing the program.
[0012] In a fourth aspect, the present invention provides a non-transitory computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the method described in the first aspect.
[0013] Beneficial effects of the present invention: The present invention is suitable for scenarios where multiple servers collaborate to process large-scale data, realizes the aggregation of event quantities across processes and nodes, ensures the accuracy and real-time nature of statistics, has real-time aggregation, efficient synchronization and fault-tolerant recovery functions, and improves the concurrent processing capability and stability of the system. BRIEF DESCRIPTION OF THE DRAWINGS
[0014] Figure 1 One of the flow charts of a method for calculating the number of events that can be processed per second for an audit product provided in an embodiment of the present invention; Figure 2 A second flow chart of a method for calculating the number of events that can be processed per second for an audit product provided by an embodiment of the present invention; Figure 3 A schematic diagram of the structure of a system for calculating the number of events that can be processed per second for an audit product provided by an embodiment of the present invention; Figure 4 A structural block diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION
[0015] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution in the embodiment of the present invention will be clearly described below in conjunction with the drawings in the embodiment of the present invention. Obviously, the described embodiment is a part of the embodiment of the present invention, not all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0016] In a single process, it is relatively easy to count the number of events processed per second by a single process. However, in a high-traffic scenario, the performance of a single process or even a single machine is often insufficient. At this time, multiple processes or multiple servers need to be started to process a large number of events at the same time. How to accurately count and summarize the number of events processed per second by all processes when multiple processes are working at the same time is a problem that the present invention needs to solve. In a multi-process environment, the present invention solves the problem of event number statistics when multiple processes are processed in parallel by designing an independent event counter and a central summary mechanism.
[0017] like Figure 1 As shown, an embodiment of the present invention provides a method for calculating the number of events that can be processed per second for an audit product, comprising the following two steps: S101: When the audit product operates in a multi-process mode, each process independently calculates the number of events processed per second; S102: The number of events processed per second calculated by each process is aggregated by using a synchronization mechanism to obtain the number of events that can be processed per second by the audit product.
[0018] The method provided by the embodiment of the present invention is suitable for multiple servers to collaboratively process large-scale data scenarios, realizes the aggregation of event quantities across processes and nodes, and ensures the accuracy and real-time nature of statistics.
[0019] like Figure 2As shown, the method provided by the embodiment of the present invention mainly includes three stages: event processing process, statistical process to obtain data and statistical process to summarize the number of events: 1. Event processing process: A first-in-first-out (FIFO) queue is pre-set. When each process processes an event, it inserts the number of events processed within the preset time (the preset time is less than 1 second) and the current timestamp into the FIFO queue. The FIFO queue is used to store the real-time processing data of each process to ensure the consistency of data order.
[0020] 2. Statistical process data acquisition: Create a statistical process in advance, and the statistical process uses polling to continuously extract data from the FIFO queue; each time data is extracted, the number of events and their timestamps within the corresponding time period are obtained, that is, the data extracted each time is the number of events at the same timestamp.
[0021] 3. Event quantity summary: The statistical process accumulates the number of events with the same timestamp taken out from the FIFO queue to obtain the number of events that can be processed per second at the current timestamp of the audit product. The number of events per second is updated in real time to ensure that the number of events processed by different processes in the same time period can be accurately summarized.
[0022] The method provided by the embodiment of the present invention can ensure accurate statistics of EPS in high-concurrency scenarios, and has fault tolerance capabilities, thereby ensuring the stability and reliability of the system.
[0023] like Figure 3 As shown, an embodiment of the present invention also provides a system for calculating the number of events that can be processed per second for audit products, including a calculation module and a summary module; Among them, the calculation module is used for each process to independently calculate the number of events processed per second; the summary module is used to use the synchronization mechanism to summarize the number of events processed per second calculated by each process to obtain the number of events that can be processed per second by the audit product.
[0024] Specifically, the calculation module is used to pre-set a first-in-first-out FIFO queue; each process inserts the number of events processed within the preset time length and the current timestamp into the FIFO queue every preset time length; wherein, the time length is assumed to be less than 1 second. The summary module is used to pre-create a statistical process, and use the statistical process to continuously extract data from the FIFO queue in a polling manner; wherein, each extracted data is the number of events at the same timestamp; and the number of events at the same timestamp is accumulated to obtain the number of events that can be processed per second at the current timestamp of the audit product.
[0025] The system provided by the embodiment of the present invention ensures that the number of events in a multi-process environment can be accurately summarized by time period, while maintaining efficient data transmission and real-time statistical capabilities. By improving the efficiency of multi-process parallel processing, reducing system resource usage, and reducing hardware expansion requirements, the operating cost of a large-scale event processing system is reduced, and economic benefits are improved.
[0026] The system provided in the embodiment of the present invention is to implement the above method. The specific functions thereof can be referred to the above method embodiments, which will not be described in detail here.
[0027] Figure 4 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 4 As shown, the electronic device may include: a processor (processor) 401, a communication interface (Communications Interface) 402, a memory (memory) 403 and a communication bus 404, wherein the processor 401, the communication interface 402, and the memory 403 communicate with each other through the communication bus 404. The processor 401 may call the logic instructions in the memory 503 to execute a method for calculating the number of events that can be processed per second for an audit product, the method comprising: when the audit product operates in a multi-process mode, each process independently calculates the number of events processed per second, and the number of events processed per second calculated by each process is summarized by a synchronization mechanism to obtain the number of events that can be processed per second for the audit product.
[0028] In addition, when the logic instructions in the above-mentioned memory 403 are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.
[0029] An embodiment of the present invention also provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer can execute the methods provided by the above-mentioned method embodiments, for example, including: when the audit product operates in multi-process mode, each process independently calculates the number of events processed per second, and uses a synchronization mechanism to summarize the number of events processed per second calculated by each process to obtain the number of events that can be processed per second by the audit product.
[0030] An embodiment of the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the methods provided by the above-mentioned method embodiments are implemented, for example, including: when the audit product operates in a multi-process mode, each process independently calculates the number of events processed per second, and the number of events processed per second calculated by each process is summarized using a synchronization mechanism to obtain the number of events that can be processed per second by the audit product.
[0031] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.
[0032] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims
1. A method for calculating the number of events that can be processed per second for an audit product, characterized in that: When the audit product operates in multi-process mode, each process independently calculates the number of events processed per second, and the number of events processed per second calculated by each process is aggregated using a synchronization mechanism to obtain the number of events that can be processed per second by the audit product.
2. A method for calculating the number of events that can be processed per second for an audit product according to claim 1, characterized in that: Each process independently calculates the number of events processed per second, including: A first-in-first-out FIFO queue is pre-set; Each process inserts the number of events processed within the preset time length and the current timestamp into the FIFO queue every preset time length; wherein the preset time length is less than 1 second.
3. A method for calculating the number of events that can be processed per second for an audit product according to claim 2, characterized in that: The synchronization mechanism is used to aggregate the number of events processed per second calculated by each process to obtain the number of events that can be processed per second by the audit product, specifically including: Create a statistics process in advance; The statistical process continuously extracts data from the FIFO queue in a polling manner; wherein the data extracted each time is the number of events at the same timestamp; The statistical process accumulates the number of events at the same timestamp to obtain the number of events that can be processed per second at the current timestamp of the audit product.
4. A system for calculating the number of events that can be processed per second for audit products, characterized in that: include: The calculation module is used for each process to independently calculate the number of events processed per second; The summary module is used to summarize the number of events processed per second calculated by each process using a synchronization mechanism to obtain the number of events that can be processed per second by the audit product.
5. A system for calculating the number of events that can be processed per second for audit products according to claim 4, characterized in that: The calculation module is specifically used to pre-set a first-in-first-out FIFO queue; each process inserts the number of events processed within the preset time length and the current timestamp into the FIFO queue every preset time length; wherein, the time length is assumed to be less than 1 second.
6. A system for calculating the number of events that can be processed per second for audit products according to claim 5, characterized in that: The summary module is specifically used to pre-create a statistical process, and use the statistical process to continuously extract data from the FIFO queue in a polling manner; wherein the data extracted each time is the number of events at the same timestamp; and the number of events at the same timestamp is accumulated to obtain the number of events that can be processed per second at the current timestamp of the audit product.
7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the method according to any one of claims 1 to 3 is implemented.
8. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 3 is implemented.