Efficient data batch processing method and system based on memory

Abstract

The invention discloses an efficient data batch processing method and system based on a memory. The system comprises a memory management module, a concurrent processing module, a data flow control module, a data processing module and a monitoring and tuning module. The memory management module allocates a buffer area and a temporary array area in a memory to form a batch queue core storage area; the server stores request data; the concurrent processing module is used for performing parallel service processing on a plurality of service units in a high-concurrency scene; the data flow control module is used for effectively controlling the speed difference between data generation and consumption, keeping a balance state, and setting in a rejection strategy and dynamic adjustment during unbalance; the data processing module is used for selecting an optimal processing strategy and algorithm according to different types of data and business requirements; and the monitoring and tuning module is used for monitoring various indexes of the system and performing health examination, and dynamically adjusting system parameters by analyzing abnormal index data.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing, and in particular to a memory-based efficient data batch processing method and system. Background Art

[0002] With the rapid development of information technology, big data processing has become an indispensable part of all walks of life. Traditional data processing methods are often limited by memory size, processing speed and concurrency, and it is difficult to meet the needs of application scenarios with high real-time performance and large data volumes. Therefore, how to design an efficient and reliable data batch processing method has become a technical problem that needs to be solved urgently. In a high-concurrency business system, a single service needs to process hundreds or thousands of request data per second. If the system involves operating a database or caching middleware Redis, etc., each piece of data is processed separately, and each request will generate a network IO. 1,000 pieces of data will generate 1,000 network IOs, and a lot of time will be wasted on network IO. A single processing involves multiple queries to the database or cache, and the time spent on network IO will also increase exponentially, and the system's concurrency will be affected. If these 1,000 pieces of data can be merged into a group, and this group of data can be batched to operate the database or cache, the number of network requests will be greatly reduced, and the system's concurrent processing capabilities will be improved. How to achieve batch task processing, usually we can achieve it in the following ways:

[0003] The first method uses a distributed message framework (such as RabbitMQ, Kafka) to implement it. By taking advantage of the efficient batch-pull feature of the message framework, business data is classified and placed in different message queues. Consumers pull messages in batches at regular intervals for consumption, thereby achieving the effect of batch data processing. The advantage of using this method is that data placed in a third-party queue is not easily lost. When encountering high-concurrency requests, it can play a peak-cutting role and avoid system memory overflow. However, accessing a third-party message queue also means an increase in technical costs and application operating costs, and the complexity of the system will also increase. Developers need to be proficient in the use of related queues. Once a problem occurs with the message framework, the entire system will be unusable.

[0004] The second method is to store the data in a database or write it into a file according to the business classification, and then obtain the data in batches from the database or file to achieve the effect of batch classification of a large amount of data. Using this method, the data can be stored in the database or file without being lost. The data has records that can be queried and traced, and the original data can be queried and processed multiple times. However, this method is suitable for application scenarios where the real-time requirements of data are not high and the processing can be delayed for a period of time, such as processing the data generated the day before on the second day. It is not applicable to business scenarios where the real-time requirements are at the second level.

[0005] In response to the problems of the above two methods, the following patents provide their own solutions. The following is an analysis of the technical solutions adopted by these patents:

[0006] "Real-time event processing system and method based on cloud computing in computer software system", patent number CN201210407676.3, the present invention relates to a real-time event processing system and method based on cloud computing in computer software system, belonging to the field of computer software application technology. The system includes a SEDA distributed message framework and a Redis core memory processing framework. The method using the system includes a message receiving operation and a message sending operation. Through the method, the distributed message framework and the Redis core memory processing framework are integrated, so that high concurrent requests and large data volume processing requirements can be effectively responded to, complex business logic processing procedures can be realized, and high throughput and low latency performance requirements can be met. In addition, system resources can be effectively controlled, and the reliability of the operation process can be improved, so as to achieve the goal that performance can be designed in sections and can be freely scalable within a certain range. It is suitable for application in business activity monitoring or quasi-real-time analysis systems with demanding performance requirements, and the system and method for implementing real-time event processing based on cloud computing in the computer software system of the present invention have a simple system structure and low application cost.

[0007] The above invention mainly utilizes SEDA distributed message framework and Redis core memory processing framework to perform message receiving and sending operations. The design has high real-time performance and can meet the processing requirements of high concurrency and big data. However, this method requires the integration of Redis and distributed message framework at the same time, which increases the complexity of the system and the difficulty of system stability. Any problem with Redis and message framework will cause system failure and cannot be used. The skill requirements for developers are also high, and they need to be familiar with Redis and message framework at the same time. At the same time, a large amount of data requires the message framework and Redis to have high read and write performance, and the corresponding use cost is also high, which is not friendly to low-cost and low-budget system design.

[0008] "Batch data processing method, front-end system, host and batch data processing system", patent number CN201610754742.2, the present invention provides a batch data processing method, front-end system, host and batch data processing system, the method includes: obtaining pending transaction data submitted by users; sorting the pending transaction data according to the processing type of the pending transaction data, extracting the transaction data that needs to be processed by the host; temporarily storing the transaction data that needs to be processed by the host, and generating an unprocessed transaction record with the primary key as the user transaction ID; dequeuing the temporarily stored transaction data in a first-in-first-out manner, and aggregating the dequeued transaction data and sending them to the host in batches; obtaining the transaction processing results generated by the host according to the transaction data, and updating the status of the corresponding unprocessed transaction records according to the transaction processing results, and feeding back to the user at the same time. The present invention can solve the system bottleneck problem caused by high transaction concurrency, reduce the pressure of network bandwidth and improve the data processing efficiency, improve the stability of the system when data concurrency bursts, and ensure production safety.

[0009] The above invention mainly places the function of batch data classification on the initiator of the business request, that is, the front end to process it, and the front end sends the classified batch data to the host for batch processing. This method can effectively reduce the network bandwidth pressure in high-concurrency scenarios, put the logic of batch classification in the front-end system for processing, and reduce the operating pressure of the host. This method is tried in business scenarios where there are multiple requests from the same user, and multiple data belong to one person, and batch data for the same person can be formed on the front end. However, for business scenarios with multiple requests from multiple users, multiple request data are still sent to the host, and the data has strong isolation, does not reduce the network bandwidth, and the data processing efficiency is also improved.

[0010] "Data batch processing method, device, computer equipment and storage medium", patent number CN201811213241.9, the present invention discloses a data batch processing method, device, computer equipment and storage medium, the method includes: obtaining a table to be imported, the table to be imported includes a header field and basic data; parsing the table to be imported, reading the header field of the table to be imported; obtaining the entry field of the data pool, establishing a mapping relationship between the header field and the entry field, and storing the header field, the entry field and the mapping relationship in the mapping relationship table; based on the mapping relationship, entering the basic data corresponding to the header field into the corresponding entry field in the data pool to obtain the entry data. By establishing a mapping relationship between the header field in the table to be imported and the entry field in the data pool, the basic data of the header field is entered into the corresponding entry field in the data pool, so that the table to be imported can be dynamically adaptive when entering the database, which is more flexible and convenient, so that the table to be imported and the import database are adaptively associated, thereby improving the efficiency of data batch processing.

[0011] The above invention mainly uses table files to store data, uses the mapping relationship between the database storage table header and the input field, relies on the mapping relationship to write the field basic data into the data pool field, and realizes the batch conversion of data from table files into data that can be written into the database. However, this invention is mainly suitable for business scenarios where data files already exist and the data in the data files need to be batch acquired and processed. Similarly, the process of batch acquiring data also realizes the process of batch splitting of data, at which time the business logic of batch computing and processing of data can be performed. Summary of the invention

[0012] The purpose of the present invention is to provide an efficient memory-based data batch processing method and system, aiming to solve the technical problems of performance bottlenecks and stability challenges faced by existing data processing technologies when processing large-scale data.

[0013] In order to achieve the above object, the present invention adopts the following technical solutions:

[0014] An efficient data batch processing system based on memory, comprising: a memory management module, a concurrent processing module, a data flow control module, a data processing module, and a monitoring and tuning module;

[0015] The memory management module allocates a buffer and a temporary array area in the memory to form a batch queue core storage area for storing request data;

[0016] The concurrent processing module is used to handle high-concurrency scenarios where multiple business units perform business processing in parallel;

[0017] The data flow control module is used to effectively control the speed difference between data generation and consumption, maintain a balanced state, and has built-in rejection strategies and dynamic adjustments when imbalance occurs;

[0018] The data processing module is used to select the optimal processing strategy and algorithm according to different types of data and business requirements;

[0019] The monitoring and tuning module is used to monitor and perform health checks on various system indicators, and dynamically adjust system parameters by analyzing abnormal indicator data.

[0020] In some embodiments, the memory management module uses the ConcurrentHashMap efficient data structure to reduce memory access conflicts and delays, pre-allocates and manages memory resources through memory pool technology, and reduces the overhead caused by frequent memory allocation and release operations.

[0021] In some embodiments, the concurrent processing module decomposes the data processing task into multiple subtasks that can be executed in parallel through multi-threading or multi-process technology, and assigns them to different processing units for simultaneous execution.

[0022] In some embodiments, the concurrent processing module adopts a lock-free or low-lock concurrency control strategy, atomic operation-based updates, and read-write lock-based access control.

[0023] In some embodiments, the data flow control module introduces a buffer queue ArrayBlockingQueue as a buffer area between producers and consumers to balance the speed difference between data production and consumption.

[0024] In some embodiments, when the buffer is close to being full, the data flow control module automatically adjusts the producer's rate to prevent data overflow and system overload.

[0025] In some embodiments, the data processing module can automatically select the optimal processing strategy and algorithm according to different types of data and business requirements; and divide a large amount of data into multiple small batches for processing through batch processing mode.

[0026] In some embodiments, the monitoring and tuning module provides users with an intuitive view of the system operation status by monitoring system performance indicators in real time; and provides a parameter adjustment interface to allow users to manually or automatically adjust system parameters according to actual needs.

[0027] This embodiment also provides an efficient memory-based data batch processing method, which is implemented based on the system and includes the following steps:

[0028] A batch queue core storage area is formed by allocating a buffer and a temporary array area in the memory to store the request data;

[0029] Based on the concurrent processing module to handle high-concurrency scenarios, multiple business units perform business processing in parallel;

[0030] Effectively control the speed difference between data generation and consumption to maintain a balanced state, with built-in rejection strategies and dynamic adjustments when imbalance occurs;

[0031] Select the best processing strategy and algorithm based on different types of data and business requirements; split large amounts of data into multiple small batches for processing through batch processing mode;

[0032] Monitor and perform health checks on various system indicators, and dynamically adjust system parameters by analyzing abnormal indicator data.

[0033] The present invention provides a memory-based efficient data batch processing method and system with beneficial effects including but not limited to:

[0034] The memory-based efficient data batch processing method and system of the present invention can greatly improve the speed and throughput of data processing compared to traditional data processing methods, especially when processing large-scale data sets. Refined memory management and resource scheduling strategies ensure that the system can maintain stable performance under high load, while reducing unnecessary resource waste and improving resource utilization. The introduced back pressure mechanism, exception handling and fault tolerance mechanism, as well as comprehensive monitoring and tuning functions, together constitute a powerful system protection barrier. It can effectively reduce the risk of system crash and data loss, and ensure the continuity and reliability of the data processing process. A flexible monitoring and tuning mechanism is provided to facilitate users to optimize the system according to actual needs. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 This is the batch processing procedure for this application.

[0036] Figure 2 This is a flow chart of the core components of a batch queue.

[0037] Figure 3 Flowchart of health check and back pressure mechanism.

[0038] Figure 4 This is a flow chart of the dynamic adjustment mechanism.

[0039] Figure 5 Schematic diagram of spin and non-spin lock mechanism. DETAILED DESCRIPTION

[0040] In order to make the purpose, technical solutions and advantages of the present application clearer, the technical solutions in the embodiments of the present application will be described in more detail below in conjunction with the drawings in the preferred embodiments of the present application. In the drawings, the same or similar reference numerals throughout represent the same or similar parts or parts with the same or similar functions. The described embodiments are part of the embodiments of the present application, not all of the embodiments. The embodiments described below with reference to the drawings are exemplary and are intended to be used to explain the present application, and should not be construed as limitations on the present application. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present application.

[0041] The following will be combined Figure 1-5 It is worth noting that the following embodiments are only used to explain the present application and do not constitute a limitation on the present application.

[0042] Example

[0043] The core of the invention is to propose an innovative memory-based efficient data batch processing method and system, aiming to solve the performance bottlenecks and stability challenges faced by existing data processing technologies when processing large-scale data, improve the system's concurrency and stability, and achieve a significant improvement in data processing capabilities.

[0044] The present invention includes a memory management module, a concurrent processing module, a data flow control module, a data processing module, and a monitoring and tuning module. The memory management module allocates a buffer and a temporary array area in the memory to store the request data, which constitutes the core storage area of ​​the batch queue. The concurrent processing module is designed to handle high-concurrency scenarios in which multiple business units can perform business processing in parallel, aiming to improve the parallel capability of the system. The data flow control module effectively controls the speed difference between data generation and consumption, maintains a balanced state, and has a built-in rejection strategy and dynamic adjustment function when imbalanced. The monitoring and tuning module can monitor and perform health checks on various system indicators, and dynamically adjust system parameters by analyzing abnormal indicator data.

[0045] The present invention introduces a refined memory management strategy, which not only uses an efficient data structure such as ConcurrentHashMap to reduce memory access conflicts and delays, but also pre-allocates and manages memory resources through memory pool technology, effectively reducing the overhead caused by frequent operations of memory allocation and release. At the same time, it fully utilizes the thread-safe characteristics of ConcurrentHashMap to avoid the problem of data insecurity introduced in high-concurrency scenarios. This strategy ensures that the system can still maintain stable memory usage efficiency under high load, avoiding performance degradation caused by memory fragmentation or shortage.

[0046] In order to make full use of the computing power of multi-core processors, the present invention designs an efficient concurrent processing mechanism. Through multi-threading or multi-process technology, the data processing task is decomposed into multiple subtasks that can be executed in parallel, and assigned to different processing units for simultaneous execution. At the same time, in order to reduce the synchronization overhead between threads, the present invention adopts a lock-free or low-lock concurrency control strategy, based on atomic operation updates and access control based on read-write locks, thereby significantly improving the concurrent processing capability and response speed of the system while ensuring data consistency.

[0047] In order to deal with the speed mismatch problem that may occur during data processing, the present invention designs a flexible data flow control mechanism. By introducing the buffer queue ArrayBlockingQueue as a buffer area between producers and consumers, the speed difference between data production and consumption is effectively balanced. At the same time, a back pressure mechanism is also implemented. When the buffer is close to full load, the producer's rate is automatically adjusted to prevent data overflow and system overload. This mechanism ensures the smooth progress of the data processing process and improves the stability and reliability of the system.

[0048] The present invention adopts intelligent data processing logic, which can automatically select the optimal processing strategy and algorithm according to different types of data and business requirements. By dividing a large amount of data into multiple small batches for processing through batch processing mode, it not only reduces the amount of data processed at a single time, reduces the pressure of memory usage, but also improves the flexibility and scalability of data processing. At the same time, for scenarios with high real-time requirements, it also combines stream processing technology to realize data processing while receiving, further improving the real-time response capability of the system.

[0049] In order to ensure that the system can still maintain stable and efficient operation under high load, the present invention integrates a comprehensive monitoring and tuning mechanism. By real-time monitoring of system performance indicators (such as throughput, latency, memory usage, CPU usage, etc.), an intuitive system operation status view is provided to the user. At the same time, a flexible parameter adjustment interface is also provided, allowing the user to manually or automatically adjust system parameters (such as the number of threads, buffer size, etc.) according to actual needs to achieve optimal performance. Taking into account the various abnormal situations that may occur during the data processing process, the present invention designs a robust exception handling and fault tolerance mechanism. Through a comprehensive exception capture and processing strategy, it is ensured that the failure of a single data processing task will not affect the stable operation of the entire system. At the same time, a retry mechanism and a data rollback mechanism are introduced to further improve the fault tolerance and reliability of the system.

[0050] This application is based on efficient memory-based data batch processing. Different business queues are initialized according to the business type and stored in the ConcurrentHashMap mapping table. The Key is the business type and the Value is the batch queue. Each business queue has a corresponding thread pool to asynchronously consume the batch queue data. When a business request comes in, it will be classified according to the business, and the queue will be obtained from the mapping table according to the business type, and the business data will be added to the queue. When any of the capacity listeners or time listeners in the queue is met, the data in the queue will be handed over to the threads in the thread pool for business processing. Since the business data is merged, the database, cache, and computing unit can be accessed and operated in batches in the business processing unit, which reduces the access frequency, reduces the request pressure, and greatly improves the parallel processing capability. It solves the performance bottlenecks and stability challenges faced by existing data processing technologies when processing large-scale data, and improves the concurrency and stability of the system. The batch business processing process is as follows: Figure 1 shown.

[0051] In this application, the batch queue is composed of several parts: a memory allocator, a buffer, a temporary data storage area, and a listener. When the system starts, the memory allocator will allocate memory space for the buffer (ArrayBlockingQueue) and the temporary data storage area. The space can be dynamically adjusted according to the health monitoring of the system. The buffer listener is used to monitor whether there is data in the buffer. If so, it determines whether there is still space in the array of the temporary data storage area. If so, it is placed in the temporary array. If so, it is temporarily stored in the cache area and placed in the temporary array when there is space in the temporary array. The temporary array is monitored by a time listener and a capacity listener. When the time listener detects that the timer has reached the configured maximum time limit, no matter how many data items are in the array, they are handed over to the threads in the thread pool for execution. When the capacity listener detects that the capacity has reached the maximum capacity limit, the data in the array is also handed over to the threads in the thread pool for execution. Its core design is as follows: Figure 2 shown.

[0052] In order to ensure the stability of the system, a health check and back pressure mechanism are designed to monitor the operation of the system, collect abnormal information, and analyze and judge the information. When there is an imbalance between task reception and processing, adjustments can be made according to the set adjustment strategy to ensure the stability of the system and prevent system crashes due to excessive traffic and untimely processing. For abnormal situations, early warnings can be issued in time to notify relevant personnel to pay attention to services in a timely manner and make contingency plan adjustments. The overall process is as follows: Figure 3 shown.

[0053] According to the abnormal data of health check, the memory space of cache and temporary array, the time of temporary array timer, and the number of threads in thread pool can be adjusted dynamically. The adjustment can be made based on the analysis of CPU usage, memory utilization, and data delay time. The specific adjustment process is as follows: Figure 4 shown.

[0054] This application takes into account the high concurrency characteristics of the system when designing. In order to reduce competition between threads, a low-lock and lock-free concurrency control strategy is adopted. The lock mechanism uses a combination of spin and non-spin. Spin reduces the consumption caused by CPU switching and recovery, but long-term spin will occupy processor time. The combination of the two can strike a balance between reducing CPU switching consumption and processor time occupancy. Figure 5 shown.

[0055] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. An efficient data batch processing system based on memory, characterized in that: include: Memory management module, concurrent processing module, data flow control module, data processing module, monitoring and tuning module; The memory management module allocates a buffer and a temporary array area in the memory to form a batch queue core storage area for storing request data; The concurrent processing module is used to handle high-concurrency scenarios where multiple business units perform business processing in parallel; The data flow control module is used to effectively control the speed difference between data generation and consumption, maintain a balanced state, and has built-in rejection strategies and dynamic adjustments when imbalance occurs; The data processing module is used to select the optimal processing strategy and algorithm according to different types of data and business requirements; The monitoring and tuning module is used to monitor and perform health checks on various system indicators, and dynamically adjust system parameters by analyzing abnormal indicator data.

2. The memory-based efficient data batch processing system according to claim 1, characterized in that: The memory management module adopts the ConcurrentHashMap efficient data structure to reduce memory access conflicts and delays, and pre-allocates and manages memory resources through memory pool technology to reduce the overhead caused by frequent operations of memory allocation and release.

3. The memory-based efficient data batch processing system according to claim 1, characterized in that: The concurrent processing module uses multi-threading or multi-process technology to decompose the data processing task into multiple sub-tasks that can be executed in parallel, and assigns them to different processing units for simultaneous execution.

4. The memory-based efficient data batch processing system according to claim 3, characterized in that: The concurrent processing module adopts lock-free or low-lock concurrency control strategies, atomic operation-based updates, and read-write lock-based access control.

5. The memory-based efficient data batch processing system according to claim 1, characterized in that: The data flow control module introduces the buffer queue ArrayBlockingQueue as a buffer area between producers and consumers to balance the speed difference between data production and consumption.

6. The memory-based efficient data batch processing system according to claim 5, characterized in that: When the buffer is close to full, the data flow control module automatically adjusts the producer rate to prevent data overflow and system overload.

7. The memory-based efficient data batch processing system according to claim 1, characterized in that: The data processing module can automatically select the optimal processing strategy and algorithm based on different types of data and business requirements; and divide large amounts of data into multiple small batches for processing through batch processing mode.

8. The memory-based efficient data batch processing system according to claim 1, characterized in that: The monitoring and tuning module provides users with an intuitive view of the system operation status by monitoring system performance indicators in real time; it provides a parameter adjustment interface, allowing users to manually or automatically adjust system parameters according to actual needs.

9. An efficient data batch processing method based on memory, characterized in that: The system according to any one of claims 1 to 8 is implemented, comprising the following steps: A batch queue core storage area is formed by allocating a buffer and a temporary array area in the memory to store the request data; Based on the concurrent processing module to handle high-concurrency scenarios, multiple business units perform business processing in parallel; Effectively control the speed difference between data generation and consumption to maintain a balanced state, with built-in rejection strategies and dynamic adjustments when imbalance occurs; Select the best processing strategy and algorithm based on different types of data and business requirements; split large amounts of data into multiple small batches for processing through batch processing mode; Monitor and perform health checks on various system indicators, and dynamically adjust system parameters by analyzing abnormal indicator data.

Images