Distributed big data batch processing optimization method based on springbatch

Abstract

The application relates to the technical field of big data processing and distributed computing, in particular to a distributed big data batch processing optimization method based on SpringBatch, which comprises the following steps: multi-dimensional entropy analysis and fragmentation: performing random pre-sampling calculation key value information entropy, dynamically adjusting histogram bucket number, and establishing a dynamic fragmentation strategy based on a work-stealing mode; self-adaptive feedback control: collecting JVM heap memory usage, dynamically adjusting data grabbing size by using an incremental PID algorithm and a variable step limiter, and forming a self-adaptive back pressure mechanism; zero copy and reuse: performing zero copy conversion by using off-heap memory mapping, and reducing object creation frequency by using object pool reuse technology; asynchronous submission and fault tolerance: combining and submitting large transactions based on an asynchronous disk brushing strategy, and performing hierarchical retry operation; the application realizes statistical perception self-adaptive control of a batch processing process, and significantly improves system throughput and data consistency.

Description

Technical Field

[0001] This invention relates to the field of big data processing and distributed computing technology, specifically to a distributed big data batch processing optimization method based on SpringBatch. Background Technology

[0002] In the field of big data distributed batch processing, the SpringBatch framework has gradually become a core technology for building enterprise-level massive data cleaning and migration systems due to its powerful transaction management and process orchestration capabilities. Currently, such batch processing systems generally support data sharding and reading based on static rules. If it is necessary to process data sources of PB scale, tasks are usually divided by primary key hash modulo or preset fixed range in order to maintain basic parallel operation of the job.

[0003] However, when processing massive amounts of data with uneven key-value distribution, the above methods often assume that the data is distributed at equal density in the logical space. This leads to a severe imbalance in the data load between different computing nodes, making it impossible to eliminate the long-tail effect. Furthermore, it is impossible to perceive the resource status of computing nodes and dynamically adjust task allocation during job execution, resulting in low cluster computing power utilization. At the same time, this processing method uses a fixed step size strategy in the data crawling stage, and involves a large number of in-heap memory copies and temporary object creations during data flow, which can easily lead to memory overflow or frequent garbage collection pauses. Moreover, it relies on a simple full retry mechanism, which is difficult to implement hierarchical self-healing for network jitter or persistent failures, resulting in weak data eventual consistency assurance capabilities. Summary of the Invention

[0004] To address the aforementioned technical problems, this invention provides a distributed big data batch processing optimization method based on SpringBatch. Specifically, the technical solution of this invention includes: Presampling analysis is performed on the big data source to be processed to construct a data distribution histogram, and a dynamic sharding strategy is established based on the work-stealing pattern, wherein the dynamic sharding strategy is used to define the initial logic of task allocation; During job execution, the JVM heap memory usage of the current working node is collected in real time by the intelligent reading component, and the data capture size is dynamically adjusted by the PID controller algorithm to form an adaptive back pressure mechanism. The adaptive back pressure mechanism adjusts the data throughput of a single task and works in conjunction with the initial logic of task allocation defined in the dynamic sharding strategy to achieve a closed-loop matching between the working node load and real-time computing resources. Zero-copy transformation is performed on the read data stream, and object pooling technology is used to reduce the frequency of temporary object creation in order to complete the data cleaning and transformation operations. The asynchronous disk flushing strategy merges structured data records that conform to the preset target format after data cleaning and transformation into a large transaction for submission, and performs tiered retry operations when an exception occurs to ensure eventual data consistency.

[0005] Preferably, the large data source to be processed performs presampling analysis to construct a data distribution histogram, and establishes a dynamic sharding strategy based on the work-stealing pattern, including: Before the job starts, the key value distribution of the big data source is randomly sampled and scanned to identify data skew characteristics and generate the data distribution histogram. The data skew characteristics include extreme value distribution characteristics where the record density in a specific key value interval exceeds a preset multiple of the global average density. Based on the data distribution histogram, the entire data is divided into multiple logical partitions, and the logical partitions are stored in a preset global partition pool. During the operation, each working node actively requests the next logical segment to be processed from the global segment pool after completing the processing of the current segment, until the global segment pool is empty.

[0006] Preferably, the method further includes: During the generation of the data distribution histogram, it is determined whether the amount of data in a specific logical segment exceeds a preset single node processing threshold. If the single node processing threshold is exceeded, the logical segment is marked as a hotspot segment, and the hotspot segment is recursively decomposed into multiple sub-segments. The multiple sub-shards are distributed to different worker nodes for parallel processing to eliminate the long-tail effect.

[0007] Preferably, the step of collecting the JVM heap memory usage of the current working node in real time through the intelligent reading component and dynamically adjusting the data fetching size using the PID controller algorithm includes: Set a preset memory safety threshold and a preset memory warning threshold, wherein the memory safety threshold is less than the memory warning threshold; When the JVM heap memory usage is lower than the memory safety threshold, the intelligent reading component is controlled to linearly increase the data fetching size to improve data throughput. When the JVM heap memory utilization is greater than or equal to the memory safety threshold and less than or equal to the memory warning threshold, the current data fetch size remains unchanged. When the JVM heap memory usage exceeds the memory warning threshold, the intelligent reading component is controlled to exponentially reduce the data fetching size or perform a thread sleep operation to release memory resources.

[0008] Preferably, the step of performing zero-copy conversion processing on the read data stream and employing object pooling reuse technology to reduce the frequency of temporary object creation includes: The data stream is pipelined using a streaming processing interface, and pre-created object instances are reused to carry intermediate data during the transformation process; Identify large object data in the data stream whose data volume exceeds a preset volume threshold, and temporarily store the large object data in a preset off-heap memory area to reduce the frequency of JVM garbage collection mechanism triggering; After the data conversion is complete, the resources in the off-heap memory region are explicitly released.

[0009] Preferably, the step of performing tiered retry operations to ensure eventual data consistency when an anomaly occurs includes: Capture the exception type; if the exception type is a network connection interruption or network timeout exception, immediately trigger a memory-level retry. If the memory-level retry fails or the exception type is persistence failure, the failed data record will be serialized and stored in the local key-value database, and a persistence-level retry will be triggered at the end of the current job batch. If persistent retries still fail, the relevant data will be routed to a pre-defined dead-letter queue, and an alarm signal will be generated for manual intervention.

[0010] Preferably, the method further includes: After each transaction is successfully committed, the currently processed shard offset is asynchronously updated to the distributed coordination service center; When the system crashes and restarts, the shard offset is read from the distributed coordination service center, and a breakpoint resume operation is performed based on the shard offset.

[0011] Compared with the prior art, the present invention has the following beneficial effects: 1. This method performs pre-sampling analysis on the large data source to be processed and uses information theory to calculate the key-value distribution entropy, thereby constructing a data distribution histogram reflecting the data density. This can accurately identify data skew characteristics and dynamically adjust the statistical interval of the bucketing. Combined with the work-stealing mode, a dynamic sharding strategy is established to logically cut the full data into fine-grained micro-tasks and allocate them on demand. This effectively solves the problem of severe load skew of computing nodes and long-tail effect caused by uneven key-value distribution in the traditional static hash modulo method, and significantly improves the overall parallel processing efficiency of the cluster. 2. This method uses an intelligent reading component to collect the heap memory usage of worker nodes in real time and introduces an incremental PID controller algorithm to build an adaptive back pressure mechanism, which can dynamically adjust the data fetching size according to changes in memory level. When memory is sufficient, the fetching amount is linearly increased to improve throughput, and when memory is tight, the fetching amount is exponentially reduced or hibernation is executed to release resources. This avoids the risk of memory overflow caused by fixed step size strategy when data volume fluctuates, and achieves the optimal balance between system throughput and resource stability. 3. This method performs zero-copy conversion processing on the read data stream and adopts object pool reuse technology. During the data conversion process, pre-created object instances are reused to carry intermediate data, and large object data is identified and temporarily stored in the off-heap memory area. This approach significantly reduces the frequency of creation of temporary objects in the JVM heap and the frequency of garbage collection mechanism triggering, effectively reducing redundant copying of data between network transmission and processing buffers, and solving the problem of high CPU overhead and GC pauses caused by frequent object creation in traditional batch processing. 4. This method constructs a hierarchical retry mechanism that includes memory-level, persistent-level, and dead-letter queues, which can trigger different levels of recovery strategies based on the type of exception. Combined with the serialization storage of failure records in the local key-value database and the asynchronous disk flushing strategy for large transactions, it ensures the eventual consistency of data in extreme scenarios such as network jitter or service downtime, makes up for the shortcomings of simple retry mechanisms in dealing with complex faults, and realizes automatic fault tolerance and breakpoint resumption of jobs. Attached Figure Description

[0012] The present invention will be further explained below with reference to the accompanying drawings and embodiments: Figure 1 This is a flowchart of the method of the present invention; Figure 2 This is a flowchart demonstrating how to dynamically adjust the data capture size using a PID controller algorithm. Detailed Implementation

[0013] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to specific embodiments.

[0014] Example 1: Please see Figure 1 The distributed big data batch processing optimization method based on SpringBatch includes the following steps: Presampling analysis is performed on the big data source to be processed to construct a data distribution histogram, and a dynamic sharding strategy is established based on the work-stealing pattern, wherein the dynamic sharding strategy is used to define the initial logic of task allocation; During job execution, the JVM heap memory usage of the current working node is collected in real time by the intelligent reading component, and the data capture size is dynamically adjusted by the PID controller algorithm to form an adaptive back pressure mechanism. The adaptive back pressure mechanism adjusts the data throughput of a single task and works in conjunction with the initial logic of task allocation defined in the dynamic sharding strategy to achieve a closed-loop matching between the working node load and real-time computing resources. Zero-copy transformation is performed on the read data stream, and object pooling technology is used to reduce the frequency of temporary object creation in order to complete the data cleaning and transformation operations. The asynchronous disk flushing strategy merges structured data records that conform to the preset target format after data cleaning and transformation into a large transaction for submission, and performs tiered retry operations when an exception occurs to ensure eventual data consistency.

[0015] This embodiment details the specific process of reconstructing a traditional linear batch processing workflow into an adaptive control system with statistical sensing capabilities. It should be noted that the distributed big data in this invention specifically refers to the massive multimodal medical diagnosis and treatment data stored in the medical image archiving and communication system and the electronic medical record system, specifically including high-resolution CT image data and text-based patient visit records. Its technical meaning refers to clinical business data entries that need to be desensitized and cleaned for patient privacy, converted for image sequence format, and migrated to the medical research target database through batch processing. The system initiates a multidimensional data entropy analysis module. This module not only performs data reading but, more importantly, utilizes information theory methods to evaluate the uniformity of data distribution. Specifically, the module performs random pre-sampling analysis on petabyte-scale or terabyte-scale data sources, such as HDFS or HBase, and calculates the information entropy of the key-value distribution based on the sampled data. The formula is as follows: in, The total number of unique feature key values ​​in the sampled data; A characteristic prefix defined as a key value, such as taking the first part of RowKey. Bit-byte, or business aggregation dimension key, to solve the problem that the entropy value is constant and cannot represent the distribution skew in the unique primary key RowKey scenario because the key value is not repeated; For this feature key The probability of occurrence in the sampling set; based on the calculated entropy value, the module dynamically adjusts the construction parameters of subsequent histograms—the lower the entropy value, the more severe the skewness, and the module automatically refines the statistical intervals of the histogram; specifically, the number of statistical intervals. It follows the following nonlinear mapping equation: in, The number of bins is the baseline, for example, 100, and the unit of measurement is "pieces". This is a dimensionless gain coefficient, for example, set to 20. The range of values ​​is generally as follows: Determined based on system load characteristics. As a sensitivity gain coefficient, its magnitude determines the system's sensitivity to data skew; when When the value is large, even the entropy value Only a slight decrease, number of buckets This will also increase significantly, thereby improving the statistical resolution of the histogram in the sloping region; It needs to be clarified that in the formula Since the entropy ratio is dimensionless, therefore... It is also a dimensionless scalar. It is also a dimensionless parameter to ensure that the expression within the parentheses is a purely numerical scaling factor, thereby... The dimensions are strictly inherited from That is, the number of buckets; The total number of samples is represented by the number of records. The theoretical maximum entropy when the sample is perfectly uniformly distributed; This formula ensures that even when the data is extremely skewed, i.e. At that time, the number of buckets increases quadratically, thereby constructing a data distribution histogram that can accurately reflect the distribution density of data key values; the system abandons the traditional static ID modulo sharding method and instead adopts a work-stealing mode to establish a dynamic sharding strategy. This strategy logically cuts the full data into fine-grained micro-tasks, defines the initial logic of task allocation on demand and dynamic load, and establishes the load balancing benchmark for subsequent processing. During job execution, the intelligent reading component is activated. This component has a built-in non-intrusive JVM probe that collects the JVM heap memory usage of the current working node in real time. Based on this, this embodiment introduces a PID controller algorithm to construct a feedback loop. To avoid integral saturation and adapt to the discrete control characteristics of the computer, as well as to better connect with subsequent linear / exponential adjustment strategies, the system adopts an incremental PID algorithm, also known as a velocity algorithm, to dynamically calculate and adjust the data capture size. The control law is defined as follows: in, : The current discrete sampling time, corresponding to the time point ; Sampling period, for example, set to 0.5 seconds; The target data capture volume calculated at the current moment, in units of records; : The increment value of the captured data at the current time relative to the previous time. The memory state error value at the current moment, which is the memory safety threshold minus the actual heap memory utilization; : Proportional coefficient, which acts on the rate of change of error and is used to quickly respond to memory fluctuations; Integral coefficients, in incremental algorithms, directly affect the current error value. When memory remains below a threshold, i.e. At that time, this driver Continuous cumulative growth leads to a linear increase in throughput; : Differential coefficients, which act on the second difference of the error, and are used to predict trends and suppress overshoot; Here, to ensure the rigor of the above control formula in physical implementation, it is necessary to clarify the physical dimensions of each parameter: due to the control objective and increment The physical dimension is the number of records, while the error term... Defined as the difference in memory usage, it is essentially a dimensionless pure numerical ratio. Based on the principle of homogeneity of dimensions, the proportionality coefficient... Integral coefficient and differential coefficients The physical dimensions of all three parameters are strictly defined as the number of records. This indicates that these three parameters are not merely mathematical weights, but conversion factors with clear physical meanings, representing the number of records adjusted for each unit of memory deviation, thus ensuring that the dimensions on both sides of the formula are strictly consistent. It should be noted that the present invention employs a gain scheduling strategy in its specific implementation, that is, it allows the gain scheduling of the above standard formula to be adjusted according to the system state. , or Dynamically set to zero or adjust to achieve piecewise linear control objectives; Furthermore, addressing the shortcomings of the standard incremental PID formula in handling JVM non-linear GC pauses, this embodiment makes key customized improvements to this general formula: In calculating... Afterwards, instead of directly accumulating, a variable step size limiter is introduced, defined as follows: in, For a symbolic function, take The plus or minus sign, This is the absolute value operator, used to extract the modulus of the crawled data increment. The maximum allowed mutation rate based on the current total heap memory, for example 5%, is the upper limit for the number of records obtained after conversion. The formula for calculating this limit is: in, This refers to the total JVM heap memory. The percentage of allowed mutations, such as 0.05, is typically set to a percentage of the JVM heap memory. To avoid GC pressure caused by sudden changes in the amount of data crawled in a single session, This represents the average byte size of records in the current batch, as counted through the sliding window. Specifically, this applies at the beginning of the job or when the sliding window is empty. Use a preset global average record size, such as 1KB, or initialize by reading the size of the first record in bytes to avoid calculation errors where the denominator is zero; increment the crawling volume after calculating the limit. Then, update the target data crawling volume at the current moment to... Meanwhile, to eliminate the risk of integral saturation, the system employs anti-saturation back-calculation logic. If the calculated... touch If the maximum limit is reached, then the historical error term is corrected in reverse. The specific correction formula is as follows: in, This refers to the actual amount of data captured after the aforementioned saturation truncation process. This refers to the historical error term after anti-saturation backcalculation correction; the actual executed value is compared with the theoretical calculated value. The deviation is fed back to the historical error term, enabling the controller to sense the boundary limits of the actuator, thereby eliminating the risk of integral saturation. : Proportionality coefficient, here The same physical parameter as the aforementioned proportionality coefficient; when the system is in gain scheduling mode and When dynamically set to zero, the formula uses a preset non-zero baseline scaling factor for back calculation to ensure the numerical stability of the anti-saturation logic. By feeding back the truncation difference to the historical error term, the integral term is forced to automatically exit the saturation region in the next cycle; this improvement enables the algorithm to truly adapt to the discrete control characteristics of computers and smoothly connect to subsequent linear / exponential regulation strategies. At the same time, to prevent the calculated dynamic crawling volume If a negative or non-integer value is encountered, the program crashes, and the system performs saturation and rounding processing on the PID output. in, Set the minimum number of records to retrieve, for example, 1 record; The round-down operator is used to map the continuous calculated value output by the PID controller to discrete integer records. This formula then forms an adaptive backpressure mechanism: when memory is tight, the error value is negative, leading to a decrease in the amount of data fetched, thus reducing the upstream data inflow rate; when memory is ample, the amount of data fetched increases, improving throughput. Simultaneously, zero-copy conversion is performed on the read data stream, employing off-heap memory mapping technology to allow data to be directly transferred between the network transmission and processing buffers, avoiding redundant data copying within the JVM heap. Furthermore, object pooling reuse technology is introduced, pre-initializing a set of reusable business object instances in the Processor stage. When new data arrives, the properties of objects in the pool are directly reset and populated, rather than creating new objects using the `new` keyword, aiming to significantly reduce the frequency of temporary object creation. The processed data is not committed directly, but is written to the buffer based on an asynchronous disk flushing strategy. When the buffer reaches a preset threshold or time window, multiple small batches of data are merged into a large transaction for commit. In the event of an exception, different levels of recovery strategies are triggered according to the exception type to ensure eventual data consistency.

[0016] Example 2: Presampling analysis is performed on the large data source to be processed to construct a data distribution histogram, and a dynamic sharding strategy is established based on the work-stealing pattern, including: Before the job starts, the key-value distribution of the big data source is randomly sampled and scanned to identify data skew characteristics and generate a data distribution histogram. The data skew characteristics include extreme value distribution characteristics where the record density within a specific key-value interval exceeds a preset multiple of the global average density. The global average density is the ratio of the total number of sampled samples to the span of the key-value space of the entire data, and the preset multiple is usually set to 3 to 5 times. Extreme value distribution characteristics refer to statistical measures used to quantify the degree of abnormal concentration of data within the specific interval, specifically including one or more of the following indicators: the local highest number of records, the actual ratio of local density to global average density, skewness coefficient, or the frequency of occurrence of abnormally high-frequency key values. The entire dataset is divided into multiple logical partitions based on the data distribution histogram, and the logical partitions are stored in a preset global partition pool. During job execution, each worker node actively requests the next logical segment to be processed from the global segment pool after completing the processing of the current segment, until the global segment pool is empty.

[0017] This embodiment further specifies the pre-sampling analysis and dynamic sharding strategy in Embodiment 1, aiming to solve the data skew problem in a large-scale distributed environment. In the preparation phase before job startup, a sparse sampling scan is performed to randomly sample the key-value distribution of the large data source, with a sampling rate set to one-thousandth to one-hundredth of the total data. The system identifies high-frequency key-values ​​in the sampled data and generates a data distribution histogram. This histogram visually reflects the density differences of data in different key-value intervals, thereby accurately identifying potential data hotspots. Based on the data distribution histogram, an equal-splitting algorithm is used to divide the total data into multiple logical shards. The logic for generating the shard boundary set is as follows: in, The integral variable represents the key-value pair of a record in the data source. The coordinate values ​​after mapping to numerical space; for non-numerical row keys commonly found in big data scenarios. For types like String, this embodiment uses lexicographical numerical mapping to ensure that the monotonicity of variable-length strings after mapping to integers is consistent with the original lexicographical order, i.e., to guarantee... The system obtains the preset global maximum key length. For example, for a 1024-byte key, the binary sequence of the key is padded with zero bytes (0x00) to the end until the length is reached. This interprets the fixed-length sequence as a high-precision unsigned integer, BigInteger, thereby continuously mapping the discrete string key-value space onto the real number axis. This ensures that the lexicographical order of the key values ​​is consistent with the monotonicity of the numerical space, providing a continuous field basis for the construction of the density function and subsequent calculus operations. : The index of the slice sequence, with a value range of to ,in, This represents the total number of partitions estimated based on the size of the full dataset; : No. The initial boundary values ​​of each segment are derived from the cumulative distribution function of the histogram. The data record density function, whose physical dimension is explicitly defined as the number of records divided by the numerical key unit; It should be noted here that the numerical key-value unit refers to the unit distance on the real number axis after projecting the string key-value pair onto the real number axis using the mapping technique described above; that is, the unit increment of BigInteger. Although this unit is an abstract address space metric, it acts as a differential variable in integration operations. Dimensional basis; This represents a tiny increment on the real number axis after key-value mapping, in units of numerical key-value units. and After multiplication, the integral result The dimension of the quantity strictly reverts to the number of records, which is consistent with the constant on the right side of the equation. Strict consistency of physical dimensions was maintained, among which, The target data volume is set as a constant, with the unit being the number of items, thus eliminating logical loopholes; This function is derived from the histogram interpolation generated by presampling; specifically... The construction uses piecewise linear interpolation, defining the histogram's first... The frequency of each bucket is Width is The average density of the bucket Define the instantaneous density at the interpolation nodes. This is the average density of adjacent buckets, i.e. Then for the interval , in, For the key-valued integral variable after continuous mapping; and These are the left and right boundary coordinates of adjacent sampling intervals in the histogram, respectively. The sharding logical order index is arranged lexicographically, assuming that the density change between adjacent sampled buckets is linear; for shard boundaries... The system employs a numerical integration approximation method to solve the problem. Iterate through each bucket of the histogram, accumulating its frequency. When the accumulated value reaches... At that time, the exact cutoff point is calculated as the boundary based on the density slope within the current bucket. And reset the accumulator to achieve equal division of the entire data; specifically, let the currently traversed histogram Bucket interval be... ,in, : Instantaneous density at the starting position of the current Bucket; : Instantaneous density at the current bucket termination position; The current fragment has not reached the target value. The remaining amount of data that still needs to be filled, i.e. The unit of measurement is the number of records. This represents the total frequency of histogram buckets accumulated in the current iteration step; Assuming the density change within the interval follows a linear characteristic, then the density slope is: According to the formula for the area of ​​a trapezoid, the cutoff boundary needs to be determined. Satisfies the integral equation: After unfolding, we can obtain information about The quadratic equation of : The system solves for the positive real roots of the equation by calling a standard mathematical library, thereby accurately calculating the... Coordinates: In particular, for extreme scenarios where the data distribution has gaps, i.e., when the slope caused by interpolation of adjacent buckets... Make the density function within the interval Negative values ​​or discriminant values ​​appear. When it is impossible to find real roots, or when the slope is... When the risk of division by zero occurs, among which, For extremely small positive numbers, the system automatically triggers an anomaly avoidance mechanism: abandoning the second-order fit of the current interval and forcibly downgrading to a zero-order preservation strategy, that is, assuming that the density of the current interval is constant. Calculate directly: like If the value also approaches zero, then the partition boundary is directly set. Extend to the starting position of the next Bucket and the remaining unfilled Carry over to the next round of calculation; this fault-tolerant logic ensures that the algorithm can stably calculate the effective partition boundaries under any sparse or distorted data distribution; This calculation step eliminates the granularity error introduced by traditional discrete bucket statistics, ensuring that the data volume of each slice closely approximates the target value. ; This is a preset target data volume constant for a single shard, and its physical meaning is the expected number of data entries per shard. The calculated numerical form of the partition boundary Subsequently, to ensure that this boundary can be recognized by the underlying storage system, such as HBase or HDFS, the system performs the necessary reverse mapping operation: calling the BigInteger's toByteArray() method to convert the value into a byte sequence, and based on the original key length recorded in the previous steps or using a fixed-length encoding rule, such as retaining the original key length as metadata, extracting the precise original byte length from the byte sequence, and re-encoding the remaining valid byte sequence into a string format RowKey, thereby generating a physically executable sharding range. ; Based on this, the metadata of all logical shards, including ID, range and status, is stored in a preset global sharding pool, which is usually implemented by a high-performance distributed coordination service. During job execution, each worker node is controlled to adopt a pull mode. After completing the processing of the current shard, each node does not wait for the master node to allocate, but actively requests the next logical shard with the status of pending processing from the global sharding pool. This process is repeated until the global sharding pool is empty.

[0018] Example 3: The method also includes: during the process of generating the data distribution histogram, determining whether the amount of data in a specific logical segment exceeds a preset single node processing threshold; If the threshold for processing a single node is exceeded, the logical shard is marked as a hot shard, and the hot shard is recursively decomposed into multiple sub-shards. Multiple sub-shards are distributed to different worker nodes for parallel processing to eliminate the long-tail effect.

[0019] This embodiment further optimizes the sharding strategy, specifically targeting highly skewed hotspot data. During the generation of the data distribution histogram, the system executes threshold determination logic to determine whether the estimated data volume within a specific logical shard exceeds a preset single-node processing threshold, such as 80% of the maximum hourly processing capacity of a single node. In response to exceeding the single-node processing threshold, the system marks the logical shard as a hotspot shard and does not distribute it as a whole, but instead performs recursive decomposition. The decomposition factor is calculated as follows: in, : Decomposition factor, physically meaning the number of sub-fragments that need to be divided; The estimated total amount of data for hotspot sharding is derived from histogram integral calculation, and the unit is the number of records. The single node processing threshold is derived from the node hardware specification preset, and the unit is the number of records. Furthermore, the system further divides this hotspot segment into... Sub-sharding; in this step, to clarify the specific decomposition implementation path, this embodiment performs differentiated segmentation for two different hotspot types: If hotspot sharding is due to a single key This is caused by excessive data volume, such as for specific high-profile users. The system keeps the RowKey unchanged and uses the ColumnPaginationFilter interface to convert the single key... Logical partitioning of column family data into Each sub-task; for the first Each sub-task, among which... In practical implementation, for key-value stores such as HBase, the ColumnPaginationFilter or setRowOffset interface is called, and the specific parameter configuration logic is as follows: Set the target number of sub-shards For the serial number Set the page start offset for sub-parts. Set the limit on the number of lines read. ,like Then set to To read all remaining lines; for file storage, such as HDFS, call the seek and read byte range interfaces to enable each sub-shard to read all remaining lines. Only specific data segments within the original large object are physically read, thus distributing the load evenly across a single point of failure. In each subtask; if hotspot partitioning is caused by excessively high density in a certain key value interval, then perform linear interpolation cutting and calculate the sub-partition boundaries. in, The physical starting key value for hotspot sharding. The physical end key value for hotspot sharding. The decomposition factor is the one calculated above. For sub-shard indexes; to strictly execute the recursive decomposition logic, in the initial generation After sub-sharding, the system performs recursive verification on each sub-shard: recalculating the data density of the sub-shard; if the data density of a certain sub-shard still exceeds the limit... For example, if the data is extremely unevenly distributed within a sub-interval, then that sub-partition is used as new input, and the above decomposition logic is recursively called until the data volume of all leaf partitions is less than [a certain value]. Or reach the preset maximum recursion depth, such as 3 layers, to ensure that there are no hidden hotspots left behind; Multiple sub-shards are treated as independent tasks, injected back into the global shard pool, and distributed to different worker nodes for parallel processing.

[0020] Example 4: Please see Figure 2 The system uses an intelligent reading component to collect the JVM heap memory usage of the current worker node in real time, and dynamically adjusts the data fetching size using a PID controller algorithm, including: Set a preset memory safety threshold and a preset memory warning threshold, where the memory safety threshold is less than the memory warning threshold; When the JVM heap memory utilization rate is lower than the memory safety threshold, the intelligent reading component is controlled to linearly increase the data fetch size in order to improve data throughput; When the JVM heap memory utilization is greater than or equal to the memory safety threshold and less than or equal to the memory warning threshold, the current data fetch size remains unchanged. When the JVM heap memory usage exceeds the memory warning threshold, the intelligent read component is controlled to exponentially reduce the data fetch size or perform thread sleep operations to release memory resources.

[0021] This embodiment details a JVM memory adaptive adjustment mechanism based on a PID controller; it sets two key memory waterline levels, namely memory safety thresholds. With memory warning threshold And satisfy Specifically, to ensure that the threshold parameter has a clear physical meaning and is executable, a memory safety threshold is required. Initialized to 60% of the JVM's maximum heap memory, a memory warning threshold. Initialized to 85% of the maximum heap memory, where the maximum heap memory value is dynamically obtained during system startup by calling the JMX interface ManagementFactory.getMemoryMXBean().getHeapMemoryUsage().getMax(), thus ensuring that the threshold setting adapts to the hardware resource limitations of the current container; JVM heap memory usage is collected in real time through an intelligent reading component. and according to Differential control is implemented within the specified interval; To address the logical mapping issue between the PID algorithm defined in Example 1 (i.e., continuous linear control) and the nonlinear piecewise control (linear increase / hold / exponential decrease) defined in this example, this example employs gain-scheduled PID technology. This involves dynamically reconstructing the error definition of the PID controller across different memory intervals. With control parameters This allows for precise implementation of the algorithm. This embodiment is essentially a special case of gain scheduling for the general incremental PID algorithm in Embodiment 1, where some parameters are set to zero or dynamically adjusted based on the system state. The specific control law is defined as follows: Linear growth region The system goal at this point is to fully utilize memory; define the error. Configure PID parameters to integral-dominant type: Set And set the integral coefficient A positive constant, such as 100; according to the PID increment formula ,because Continuously positive Maintaining a positive value, thereby driving The continuous cumulative growth achieved the control target of linear increase; Steady-state maintenance region At this point, the system is in its ideal operating range; a dead-time mechanism is introduced to force a reset of the error input. Regardless of actual memory fluctuations, the PID output remains unchanged because input errors are masked. Thus This remained unchanged, avoiding unnecessary fluctuations within the safe range; Exponential inhibition zone At this point, there is a risk of OutOfMemoryError (OOM), requiring a drastic reduction in load; define the error. That is, negative error; configure the PID parameters as variable gain integral-dominant type: given the characteristics of incremental PID algorithm, the proportional term Only the rate of change of response error cannot exceed the limit in steady state, i.e. When the value is a constant negative value, it produces a continuous regulating effect; therefore, the integral term must be utilized. To drive the control output; in order to achieve exponential decay, that is: in, , Usually taken Adjust according to the system response speed requirements, that is, require Substitute into the PID integral term formula The system calculates the dynamic integral coefficients in real time. in, To prevent extremely small positive numbers with a denominator of zero, they are usually set to 0. ; Simultaneously set It should be noted that here... The dimension of the numerator is explicitly designed to be the number of data points, because the numerator in the formula... The dimension is the number of items, and the denominator is the dimensionless percentage difference; although this makes The coefficients are no longer constant as in traditional PID controllers, but rather vary with the system state. The dynamic gain varies, but this design is completely self-consistent in terms of dimensions, i.e., the PID output term. The dimension of the quantity is: number of items Dimensionless = number of lines, and it is precisely by utilizing this variable gain characteristic that the linear PID controller can output a nonlinear multiplicative decrease in control quantity, thereby quickly releasing memory resources; If the above adjustments are made and the capture size is reduced to the minimum value... and If the memory level still doesn't drop, a thread sleep operation is triggered, pausing data reading and waiting for the garbage collector to release memory resources; the specific sleep strategy uses an exponential backoff mechanism. in, Based on the duration of hibernation, This is the maximum duration. The number of times the warning threshold is continuously triggered is set to avoid frequent thread wake-ups that exacerbate CPU jitter when memory is extremely tight. Through the above rigorous mathematical definition, the system realizes the engineering mapping from PID principle to linear increasing / exponential decreasing behavior.

[0022] Example 5: Zero-copy conversion is performed on the read data stream, and object pooling technology is used to reduce the frequency of temporary object creation, including: A streaming processing interface is used to pipe data streams, and pre-created object instances are reused to carry intermediate data during the transformation process; Identify large object data in the data stream that exceeds a preset volume threshold, and temporarily store the large object data in a preset off-heap memory area to reduce the frequency of JVM garbage collection mechanism triggering; After the data conversion is complete, explicitly release the resources in the off-heap memory area.

[0023] This embodiment has made in-depth optimizations to the memory overhead of the data transformation process; it uses a streaming processing interface to pipeline the data stream, ensuring that the data is transferred in the form of a stream between the Mapper, Processor, and Filter stages, rather than loading the entire list into memory; During processing, the system identifies large object data in the data stream in real time. The judgment logic is that if the data volume of a single record exceeds the preset volume threshold, it is identified as a large object. The preset volume threshold is not a fixed value, but is dynamically set according to the JVM startup parameters. It is usually set to 50% of the G1 garbage collector region size. For example, if the region size is 2MB, the threshold is 1MB to prevent the object from being directly allocated to the Humongous region, which would cause heap memory fragmentation. In practice, to ensure that object data is identified before entering the JVM heap memory, thereby truly reducing the frequency of GC triggering, the system performs a protocol header pre-read operation: in the input stream's socket channel or file channel, only the leading byte LengthField defining the payload length is pre-read, such as the first 4 bytes of the standard protocol, and parsed into an integer length value. ; Only when When the object is identified as a large object, ByteBuffer.allocateDirect is called directly to allocate off-heap memory. The preset volume threshold is used; otherwise, allocate is called to allocate memory in the heap. This pre-judgment strictly avoids redundant memory allocation caused by first reading into the heap to determine the size and then moving it to the outside of the heap. The system calls the underlying allocation method to temporarily store large object data in a preset off-heap memory area, which is not directly managed by the JVM heap memory garbage collection mechanism; in terms of specific technical implementation, the system calls the ByteBuffer.allocateDirect method in the JavaNIO package to request local memory and create a DirectByteBuffer instance; This instance holds only a tiny reference object within the JVM heap, while the actual data is stored outside the heap. For writing large object data, zero-copy transfer is achieved using the FileChannel's transferTo method or MappedByteBuffer. Furthermore, for object pool reuse technology, the system uses a bounded blocking queue based on ArrayBlockingQueue as the object container, with the pool size initialized to 1.2 times the number of worker threads. At this point, the pre-created object instances in the pool are designed as lightweight view wrappers in the flyweight pattern. These objects only hold direct address references to off-heap memory blocks and their data lengths, without copying the actual data into the JVM heap. When intermediate data needs to be processed, the wrapper object is borrowed from the queue and points to the corresponding off-heap memory region. The Processor component directly performs cursor-style reading and cleaning of the off-heap data through the API provided by this wrapper. To resolve the logical paradox of not being able to perform data cleaning (i.e., modification) under zero-copy constraints, this embodiment clarifies the micro-operation mechanism of the Processor component: The wrapper actually holds two sets of memory references—a source address pointer and a target address pointer. The data cleaning process is not an in-place modification, but rather based on a streaming memory copying model. Business logic transformations, such as desensitization and format conversion, are performed in the stack memory of the worker thread as local variables. After that, the results are directly written to the off-heap memory block pointed to by the target address. During this process, data only flows between off-heap memory, thread stack, and off-heap memory, strictly avoiding the overhead of copying byte arrays as object instances to the JVM heap to generate Java objects. This satisfies the zero-heap object copy principle while achieving complex data cleaning and reorganization functions. After the operation is completed, the reset() interface implemented by the object is called to reset the reference address and then returned to the queue, thus truly achieving zero copy of data content and no dirty data residue. After the data conversion is completed and the result is written to the output buffer, the system immediately calls the cleanup hook to explicitly release resources in the off-heap memory area. In order to prevent off-heap memory leaks, this embodiment introduces a phantom reference monitoring mechanism. When the DirectByteBuffer object in the JVM heap is reclaimed, the Cleaner thread is triggered to call the Unsafe.freeMemory interface to forcibly reclaim the corresponding native memory block.

[0024] Example 6: To ensure eventual data consistency, tiered retry operations are performed when an anomaly occurs, including: Capture exception types; if the exception type is a network connection interruption or network timeout exception, immediately trigger a memory-level retry. If a memory-level retry fails or the exception type is persistence failure, the failed data record is serialized and stored in the local key-value database, and a persistence-level retry is triggered at the end of the current job batch. If persistent retries still fail, the relevant data will be routed to a pre-defined dead-letter queue, and an alarm signal will be generated for manual intervention.

[0025] This embodiment constructs a multi-level fault tolerance mechanism to ensure eventual data consistency when anomalies occur. During job execution, all thrown anomalies are captured and their types are identified. In response to transient anomalies, such as network connection interruption or network timeout, the system immediately triggers a memory-level retry within the current thread. The number of retries is set to a predetermined value, and the interval uses an exponential backoff algorithm, specifically calculated as follows: in, Set to 200ms Set to 5000ms. Explicitly defined as the number of retries already performed, i.e. Starting from 0, the count increments by 1 with each retry, thus avoiding request storms during network jitter. This process does not involve disk I / O. In response to memory-level retry failures or persistent exceptions, the system serializes the currently failed data records and stores them in a local embedded key-value database. Specifically, it uses RocksDB or LevelDB as the underlying storage engine, leveraging the high write performance of their LSM-Tree structure to quickly write the failed objects to disk after serializing them into Protobuf binary streams. During this process, to ensure that subsequent retries can accurately retrieve specific records from the current batch, the system defines strict storage keys. Generation rules: in, This is a unique identifier for the assignment. This is a preset logical delimiter used to build hierarchical indexes within an ordered key structure. This is the batch serial number currently being processed. The timestamp in milliseconds is the time the exception was triggered. The original message content for the failure record. Using the MD5 fingerprint hash value calculated from the failed record content, and leveraging this ordered key structure, the retry thread can efficiently locate the current key using the RocksDB prefix_seek interface. All failure records in the prefix enable the system to pass and The combined prefix identifiers utilize a prefix scanning mechanism to achieve fast, non-traversal retrieval of failed records, avoiding full table scans. After all normal data processing for the current job batch is completed, a persistent retry is triggered, resubmitting the failed records from the local database. If the persistent retry still fails, the system determines the data to be poison pill data, routes the relevant data to a preset dead-letter queue, and generates an alarm signal to notify maintenance personnel to intervene and perform manual repair. This embodiment upgrades anomaly handling to a hierarchical self-healing system, which can automatically filter out most temporary faults in scenarios with unstable network environments or fluctuating downstream services, ensuring the continuity of operations. At the same time, through a dead-letter queue mechanism, it ensures that even under extreme faults, bad data can be traced, achieving zero data loss.

[0026] Example 7: The method also includes: after each transaction is successfully committed, the currently processed shard offset is asynchronously updated to the distributed coordination service center; When the system crashes and restarts, it reads the shard offset from the distributed coordination service center and performs a breakpoint resume operation based on the shard offset.

[0027] This embodiment provides the ability to resume interrupted data transmission after system failure; after each transaction is successfully committed, the system obtains the currently processed shard read position, i.e., the shard offset; through an asynchronous thread, the offset is updated to the highly available distributed coordination service center; it is recommended to use a write-before log or transaction log to ensure the atomicity of the offset and data commit, in order to deal with the data consistency problems that may be caused by the delay of asynchronous update; When the system crashes and restarts, or is rescheduled through the container orchestration system, the worker node connects to the distributed coordination service center and reads the latest shard offset of the shard it is responsible for. The reading component locates the offset position, executes a resume operation, skips the processed data, and continues with subsequent tasks. To ensure efficiency in the recovery process under big data scenarios, avoid using... The system skips complex traversal operations and performs physical pointer relocation for different storage media. If the data source is a distributed file system, call FSDataInputStream.seek(offset) to move the file cursor to the target byte position directly; if the data source is a message queue, such as Kafka, call Consumer.seek to reset the consumption position; through the above underlying positioning instructions, it is ensured that no matter how large the file is, the time complexity of breakpoint recovery is always O(1); in addition, considering the problem of repeated reading of a small amount of data during the downtime window period that may be caused by asynchronous offset updates, that is, the data has been committed but the offset has not been updated, this embodiment enforces idempotent encapsulation in the data conversion stage: By using the unique business primary key or full-field MD5 fingerprint of the source data, a deterministic ID is generated, and INSERTIGNORE or ONDUPLICATEKEYUPDATE semantics are used at the database commit layer to ensure that no duplicate dirty data is generated even in the case of offset rollback, thus making up for the consistency vulnerability of asynchronous coordination in terms of mechanism.

[0028] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention.

Claims

1. A distributed big data batch processing optimization method based on SpringBatch, characterized in that, Includes the following steps: Presampling analysis is performed on the big data source to be processed to construct a data distribution histogram, and a dynamic sharding strategy is established based on the work-stealing pattern, wherein the dynamic sharding strategy is used to define the initial logic of task allocation; During job execution, the JVM heap memory usage of the current working node is collected in real time by the intelligent reading component, and the data capture size is dynamically adjusted by the PID controller algorithm to form an adaptive back pressure mechanism. The adaptive back pressure mechanism adjusts the data throughput of a single task and works in conjunction with the initial logic of task allocation defined in the dynamic sharding strategy to achieve a closed-loop matching between the working node load and real-time computing resources. Zero-copy transformation is performed on the read data stream, and object pooling technology is used to reduce the frequency of temporary object creation in order to complete the data cleaning and transformation operations. The asynchronous disk flushing strategy merges structured data records that conform to the preset target format after data cleaning and transformation into a large transaction for submission, and performs tiered retry operations when an exception occurs to ensure eventual data consistency.

2. The method according to claim 1, characterized in that, The large data source to be processed undergoes presampling analysis to construct a data distribution histogram, and a dynamic sharding strategy is established based on the work-stealing pattern, including: Before the job starts, the key value distribution of the big data source is randomly sampled and scanned to identify data skew characteristics and generate the data distribution histogram. The data skew characteristics include extreme value distribution characteristics where the record density in a specific key value interval exceeds a preset multiple of the global average density. Based on the data distribution histogram, the entire data is divided into multiple logical partitions, and the logical partitions are stored in a preset global partition pool. During the operation, each working node actively requests the next logical segment to be processed from the global segment pool after completing the processing of the current segment, until the global segment pool is empty.

3. The method according to claim 2, characterized in that, Also includes: During the generation of the data distribution histogram, it is determined whether the amount of data in a specific logical segment exceeds a preset single node processing threshold. If the single node processing threshold is exceeded, the logical segment is marked as a hotspot segment, and the hotspot segment is recursively decomposed into multiple sub-segments. The multiple sub-shards are distributed to different worker nodes for parallel processing to eliminate the long-tail effect.

4. The method according to claim 1, characterized in that, The method of collecting the JVM heap memory usage of the current working node in real time through an intelligent reading component and dynamically adjusting the data fetching size using a PID controller algorithm includes: Set a preset memory safety threshold and a preset memory warning threshold, wherein the memory safety threshold is less than the memory warning threshold; When the JVM heap memory usage is lower than the memory safety threshold, the intelligent reading component is controlled to linearly increase the data fetching size to improve data throughput. When the JVM heap memory utilization is greater than or equal to the memory safety threshold and less than or equal to the memory warning threshold, the current data fetch size remains unchanged. When the JVM heap memory usage exceeds the memory warning threshold, the intelligent reading component is controlled to exponentially reduce the data fetching size or perform a thread sleep operation to release memory resources.

5. The method according to claim 1, characterized in that, The process of performing zero-copy conversion on the read data stream and employing object pooling technology to reduce the frequency of temporary object creation includes: The data stream is pipelined using a streaming processing interface, and pre-created object instances are reused to carry intermediate data during the transformation process; Identify large object data in the data stream whose data volume exceeds a preset volume threshold, and temporarily store the large object data in a preset off-heap memory area to reduce the frequency of JVM garbage collection mechanism triggering; After the data conversion is complete, the resources in the off-heap memory region are explicitly released.

6. The method according to claim 1, characterized in that, The step of performing tiered retry operations when an anomaly occurs to ensure eventual data consistency includes: Capture the exception type; if the exception type is a network connection interruption or network timeout exception, immediately trigger a memory-level retry. If the memory-level retry fails or the exception type is persistence failure, the failed data record will be serialized and stored in the local key-value database, and a persistence-level retry will be triggered at the end of the current job batch. If persistent retries still fail, the relevant data will be routed to a pre-defined dead-letter queue, and an alarm signal will be generated for manual intervention.

7. The method according to claim 1, characterized in that, Also includes: After each transaction is successfully committed, the currently processed shard offset is asynchronously updated to the distributed coordination service center; When the system crashes and restarts, the shard offset is read from the distributed coordination service center, and a breakpoint resume operation is performed based on the shard offset.

Images