Streaming processing method for real-time cleaning of agricultural product marketing data

Abstract

The invention provides a streaming processing method for real-time cleaning of agricultural product marketing data, and belongs to the technical field of big data processing.The method includes the steps that a field dependency directed graph is constructed, dependency intensity is quantified through mutual information, a dependency network is decomposed into weak coupling sub-graph sets through a minimum cut algorithm, and the weak coupling sub-graph sets are inserted into intermediate materialization nodes to achieve decoupling; performing Hamiltonian path inspired operator fusion optimization on each sub-graph to generate a super operator, converting the super operator into a machine code through a just-in-time compiler, establishing a multi-stage cache affinity scheduling mechanism and a dynamic watermark mechanism to process out-of-order data, and adopting a two-stage dynamic repartitioning strategy and a work stealing queue to cope with data skew; a hierarchical storage architecture is constructed, and capacity distribution is adjusted through a storage redundancy optimization function, so that the technical problem that the parallel processing capacity is limited due to a strong dependency relationship between fields in the agricultural product marketing data real-time cleaning process is solved.

Description

Technical Field

[0001] This invention belongs to the field of big data processing technology, and more specifically, relates to a streaming processing method for real-time cleaning of agricultural product marketing data. Background Technology

[0002] In the field of real-time cleaning of agricultural product marketing data, traditional streaming systems typically employ a data flow processing framework based on directed acyclic graphs (DAGs). This framework constructs a data processing pipeline by defining upstream and downstream dependencies between cleaning operators. Each cleaning operator executes filtering, verification, and mapping transformation operations sequentially according to a topological order, thereby improving the quality of the original marketing data. In current large-scale agricultural e-commerce platforms, marketing data streams contain multi-dimensional fields such as product information, price fluctuations, inventory status, and order records. These fields have complex dependencies on cleaning rules; for example, price reasonableness verification depends on product category determination, and order validity checks depend on inventory data verification. However, traditional cleaning processes treat all field dependencies as equivalent constraints, employing a globally serialized or coarse-grained grouped parallel processing model. This results in even fields with only weak dependencies requiring strict adherence to the dependency order, leading to lengthy critical path blockages. In other words, existing technologies suffer from the technical problem of limited parallel processing capabilities due to strong dependencies between fields during real-time cleaning of agricultural product marketing data. Summary of the Invention

[0003] In view of this, the present invention provides a streaming processing method for real-time cleaning of agricultural product marketing data, which can solve the technical problem in the prior art where strong dependencies between fields in the real-time cleaning process of agricultural product marketing data limit parallel processing capabilities.

[0004] This invention is implemented as follows: A streaming processing method for real-time cleaning of agricultural product marketing data includes the following steps: Constructing a field-dependent directed graph of the agricultural product marketing data stream, treating each field in the data stream as a node, and the cleaning rule dependencies between fields as directed edges; quantifying the dependency strength by calculating the mutual information between fields and assigning it as edge weights; executing a graph theory minimum cut-driven data dependency decoupling algorithm on the field-dependent directed graph, using the minimum cut algorithm to calculate the optimal segmentation scheme, decomposing the directed graph into multiple subgraph sets with the minimum dependency strength, inserting the strong dependency rules corresponding to the cut edges into intermediate materialized nodes to achieve dependency decoupling; executing a Hamiltonian path-inspired operator fusion optimization algorithm on the cleaning operator sequence within each subgraph set, using the operators as vertices to construct a fusion candidate graph, using dynamic programming combined with pruning strategies to solve for the optimal fusion path, generating a fused super operator, and converting it into machine code using a just-in-time compiler.

[0005] The mutual information calculation in the field-dependent directed graph involves calculating the joint probability distribution and marginal probability distribution of fields X and Y. The mutual information represents the degree of correlation between the two fields, and the larger the value, the stronger the dependency. The result is normalized to the interval between 0 and 1 as the edge weight.

[0006] The execution process of the minimum cut algorithm is as follows: initialize all nodes as independent sets, iteratively perform node merging operations, select the node with the largest connection weight to the current set in each iteration, record the cut set weight in each iteration, and after the number of nodes is reduced by 1 iteration, select the cut set with the smallest weight as the optimal partitioning scheme.

[0007] Intermediate materialization nodes refer to cache nodes inserted during data stream processing. They are used to temporarily store the output results of preceding cleaning operators, breaking the original direct dependencies between fields. This allows subsequent operators to read data from intermediate materialization nodes instead of waiting for preceding operators to complete, thus decoupling the dependency chain.

[0008] Among them, the Hamiltonian path-inspired operator fusion optimization algorithm defines a fusion benefit function, which comprehensively considers the benefits of eliminating serialization overhead, improving cache locality, and reducing function call overhead. At the same time, it constrains the size of the fused code to not exceed the processor instruction cache capacity and optimizes the search space through state compression.

[0009] Among them, the super operator refers to the composite operator formed by fusing multiple lightweight cleaning operators. It includes filtering operations, mapping transformation operations, and data verification operations. The data transmission within the fused super operator does not require serialization and deserialization, but is directly transmitted through registers or level 1 cache, reducing the number of memory accesses.

[0010] The just-in-time compiler uses runtime code generation technology to convert the fused superoperator logic into machine code optimized for the current processor architecture, including vectorized instruction optimization, loop unrolling optimization, and branch prediction optimization. The compiled machine code resides in the instruction cache for repeated use.

[0011] After the superoperator is generated, a multi-level cache affinity scheduling mechanism is also established. This mechanism monitors data access patterns through hardware performance counters, binds relevant data streams to the same processor cache line, and allocates data to the local memory node of the processor where the thread resides under a non-uniform memory access architecture.

[0012] Among them, after establishing a multi-level cache affinity scheduling mechanism, it also includes setting a dynamic watermarking mechanism based on quantile statistics. The timing tolerance window is determined according to the 99th percentile delay value of the historical data delay distribution. Data arriving within the timing tolerance window is sorted by timestamp and then sent to the cleaning process for processing.

[0013] The quantile statistical method of the dynamic watermarking mechanism specifically involves collecting data delay samples within a certain time window, arranging the data delay samples in ascending order of delay duration, and the 99th percentile delay value indicating that 99% of the data delay is less than the 99th percentile delay value. The 99th percentile delay value serves as the upper bound of the time series tolerance window, and data exceeding the upper bound is judged as late data.

[0014] Among them, the hardware performance counter refers to the monitoring unit built into the processor, which records cache hit rate, cache miss rate, memory access latency and branch prediction failure rate. By reading the hardware performance counter values, the temporal and spatial locality characteristics of data access can be analyzed.

[0015] A cache line is the smallest unit of management in a processor cache, typically 64 bytes in size. Aligning relevant data to cache line boundaries helps avoid false sharing issues. When multiple threads frequently access different locations within the same cache line, it can increase the overhead of the cache coherence protocol.

[0016] Furthermore, after setting up the dynamic watermarking mechanism, when late data exceeding the dynamic watermark is detected, the incremental backtracking process is triggered. The aggregation state of the affected time window is recalculated only through the commutative half-cell algebraic structure, the cleaning results are updated and the eventual consistency is maintained.

[0017] Commutative semi-lattice algebraic structures refer to algebraic systems that satisfy the commutative, associative, and idempotent laws. In distributed aggregation computation, even if the data arrives in different orders, the final aggregation result remains consistent. Commutative semi-lattice algebraic structures include summation operations, maximum value operations, and set union operations.

[0018] Furthermore, following the incremental backtracking process, a two-stage dynamic repartitioning strategy is adopted to handle data skew. In the first stage, hot product identifiers are identified through lightweight sampling, and random suffixes are added to the hot product identifiers to achieve secondary hash partitioning. In the second stage, a work-stealing queue is deployed so that idle threads can steal tasks from busy threads through lock-free comparison and swap operations.

[0019] Furthermore, following the two-stage dynamic repartitioning strategy, it also includes establishing a hierarchical storage architecture with incremental snapshots and an embedded key-value database state backend. Frequently accessed states are kept in off-heap memory, while infrequently accessed states are sunk to the log structure merging tree. Copy-on-write technology is used to persist only incremental state change data.

[0020] This invention constructs a field-dependent directed graph and quantifies the mutual information between fields as a dependency strength indicator. It utilizes a graph-theoretic minimum cut algorithm to identify the weakest connection boundaries in the dependency network, decomposing data flow dependencies into a set of independent subgraphs with the lowest coupling. Intermediate materialized nodes are inserted at the cut edges to decouple the dependency chains, allowing the cleaning rules within each subgraph to be executed in parallel without waiting for other subgraphs to complete. For each decoupled subgraph set, this invention employs a Hamiltonian path heuristic algorithm to find the optimal concatenation path for cleaning operators, merging scattered filtering, verification, and mapping operators into a super operator. This eliminates serialization overhead and context switching loss between operators. Combined with a just-in-time compiler to generate highly optimized machine code, it enables direct data transfer at the register level, significantly reducing memory access latency. In summary, this invention solves the technical problem mentioned in the background art—the limitation of parallel processing capabilities caused by strong dependencies between fields in the real-time cleaning of agricultural product marketing data—by decomposing a strong dependency network into weakly coupled sub-networks for parallel processing. Attached Figure Description

[0021] Figure 1 This is a flowchart of the method of the present invention.

[0022] Figure 2 This is a schematic diagram of a field-dependent directed graph and its minimum cut decomposition.

[0023] Figure 3 This is a graph showing the dynamic adjustment process of the load variance. Detailed Implementation

[0024] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below.

[0025] like Figure 1 The diagram shows a flowchart of a real-time cleaning and streaming processing method for agricultural product marketing data provided by this invention. This method includes the following steps:

[0026] S01. Construct a field dependency directed graph of agricultural product marketing data flow, taking each field in the data flow as a node and the cleaning rule dependency relationship between fields as a directed edge. Quantify the dependency strength by calculating the mutual information between fields and assign it as the edge weight.

[0027] S02. Execute a graph theory minimum cut-driven data dependency decoupling algorithm on the directed graph that the field depends on. Use the Stoer-Wagner minimum cut algorithm to calculate the optimal partitioning scheme, decompose the directed graph into a set of multiple subgraphs with the smallest dependency strength, and insert the strong dependency rules corresponding to the cut edges into intermediate materialized nodes to achieve dependency decoupling.

[0028] S03. For the cleaning operator sequence in each subgraph set, execute the Hamilton path-inspired operator fusion optimization algorithm, use the operators as vertices to construct the fusion candidate graph, use dynamic programming combined with pruning strategy to solve the optimal fusion path, generate the fused super operator and convert it into machine code through just-in-time compiler.

[0029] S04. Establish a multi-level cache affinity scheduling mechanism, monitor data access patterns through hardware performance counters, bind related data streams to the same processor cache line, and allocate data to the local memory node of the processor where the thread resides under the non-consistent memory access architecture.

[0030] S05. Set up a dynamic watermarking mechanism based on quantile statistics. Determine the time-series tolerance window based on the 99th percentile delay value of the historical data delay distribution. Sort the data arriving within the time-series tolerance window by timestamp and send it into the cleaning process.

[0031] S06. When late data exceeding the dynamic watermark is detected, the incremental backtracking process is triggered. The aggregation state of the affected time window is recalculated only through the commutative half-cell algebraic structure to update the cleaning results and maintain final consistency.

[0032] S07. A two-stage dynamic repartitioning strategy is adopted to handle data skew. In the first stage, hot product identifiers are identified by lightweight sampling and a random suffix is ​​added to the hot product identifiers to achieve secondary hash partitioning. In the second stage, a work stealing queue is deployed so that idle threads can steal tasks from busy threads through lock-free comparison and swap operations.

[0033] S08. Establish a hierarchical storage architecture for incremental snapshots and embedded key-value database state backends. Keep frequently accessed states in off-heap memory and sink infrequently accessed states to the log structure merging tree. Use copy-on-write technology to persist only incremental state change data.

[0034] S09. Calculate the optimal storage redundancy of off-heap memory and log structure merged tree using the storage redundancy optimization function, and adjust the storage capacity allocation ratio of off-heap memory and log structure merged tree according to the optimal storage redundancy.

[0035] S10. Monitor the load variance of each parallel processing thread. When the load variance exceeds the preset variance threshold, dynamically adjust the random suffix generation strategy of the secondary hash partition and increase the splitting granularity of hot product identifiers until the load variance drops below the preset variance threshold.

[0036] S11. When the off-heap memory occupancy rate exceeds 80%, an asynchronous checkpoint mechanism is triggered. The background thread performs state persistence operation, and the foreground thread continues to process the newly arriving data stream through multi-version concurrency control. After persistence is completed, the state version identifier is updated.

[0037] The mutual information calculation method in the field-dependent directed graph is as follows: for field X and field Y, their joint probability distribution and marginal probability distribution are statistically analyzed. Mutual information represents the degree of correlation between the two fields. The larger the value, the stronger the dependency. The calculation formula involves the summation of probability logarithms, and the result is normalized to the interval between 0 and 1 as the edge weight.

[0038] The execution process of the Stoer-Wagner minimum cut algorithm includes: initializing all nodes as independent sets, iteratively performing node merging operations, selecting the node with the largest connection weight to the current set in each iteration, recording the cut set weight in each iteration, and after the number of nodes is reduced by 1 iteration, selecting the cut set with the smallest weight as the optimal partitioning scheme.

[0039] The intermediate materialization node is a cache node inserted during the data stream processing. It is used to temporarily store the output results of the preceding cleaning operator, break the original direct dependency relationship between fields, and enable subsequent operators to read data from the intermediate materialization node instead of waiting for the preceding operator to complete, thereby achieving decoupling of the dependency chain.

[0040] The Hamiltonian path-inspired operator fusion optimization algorithm defines a fusion benefit function, which comprehensively considers the benefits of eliminating serialization overhead, improving cache locality, and reducing function call overhead. At the same time, it constrains the fused code size to not exceed the processor instruction cache capacity and optimizes the search space through state compression.

[0041] The super operator is a composite operator formed by fusing multiple lightweight cleaning operators. It includes filtering operations, mapping transformation operations, and data verification operations. Data transmission within the fused super operator does not require serialization and deserialization; it is directly transmitted through registers or level 1 cache, reducing the number of memory accesses.

[0042] The just-in-time compiler uses runtime code generation technology to convert the fused superoperator logic into machine code optimized for the current processor architecture, including vectorized instruction optimization, loop unrolling optimization, and branch prediction optimization. The compiled machine code resides in the instruction cache for repeated use.

[0043] The hardware performance counter is a monitoring unit built into the processor, which records cache hit rate, cache miss rate, memory access latency and branch prediction failure rate. The temporal and spatial locality characteristics of data access are analyzed by reading the hardware performance counter values.

[0044] The cache behavior is the smallest management unit of the processor cache, which is usually 64 bytes in size. It aligns the relevant data to the cache line boundary to avoid the false sharing problem. When multiple threads frequently access different positions of the same cache line, it will lead to an increase in cache coherence protocol overhead.

[0045] The non-consistent memory access architecture is a memory organization method for multiprocessor systems. Each processor has a local memory node, and the latency of accessing the local memory node is lower than that of accessing the remote memory node. The latency of cross-node memory access is reduced by allocating data to the local memory node of the processor where the thread is located.

[0046] The quantile statistics method of the dynamic watermarking mechanism is as follows: collect data delay samples within a time window, arrange the data delay samples in ascending order of delay duration, and the 99th percentile delay value indicates that 99% of the data delay is less than the 99th percentile delay value. The 99th percentile delay value is used as the upper bound of the time sequence tolerance window, and data exceeding the upper bound is judged as late data.

[0047] The commutative semi-lattice algebraic structure is an algebraic system that satisfies the commutative, associative, and idempotent laws. In distributed aggregation computation, even if the data arrives in different orders, the final aggregation result remains consistent. Common commutative semi-lattice algebraic structures include summation, maximum value, and set union operations.

[0048] The lightweight sampling involves extracting sample data from the data stream at fixed intervals or with a fixed probability, and statistically analyzing the frequency of each product identifier in the sample data. Product identifiers with a frequency exceeding a preset frequency threshold are identified as hot product identifiers. The sampling ratio is typically set to 1% to 5% to balance accuracy and overhead.

[0049] The secondary hashing involves adding a random suffix to the hot product identifier and recalculating the hash value, thereby distributing the hot data that was originally mapped to the same partition to multiple partitions. The range of random suffixes is dynamically adjusted according to the degree of hotness; the higher the degree of hotness, the larger the range of random suffixes, thus achieving finer-grained load balancing.

[0050] The work-stealing queue is a double-ended queue maintained by each thread. Threads take tasks from the head of the double-ended queue to execute, and when the double-ended queue is empty, they steal tasks from the tail of the double-ended queues of other threads. Lock-free comparison and swap operations are used to avoid performance loss caused by multi-threaded competition.

[0051] The lock-free compare and swap operation is an atomic operation instruction that compares whether the value at the memory location is equal to the expected value. If they are equal, the value is updated to the new value and success is returned; otherwise, failure is returned. The entire operation is guaranteed to be atomic at the hardware level and does not require the use of a mutex lock.

[0052] The log structure merging tree is a hierarchical storage data structure. Data is first written to a memory table. When the memory table reaches a capacity threshold, it is flushed to the disk to form an ordered file. The ordered files on the disk are periodically merged and compressed. The log structure merging tree converts random writes into sequential writes, improving write throughput.

[0053] The copy-on-write technique involves creating a data copy instead of directly overwriting the original data when modifying it. The original data remains unchanged and is available for other operations to read. After the modification is complete, the pointer is updated to point to the new data copy, thus enabling concurrent execution of read and write operations.

[0054] The off-heap memory is a local memory area outside the Java Virtual Machine heap memory. It is not managed by the garbage collection mechanism, has a fast access speed and will not trigger garbage collection pauses. It is suitable for storing frequently accessed state data, but requires manual management of memory allocation and release.

[0055] The asynchronous checkpoint mechanism separates state persistence operations from data processing operations. The persistence operation is executed in an independent background thread, and the foreground thread can continue processing new data streams without waiting for the persistence operation to complete. The consistency of the read state version is ensured through multi-version concurrency control.

[0056] The multi-version concurrency control is a database concurrency control technology that maintains multiple versions for each state. Each version is associated with a timestamp or version number. Read operations read data of the corresponding version based on the timestamp, and write operations generate a new version, avoiding blocking between read and write operations and improving concurrency processing capabilities.

[0057] The load variance is an indicator that measures the load balance of each parallel processing thread. It is calculated by taking the square root of the sum of the squares of the difference between the amount of data processed by each thread and the average amount of data processed, divided by the number of threads. The smaller the load variance, the more balanced the load distribution and the higher the overall system throughput.

[0058] The storage redundancy optimization function is used to calculate the optimal storage redundancy of off-heap memory and the log structure merging tree. The inputs include the current off-heap memory occupancy rate, the current log structure merging tree occupancy rate, the average data flow arrival rate, the average state data update frequency, and the total available system memory capacity. The output is the optimal storage redundancy. The calculation method of the storage redundancy optimization function is as follows: Divide the current off-heap memory occupancy rate by its unit to obtain the normalized off-heap memory occupancy rate; divide the current log structure merging tree occupancy rate by its unit to obtain the normalized log structure merging tree occupancy rate; multiply the average data flow arrival rate by the average state data update frequency and then divide by the product of the average data flow arrival rate and the average state data update frequency to obtain the normalized data update intensity; divide the total available system memory capacity by its unit. The unit of storage capacity is the normalized total memory capacity. The optimal storage redundancy is equal to the sum of the normalized off-heap memory occupancy rate and the normalized log structure merge tree occupancy rate, divided by the normalized total memory capacity, multiplied by the square root of the normalized data update intensity, and finally multiplied by an adjustment coefficient, the adjustment coefficient ranging from 1.2 to 2.0. The optimal storage redundancy is used to adjust the storage capacity allocation ratio of off-heap memory and log structure merge tree. When the optimal storage redundancy is greater than 1.5, the proportion of off-heap memory capacity is increased to 70%, and the proportion of log structure merge tree capacity is decreased to 30%. When the optimal storage redundancy is less than 1.0, the proportion of off-heap memory capacity is decreased to 40%, and the proportion of log structure merge tree capacity is increased to 60%. When the optimal storage redundancy is between 1.0 and 1.5, the proportion of off-heap memory capacity is maintained at 55%, and the proportion of log structure merge tree capacity is maintained at 45%.

[0059] The graph theory-driven minimum cut data dependency decoupling algorithm identifies the weakest dependency boundaries between fields and divides the data flow dependency network into independent sub-networks with the lowest coupling. The cleaning rules within each sub-network are executed in parallel without waiting. Intermediate results are passed between sub-networks through lightweight intermediate materialized nodes, eliminating long-chain dependency blocking in traditional serial cleaning. This improves the parallelism, which was originally limited by the critical path, by orders of magnitude. The introduction of intermediate materialized nodes also provides natural fault-tolerant recovery points. When a sub-network fails, it only needs to be recalculated from the nearest intermediate materialized node, avoiding a complete rerun. The overall architecture presents a hierarchical pipeline topology, making full use of the parallel computing capabilities of multi-core processors and breaking through the fundamental constraint of data dependency on the throughput of streaming processing.

[0060] The Hamiltonian path-inspired operator fusion optimization algorithm finds the optimal concatenation path of cleaning operators, merging the originally scattered filtering operations, data verification operations, and mapping transformation operations into a single super operator. This eliminates the data serialization overhead and context switching overhead between operators. The data within the fused super operator is directly transferred between registers without going through the memory bus and cache levels, significantly reducing memory access latency and bandwidth consumption. The just-in-time compiler generates highly optimized machine code for the fused super operator, making full use of the vectorized instruction set and pipelined execution characteristics of modern processors. This compresses operations that originally required multiple function calls and multiple memory reads and writes into a compact instruction sequence, significantly improving the instruction cache hit rate. The branch prediction accuracy is also improved due to the improvement of code locality. The overall execution efficiency is close to that of manually optimized single functions, achieving zero-overhead abstraction of cleaning operator combination. While maintaining code modularity and maintainability, it achieves performance close to low-level optimization.

[0061] As an optional embodiment of the present invention, a real-time cleaning system for agricultural product marketing data based on a computer is also provided. The computer is provided with a readable storage medium, and the readable storage medium stores program instructions. When the program instructions are run in the computer, they execute the above-described streaming processing method for real-time cleaning of agricultural product marketing data.

[0062] The specific implementation methods of the above steps are described in detail below.

[0063] The specific implementation of step S01 is as follows: First, traverse all fields in the agricultural product marketing data stream and create a corresponding node object for each field. The node object contains the field name, field data type, and field access frequency attribute. Then, analyze the cleaning rule configuration file to identify the dependencies between fields. When a cleaning rule needs to use the processing result of field A to process field B, establish a directed edge between the node corresponding to field A and the node corresponding to field B. Next, calculate the mutual information value between each pair of fields, and count the joint occurrence frequency and independent occurrence frequency of fields X and Y in historical data. Quantify the information correlation between the two fields through the logarithmic operation of the probability distribution. Normalize the mutual information value to between 0 and 1 as the weight value of the directed edge. The closer the weight value is to 1, the higher the dependency strength between the two fields. After completing the modeling of all fields and dependencies, a field dependency directed graph is formed. The field dependency directed graph provides the input data structure for the subsequent dependency decoupling algorithm, enabling the algorithm to make graph partitioning decisions based on the quantified dependency strength.

[0064] The specific implementation of step S02 is as follows: Using the field-dependent directed graph constructed in step S01 as input, initialize each node as an independent set. Use the Stoer-Wagner minimum cut algorithm for graph partitioning. The algorithm iteratively executes a node merging process. In each iteration, select the node with the largest sum of edge weights connecting to the current node set and add it to the set. Record the sum of edge weights of the cut set in this iteration. After one iteration (minus the total number of nodes), select the cut set with the smallest sum of edge weights from all recorded cut sets as the optimal partitioning scheme. This optimal partitioning scheme divides the directed graph into multiple subgraph sets. The subgraph sets are related... To minimize dependency strength, for cross-subgraph dependency edges that still exist after segmentation, an intermediate materialized node is inserted between the starting and ending nodes of the dependency edge. The intermediate materialized node is configured with an independent memory buffer to store the output results of the preceding operators, so that subsequent operators can read data from the intermediate materialized node without waiting for the real-time calculation of the preceding operators. After inserting the intermediate materialized node, the cleaning rules in each subgraph set can be executed independently and in parallel, removing the original strong dependency constraints. The algorithm is based on the minimum cut theory in graph theory, and achieves the goal of minimizing the coupling between subgraphs by minimizing the edge weights of the cut set, laying the topological foundation for subsequent parallel processing.

[0065] The specific implementation of step S03 is as follows: For each subgraph set obtained in step S02, the cleaning operator sequence within the subgraph set is extracted, each operator is abstracted as a vertex, and the data dependency relationship and execution order constraint between operators are analyzed. When the input and output data types of two operators are compatible and there is no execution order conflict, an undirected edge is established between the corresponding vertices to represent fusion compatibility, forming a fusion candidate graph. Then, a fusion benefit function is defined, which considers three aspects: the benefit of eliminating serialization overhead, the benefit of improving cache hit rate, and the benefit of reducing the number of function calls. At the same time, a constraint condition is set requiring that the code size after fusion does not exceed the threshold of 64KB of the processor instruction cache capacity. A dynamic programming algorithm is used to solve for the optimal fusion path, and the algorithm maintains a state transition table to record the optimal fusion path for different operator combinations. The algorithm combines the benefits of different operators by encoding the set of operators into binary states using state compression technology to reduce storage space. It also uses a pruning strategy to terminate search branches with benefits below a threshold in advance. After solving the problem, the optimal fusion path is obtained, and the operators on the path are merged into a super operator. The code logic of the super operator includes all the operations of the original operators, and data transfer between operators is directly implemented through registers or L1 cache. Finally, the just-in-time compiler is called to convert the intermediate representation code of the super operator into machine code optimized for the current processor architecture. The machine code uses vectorized instruction sets to process batch data, uses loop unrolling to reduce branch judgment overhead, and uses pipelined parallelism to improve instruction execution efficiency. The algorithm is inspired by the Hamiltonian path problem and maximizes the performance of operator combination by finding the optimal path to traverse all fusionable operators.

[0066] The specific implementation of step S04 is as follows: Start the hardware performance counter to monitor the cache hit rate and memory access latency of each thread, collect performance data samples within a time window (10 to 30 seconds), analyze the sample data to identify frequently accessed data field combinations, align the memory addresses of the data field combinations to the 64-byte cache line boundary to avoid false sharing caused by different data fields occupying the same cache line, then detect the non-consistent memory access architecture topology of the current system, obtain the local memory node identifier of each processor, prioritize allocating the data processed by the thread to the local memory node of the thread-bound processor when allocating memory, set the memory allocation strategy through the memory affinity interface provided by the operating system to ensure that the proportion of threads accessing local memory nodes exceeds 90%, reducing the additional latency introduced by cross-node memory access. The multi-level cache affinity scheduling mechanism is based on the principle of spatial locality, improves cache utilization efficiency through data layout optimization, and reduces remote memory access overhead by utilizing the locality characteristics of the non-consistent memory access architecture.

[0067] The specific implementation of step S05 is as follows: continuously collect the arrival timestamp and data generation timestamp of agricultural product marketing data stream, calculate the delay duration of each data item, sort the delay duration samples in the most recent 24 hours in ascending order, and calculate the 99th percentile delay value. The 99th percentile delay value indicates that 99% of the data delay is less than the value. Set the 99th percentile delay value as the upper limit of the time-series tolerance window. The reference value range of the upper limit is 100 milliseconds to 500 milliseconds, which is dynamically adjusted according to the real-time requirements of the data stream. When new data arrives, compare the difference between its generation timestamp and the current processing time. If the difference is less than the upper limit of the time-series tolerance window, it is determined to be normal data and added to the time-series buffer. The data in the time-series buffer is sorted in ascending order by generation timestamp. Batch processing is triggered every 5 to 20 milliseconds. The sorted data is taken out from the time-series buffer and sent to the cleaning process. The dynamic watermarking mechanism is based on the quantile method in statistics. It predicts the time boundary of future data arrival through historical delay distribution and controls the waiting delay while ensuring that most data is processed in an orderly manner.

[0068] The specific implementation of step S06 is as follows: When the detected data delay exceeds the upper bound of the time-tolerance window set in step S05, the data is marked as late data, the generation timestamp of the late data and the associated time window identifier are recorded, the storage location of the aggregation state corresponding to the time window identifier is queried, the intermediate result of the current aggregation state is read, and the late data and the intermediate result are incrementally merged. The merging calculation adopts a commutative semi-lattice algebra structure to ensure that the final result remains consistent regardless of the data arrival order. For aggregations of the summation type, the value of the late data is added to the intermediate result. For aggregations of the maximum value type, the late data and the intermediate result are compared and the larger value is taken. For aggregations of the set union type, the late data is added to the intermediate result set. After the incremental merging is completed, the aggregation state storage is updated and the update timestamp is marked. The downstream processing module that depends on the result of the time window is notified to reread the latest state. The incremental backtracking process avoids recalculating all the data of the entire time window and only corrects the affected local state. Based on the mathematical properties of commutative semi-lattice algebra, it ensures eventual consistency in the distributed environment.

[0069] The specific implementation of step S07 is as follows: For every 100 data points received in the data stream, one data point is extracted as a lightweight sampling sample, with a sampling ratio set to 1%. The occurrence frequency of each product identifier in the sampling sample is counted, and the occurrence frequency of each product identifier is calculated. When the occurrence frequency of a certain product identifier exceeds a preset frequency threshold of 20%, the product identifier is marked as a hot product identifier. A random integer between 0 and 99 is generated as a random suffix for the hot product identifier. The product identifier and the random suffix are concatenated to calculate a hash value. The target processing thread is determined by taking the modulo of the hash value with the number of threads. This secondary hashing process distributes the hot data, originally concentrated in a single thread, to multiple threads. The range of the random suffix is ​​determined based on the number of threads. The frequency is dynamically adjusted. When the frequency exceeds 50%, the random suffix range is expanded to 0 to 199 to achieve a finer-grained distribution effect. At the same time, a double-ended queue is created for each processing thread as a work-stealing queue. Threads first take tasks from the head of their own double-ended queue for execution. When the double-ended queue is empty, the double-ended queues of other threads are scanned. Lock-free comparison and swap operations are used to attempt to steal tasks from the tail of other threads' double-ended queues. The comparison and swap operation guarantees atomicity at the hardware instruction level, avoiding the lock waiting overhead introduced by multi-threaded contention. The two-stage dynamic repartitioning strategy combined with the work-stealing queue achieves adaptive load balancing. The load-aware dynamic scheduling principle eliminates the impact of data skew on parallel efficiency.

[0070] The specific implementation of step S08 is as follows: Based on the access frequency of the state data, states are divided into two categories: hot states and cold states. States accessed more than 10 times per second are considered hot states, and states accessed less than once per second are considered cold states. Off-heap memory space is allocated for hot states. Off-heap memory directly maps to physical memory and is not managed by the Java Virtual Machine garbage collection mechanism. Log structure merging tree storage space is allocated for cold states. The log structure merging tree first writes data to a memory table. When the memory table size reaches 4MB to 16MB, the memory table is flushed to disk to form ordered files. The ordered files on disk are organized hierarchically. When the number of files at a certain level exceeds a threshold, merging is triggered. The operation merges multiple files into a larger file and moves it to the next level. During state updates, a copy-on-write technique is used. A copy of the state data is first created, and the modification operation is performed on the copy. After modification, an atomic pointer update operation points the reference to the new copy. The original copy is retained for ongoing read operations. During persistence, only incremental state change data is recorded, not the full data. The incremental data includes the changed key-value pairs and the change type identifier. The hierarchical storage architecture optimizes storage medium selection based on data access patterns. Hot states utilize the high-speed access characteristics of off-heap memory, while cold states utilize the efficient writing characteristics of the log structure merge tree. The copy-on-write technique enables non-blocking concurrent execution of read and write operations.

[0071] The specific implementation of step S09 is as follows: The current occupancy rate of off-heap memory and the current occupancy rate of the log structure merge tree, as described in step S08, are collected. The occupancy rate is calculated by dividing the used capacity by the total allocated capacity. Simultaneously, the average arrival rate of the data stream is collected; the average arrival rate is the number of data entries received in the last 60 seconds divided by 60 seconds. The average update frequency of the status data is collected; the average update frequency is the number of status updates in the last 60 seconds divided by the total number of statuses, then divided by 60 seconds. The total available memory capacity of the system is obtained. These five input parameters are input into a storage redundancy optimization function. The storage redundancy optimization function normalizes each input parameter and then obtains the optimal storage redundancy through weighted summation and square root operations. The optimal storage redundancy value ranges from 0.5 to 2.5. When the optimal... When the storage redundancy is greater than 1.5, it indicates that data updates are frequent and memory pressure is high. In this case, the off-heap memory capacity is increased to 70% to improve hot access efficiency, and the log structure merging tree capacity is reduced to 30%. When the optimal storage redundancy is less than 1.0, it indicates that data updates are slow and memory pressure is low. In this case, the off-heap memory capacity is reduced to 40% to reduce memory costs, and the log structure merging tree capacity is increased to 60%. When the optimal storage redundancy is between 1.0 and 1.5, a balanced configuration of 55% off-heap memory capacity and 45% log structure merging tree capacity is maintained. The storage redundancy optimization function is based on the load adaptive principle and dynamically adjusts the storage resource allocation strategy by quantifying data flow characteristics to achieve the optimal trade-off between access performance and storage cost.

[0072] The specific implementation of step S10 is as follows: Every 30 to 60 seconds, the number of data entries processed by each parallel processing thread within the statistical period is counted, the average number of data entries processed by all threads is calculated, and then the square of the difference between the number of data entries processed by each thread and the average is calculated. The sum of all squared differences is divided by the number of threads, and the square root of the result is taken to obtain the load variance. The preset variance threshold is set to 15% to 25% of the average. When the load variance exceeds the preset variance threshold, it indicates that the load distribution among threads is unbalanced. At this time, the random suffix generation strategy of the secondary hash in step S07 is adjusted, and the range of random suffixes of hot product identifiers is expanded by 50%. The original range is 0 to 99, and the expanded range is 0 to 149. Expanding the range of random suffixes increases the dispersion granularity of hot data, and more hot data is distributed to different threads for processing. The load variance is continuously monitored until it is reduced to below the preset variance threshold. The load variance monitoring and dynamic adjustment mechanism is based on the feedback control principle and drives the partitioning strategy optimization through real-time load indicators to ensure that the load of each thread is continuously balanced.

[0073] The specific implementation of step S11 is as follows: Continuously monitor the off-heap memory occupancy rate from step S08, reading the occupancy rate value every 1 to 5 seconds. When the off-heap memory occupancy rate exceeds 80%, trigger an asynchronous checkpoint mechanism, creating an independent background thread to perform state persistence operations. The background thread iterates through the hot state data in off-heap memory, reads the current version identifier and data content of each state, serializes the state data into a byte stream, and writes it to persistent storage. Solid-state drives are used for persistent storage to improve write speed. The foreground thread continues to process newly arriving data streams while the background thread performs persistence. Modification operations on the state by the foreground thread are performed using... A multi-version concurrency control mechanism is used to assign a new version number to the modified state. The new version data is written to an independent memory area, while the original version data remains unchanged for background threads to read. After the background thread completes the persistence operation, it sends a completion signal to the foreground thread. Upon receiving the signal, the foreground thread updates the persistence completion flag and the latest persistence version number, and releases the memory space occupied by the persisted old version data. The asynchronous checkpoint mechanism separates the persistence operation from the critical path, uses multi-version concurrency control to ensure data consistency, avoids the persistence process from blocking the data processing flow, and achieves a balance between high throughput and high reliability based on concurrency engineering principles.

[0074] It should be noted that the key technical ideas of this invention include two core technologies: graph theory minimum cut-driven data dependency decoupling and Hamiltonian path-inspired operator fusion optimization. The graph theory minimum cut-driven data dependency decoupling technology models the complex dependencies between fields in the data stream as a weighted directed graph. It uses the minimum cut algorithm to find the dividing boundary with the weakest dependency strength, dividing the originally intertwined dependency network into multiple loosely coupled independent sub-networks. The cleaning rules within each sub-network can be executed completely in parallel without waiting for the processing results of other sub-networks. Sub-networks exchange data lightweightly through inserted intermediate materialized nodes, eliminating the serial bottleneck caused by long-chain dependencies in traditional streaming processing. This increases the potential for parallel processing from being limited by the longest dependency path to being limited only by the number of physical processor cores. Simultaneously, intermediate materialized nodes naturally provide fault-tolerant recovery checkpoints. When a sub-network processes abnormally, it only needs to recalculate from the nearest materialized node instead of backtracking the entire process. Compared to traditional static partitioning methods, this technology achieves dynamic adaptive topology optimization based on quantitative analysis of dependency strength, maximizing the release of parallel computing capabilities while ensuring processing correctness. Hamiltonian path-inspired operator fusion optimization technology abstracts multiple lightweight operators in streaming processing as vertices of a graph. By solving for the optimal traversal path, it merges fusionable operator sequences into a super operator. Data transfer within the fused super operator is entirely completed in registers and L1 cache, eliminating the frequent serialization and deserialization overhead between operators and the data transport overhead across memory levels. The just-in-time compiler further converts the fused operator into machine code deeply optimized for the target processor architecture, making full use of vectorized instructions and pipelined parallelism. This compresses operations that originally required multiple function calls and memory accesses into compact instruction sequences, simultaneously improving instruction cache hit rate and branch prediction accuracy. Compared to traditional modular operator design, this technology achieves near-manual optimization execution efficiency while maintaining code maintainability, overcoming the performance loss introduced by abstraction levels. The synergistic effect of the two technologies is reflected in the fact that the former decouples data dependencies and releases parallel potential at the macro-topology level, while the latter optimizes operator combinations and improves single-core efficiency at the micro-execution level. The combination of macro-parallelism and micro-acceleration results in a multiplier effect on the overall system throughput. The independent sub-networks after dependency decoupling provide greater optimization space for operator fusion. The efficient operators after fusion and optimization further shorten the processing latency of the sub-networks and reduce the buffering pressure of intermediate materialized nodes. The two form a positive collaborative feedback loop. Compared with traditional streaming processing methods that only focus on single-level optimization, this invention breaks through performance bottlenecks in both architectural topology and execution efficiency through cross-layer collaborative design, achieving the goal of high-throughput and low-latency processing of real-time cleaning of agricultural product marketing data.

[0075] It should be noted that this invention also solves the following technical problem: In streaming data processing systems, out-of-order data can compromise the accuracy of aggregation calculations within a time window. Traditional fixed watermarking mechanisms either have too low a tolerance, leading to the discarding of a large amount of late data, or too high a tolerance, introducing excessively long processing delays, making it difficult to achieve a dynamic balance between data integrity and real-time performance. This invention establishes a dynamic watermarking mechanism based on quantile statistics. It adaptively adjusts the time tolerance window according to the 99th percentile of the historical data delay distribution. Data arriving within the window is sorted by timestamp and then sent to the cleaning process. When late data exceeding the dynamic watermark is detected, an incremental backtracking process is triggered. Utilizing the mathematical properties of commutative semi-lattice algebraic structures, only the aggregation state of the affected time window is recalculated, rather than a full recalculation. This minimizes the processing overhead of late data while ensuring eventual consistency, achieving dynamic coordination and optimization between data integrity and real-time processing.

[0076] Specifically, the principle of this invention is as follows: The invention solves this technical problem by quantifying the strength of dependencies between fields through mutual information, thus distinguishing between strong and weak dependencies. The minimum cut algorithm finds the optimal position to cut the dependency network from a graph theory perspective, ensuring that the subgraph sets after segmentation are connected only by the fewest possible dependency edges. After the cleaning rules corresponding to these dependency edges are decoupled through intermediate materialized nodes, the field cleaning operations within each subgraph are no longer constrained by the execution progress of other subgraphs. This allows for full utilization of the parallel computing resources of multi-core processors for simultaneous execution, overcoming the constraint of the critical path on overall throughput in traditional serial cleaning. Hamiltonian path-inspired operator fusion optimization further eliminates data transfer overhead between operators within subgraphs, compressing multiple memory reads and writes into register operations. Combined with the vectorized instruction optimization and loop unrolling techniques of the just-in-time compiler, the execution efficiency of the fused super operator approaches that of a manually optimized single-function. While maintaining code modularity, it achieves underlying optimization performance. The overall architecture presents a hierarchical pipeline topology, realizing the synergistic optimization of dependency decoupling and operator fusion.

[0077] The following provides a specific embodiment 1 of the present invention, and the specific implementation of each step in this embodiment 1 is described in detail below.

[0078] The specific implementation of step S01 is as follows: Construct a field dependency directed graph of the agricultural product marketing data flow, taking each field in the data flow as a node, and the cleaning rule dependency relationship between fields as a directed edge. Quantify the dependency strength by calculating the mutual information between fields and assign it as an edge weight. For fields... and fields mutual information The calculation formula is expressed as follows:

[0079] ;

[0080] In the formula, For fields With fields Mutual information between them, in bits; For fields Values and fields Values The joint probability of is dimensionless and ranges from 0 to 1; For fields Values The marginal probability is dimensionless and ranges from 0 to 1; For fields Values The marginal probability is dimensionless and ranges from 0 to 1; Represents logarithmic operations to the base 2, when When the value equals 0, the contribution value of the corresponding term is defined as 0. Normalized mutual information. As the edge weight, the formula is expressed as follows:

[0081] ;

[0082] In the formula, The normalized mutual information has a value range of 0 to 1 and is dimensionless. It is the normalized unit of mutual information, with a value of 1 bit; This is a normalization adjustment parameter, in bits, with an empirical value of 1.5. Joint probability. Obtained through statistical sample data, the calculation method is a field. Values and fields Values The number of samples divided by the total number of samples; marginal probability Calculation method is field Values The number of samples divided by the total number of samples; marginal probability Calculation method is field Values The number of samples divided by the total number of samples.

[0083] The specific implementation of step S02 is as follows: A graph-theoretic minimum cut-driven data dependency decoupling algorithm is executed on the field-dependent directed graph. The Stoer-Wagner minimum cut algorithm is used to calculate the optimal partitioning scheme, decomposing the directed graph into multiple sets of subgraphs with the lowest dependency strength. Strong dependency rules corresponding to cut edges are inserted into intermediate materialized nodes to achieve dependency decoupling. The execution process of the Stoer-Wagner minimum cut algorithm includes initializing all nodes as independent sets, iteratively performing node merging operations, selecting the node with the largest connection weight to the current set in each iteration, recording the cut set weight in each iteration, and after one iteration (the number of nodes minus one), selecting the cut set with the smallest weight as the optimal partitioning scheme. Intermediate materialized nodes are cache nodes inserted during data stream processing to temporarily store the output results of preceding cleaning operators, breaking the original direct dependencies between fields. This allows subsequent operators to read data from intermediate materialized nodes instead of waiting for preceding operators to complete, thus achieving dependency chain decoupling.

[0084] The specific implementation of step S03 is as follows: A Hamiltonian path-inspired operator fusion optimization algorithm is executed on the cleaning operator sequence within each subgraph set. Operators are used as vertices to construct a fusion candidate graph. Dynamic programming combined with pruning strategies is used to solve for the optimal fusion path, generating the fused super operator, which is then converted into machine code using a just-in-time compiler. Fusion profit function. The calculation formula is expressed as follows:

[0085] ;

[0086] In the formula, For the purpose of integrating benefits, dimensionless; The benefit of eliminating serialization overhead is expressed in nanoseconds; To improve the benefits of cache locality, the unit is nanoseconds; The benefit of reducing function call overhead is expressed in nanoseconds; The normalized unit for time gain is 1 nanosecond; The merged code size is in kilobytes. The reference code size is in kilobytes, used to convert time gains to the same unit as code size for easy comparison; the empirical value is 100. This is the serialization profit weighting coefficient, dimensionless, with an empirical value of 0.3; The caching revenue weighting coefficient is dimensionless and has an empirical value of 0.4. The benefit weighting coefficient is dimensionless and has an empirical value of 0.2. This is the code size penalty factor, in kilobytes, with a default value of 32. The constraint is the code size after merging. No more than the processor instruction cache capacity ,generally The value is 256 kilobytes. The superoperator is a composite operator formed by fusing multiple lightweight cleaning operators. It includes filtering operations, mapping transformation operations, and data verification operations. Data transfer within the fused superoperator does not require serialization and deserialization; it is directly transferred through registers or L1 cache, reducing memory access frequency. The just-in-time (JIT) compiler uses runtime code generation technology to convert the fused superoperator logic into machine code optimized for the current processor architecture, including vectorized instruction optimization, loop unrolling optimization, and branch prediction optimization. The compiled machine code resides in the instruction cache for repeated use.

[0087] The specific implementation of step S04 is as follows: A multi-level cache affinity scheduling mechanism is established. Data access patterns are monitored through hardware performance counters, and related data streams are bound to the same processor cache line. Under a non-consistent memory access architecture, data is allocated to the local memory node of the processor where the thread resides. The hardware performance counter is a monitoring unit built into the processor, recording cache hit rate, cache miss rate, memory access latency, and branch prediction failure rate. The temporal and spatial locality characteristics of data access are analyzed by reading the hardware performance counter values. The cache line is the smallest management unit of the processor cache, typically 64 bytes in size. Aligning related data to cache line boundaries avoids false sharing problems. When multiple threads frequently access different locations within the same cache line, it increases the overhead of the cache coherence protocol. A non-consistent memory access architecture is a memory organization method in multi-processor systems. Each processor has a local memory node, and the latency of accessing a local memory node is lower than that of accessing a remote memory node. Allocating data to the local memory node of the processor where the thread resides reduces the latency of cross-node memory access.

[0088] The specific implementation of step S05 is as follows: A dynamic watermarking mechanism based on quantile statistics is set up. A time-series tolerance window is determined according to the 99th percentile delay value of the historical data delay distribution. Data arriving within the time-series tolerance window is sorted by timestamp and then sent to the cleaning process. (99th percentile delay value) The calculation method is as follows: collection time window Data delay sample set within After sorting the samples in ascending order of delay duration, take the first... The delay value of each sample is used as upper bound of the timing tolerance window equal In the formula, This is a statistical time window, measured in seconds, and typically takes a value of 300. For the first The data latency value for each sample, in milliseconds; Let be the total number of samples, dimensionless, and require... A value greater than 100 is required to ensure statistical validity. This indicates the rounding up operation; This is the 99th percentile delay value, in milliseconds; This is the upper bound of the timing tolerance window, in milliseconds.

[0089] The specific implementation of step S06 is as follows: When late data exceeding the dynamic watermark is detected, an incremental backtracking process is triggered. Using a commutative semi-lattice algebraic structure, only the aggregation state of the affected time window is recalculated, the cleaning results are updated, and eventual consistency is maintained. A commutative semi-lattice algebraic structure is an algebraic system that satisfies the commutative, associative, and idempotent laws. In distributed aggregation computation, even if the data arrives in different orders, the final aggregation result remains consistent. Common commutative semi-lattice algebraic structures include summation, maximum value calculation, and set union operations.

[0090] The specific implementation of step S07 is as follows: A two-stage dynamic repartitioning strategy is adopted to handle data skew. In the first stage, hot-spot product identifiers are identified through lightweight sampling, and a random suffix is ​​added to these identifiers to achieve secondary hash partitioning. In the second stage, a work-stealing queue is deployed, allowing idle threads to steal tasks from busy threads through lock-free comparison and swap operations. Sampling ratio The percentage is typically set at 1% to 5%, and the criterion for determining "hot" product labels is the frequency of their appearance in the sample. Exceeding the preset frequency threshold ,default The value is 0.05. In the formula, The sampling ratio is dimensionless. The frequency of a product identifier in the sample is dimensionless and is calculated by dividing the number of times the product identifier appears in the sample by the total number of samples. Frequency threshold, dimensionless. Quadratic hash value. The calculation formula is expressed as follows:

[0091] ;

[0092] In the formula, It is a quadratic hash value, dimensionless; Numerical codes for identifying popular products, dimensionless; It is a random suffix, a dimensionless integer, and its value ranges from 0 to... ; The upper bound of the random suffix is ​​dimensionless and dynamically adjusted according to the degree of hotspot, usually taking a value of 10 to 100; The number of bits for the random suffix, dimensionless, defaults to 2; The basic hash function maps the input values ​​to partition numbers. Lightweight sampling involves extracting sample data from the data stream at fixed intervals or with a fixed probability, and statistically analyzing the frequency of each product identifier in the sample data. Product identifiers with a frequency exceeding a preset frequency threshold are identified as hot product identifiers. The sampling ratio is typically set to 1% to 5% to balance accuracy and overhead. The work-stealing queue is a double-ended queue maintained by each thread. Threads retrieve tasks from the head of the double-ended queue for execution. When the double-ended queue is empty, tasks are stolen from the tail of other threads' double-ended queues. Lock-free compare-and-swap operations are used to avoid performance loss caused by multi-threaded contention. The lock-free compare-and-swap operation is an atomic operation instruction. It compares whether the value at the memory location is equal to the expected value. If they are equal, the value is updated and success is returned; otherwise, failure is returned. The entire operation is guaranteed to be atomic at the hardware level, without the need for mutex locks.

[0093] The specific implementation of step S08 is as follows: A hierarchical storage architecture is established, consisting of incremental snapshots and an embedded key-value database state backend. Frequently accessed states are stored in off-heap memory, while infrequently accessed states are stored in a log structure merging tree. Copy-on-write technology is used to persist only incremental state change data. The log structure merging tree is a hierarchical storage data structure. Data is first written to a memory table. When the memory table reaches its capacity threshold, it is flushed to disk to form an ordered file. The ordered files on disk are periodically merged and compressed. The log structure merging tree converts random writes into sequential writes, improving write throughput. Copy-on-write technology avoids directly overwriting the original data when modifying it. Instead, a data copy is created, and modifications are performed on the copy. The original data remains unchanged for other operations to read. After modification, the pointer is updated to point to the new data copy, enabling concurrent execution of read and write operations. Off-heap memory is a local memory area outside the Java Virtual Machine heap memory. It is not managed by the garbage collection mechanism, has fast access speed, and does not trigger garbage collection pauses. It is suitable for storing frequently accessed state data, but requires manual management of memory allocation and release.

[0094] The specific implementation of step S09 is as follows: The optimal storage redundancy of off-heap memory and the merged log structure tree is calculated using a storage redundancy optimization function, and the storage capacity allocation ratio of off-heap memory and the merged log structure tree is adjusted according to the optimal storage redundancy. Storage redundancy optimization function The calculation formula is expressed as follows:

[0095] ;

[0096] In the formula, For optimal storage redundancy, dimensionless; This represents the current off-heap memory usage as a percentage. The current occupancy rate of the log structure merge tree, in percentage; This represents the average arrival rate of the data stream, measured in megabytes per second. The average update frequency of the status data, in Hertz; For reference data stream arrival rate, the unit is megabytes per second, and the empirical value is 10; The reference state data update frequency is in Hertz, with an empirical value of 100. This represents the total available memory capacity of the system, in gigabytes. For reference, the total memory capacity is in gigabytes, and the empirical value is 16. The adjustment factor is dimensionless and ranges from 1.2 to 2.0; dividing by 100 converts the percentage to a decimal between 0 and 1 to achieve dimensionlessness. When When the value is greater than 1.5, the off-heap memory capacity ratio is adjusted to 70%, and the log structure merge tree capacity ratio is adjusted to 30%; when When the value is less than 1.0, the off-heap memory capacity ratio is adjusted to 40%, and the log structure merge tree capacity ratio is adjusted to 60%; when When the value is between 1.0 and 1.5, the off-heap memory capacity accounts for 55% and the log structure merge tree capacity accounts for 45%.

[0097] The specific implementation of step S10 is as follows: Monitor the load variance of each parallel processing thread. When the load variance exceeds a preset variance threshold, dynamically adjust the random suffix generation strategy of the secondary hash partition, and increase the splitting granularity of hot product identifiers until the load variance drops below the preset variance threshold. Load Variance The calculation formula is expressed as follows:

[0098] ;

[0099] In the formula, This represents the load variance, expressed in the number of data entries. The total number of parallel processing threads, dimensionless; For the first The amount of data processed by each thread, in units of data rows; This represents the average amount of data processed, expressed in rows / data points. The calculation formula is expressed as follows:

[0100] ;

[0101] In the formula, the meanings of each parameter are the same as described above. Preset variance threshold. The empirical value is 500 data entries.

[0102] The specific implementation of step S11 is as follows: When the off-heap memory occupancy rate exceeds 80%, an asynchronous checkpointing mechanism is triggered. A background thread performs state persistence operations, while the foreground thread continues processing newly arriving data streams through multi-version concurrency control. After persistence is complete, the state version identifier is updated. The asynchronous checkpointing mechanism separates state persistence operations from data processing operations. The persistence operation is executed in an independent background thread, and the foreground thread can continue processing new data streams without waiting for the persistence operation to complete. Multi-version concurrency control ensures the consistency of the read state versions. Multi-version concurrency control is a database concurrency control technology that maintains multiple versions for each state. Each version is associated with a timestamp or version number. Read operations read data of the corresponding version based on the timestamp, and write operations generate a new version, avoiding blocking between read and write operations and improving concurrency processing capabilities.

[0103] To better understand and implement this invention, the following is a specific application scenario of this invention, Example 2:

[0104] To verify the effectiveness of this invention, technicians built a test environment and performed cleaning and processing verification by simulating the real-time transaction data stream of an agricultural e-commerce platform in a certain region. The test environment was deployed on a server configured with dual processors, each processor containing 16 physical cores, for a total of 32 physical cores. The total available memory capacity of the system was 128GB, of which 64GB was initially allocated for off-heap memory and 48GB for log structure merging tree storage space.

[0105] Technicians first collected seven consecutive days of historical transaction data from the platform. The data stream included 12 key fields such as order number, product identifier, transaction amount, transaction timestamp, user region, and logistics status. Statistical analysis revealed an average data arrival rate of 23,000 records per second, with a peak arrival rate of 58,000 records per second. The data latency distribution exhibited a clear long-tail characteristic, with 95% of the data latency within 200ms, the 99th percentile latency being 850ms, and a very small number of data points experiencing latency exceeding 2 seconds due to network fluctuations. Figure 2 As shown, the technicians constructed a field dependency directed graph containing 12 field nodes and quantified the dependency strength by calculating the mutual information between fields. The mutual information value between transaction amount and product identifier is 0.87, the mutual information value between transaction timestamp and logistics status is 0.72, and the mutual information value between user region and transaction amount is 0.45. These values ​​are assigned as weights to the corresponding edges in the directed graph.

[0106] Technicians applied the Stoer-Wagner minimum cut algorithm to the directed graph of field dependencies. After 11 iterations, the optimal partitioning scheme was determined, decomposing the original single dependency graph into three subgraph sets. The first subgraph set contains four field nodes: order number, product identifier, transaction amount, and inventory status. The second subgraph set contains four field nodes: transaction timestamp, user region, logistics status, and delivery distance. The third subgraph set contains four field nodes: product category, promotional identifier, discount amount, and payment method. The minimum cut algorithm identified a dependency edge with a weight of 0.34 between the transaction amount and the transaction timestamp. This edge was cut and inserted into an intermediate materialized node, achieving dependency decoupling between the different subgraph sets.

[0107] Within each subgraph set, engineers implemented a Hamiltonian path-inspired operator fusion optimization algorithm. For the first subgraph set, they merged four lightweight operators—null value filtering, amount range validation, product identifier format conversion, and inventory status mapping—into a single super operator. Fusion benefit calculations showed that eliminating serialization overhead saved 18 microseconds of processing time per record, improving cache locality reduced L2 cache misses by 23%, and reducing function call overhead lowered instruction cycle consumption by 12%. The fused super operator's code size was 47KB, within the processor's 64KB L1 instruction cache limit. The just-in-time (JIT) compiler converted the super operator into machine code containing vectorized SIMD instructions, reducing the processing latency per record from 76 microseconds to 34 microseconds.

[0108] Technicians monitored data access patterns using hardware performance counters and discovered that the data flow of the first subgraph set exhibited strong temporal locality during processing, with a 92% L1 cache hit rate and an 87% L2 cache hit rate. The technicians bound the processing threads for this subgraph set to cores 0 through 7 of processor 0 and allocated the relevant data flow to the local memory nodes of processor 0. Local memory access latency was reduced to 85 nanoseconds, a 63% decrease compared to the 230 nanosecond latency of cross-node remote memory access. Simultaneously, the technicians aligned the frequently accessed product identifier index table to 64-byte cache line boundaries, avoiding the false sharing problem when multiple threads access adjacent data.

[0109] Based on historical data latency distribution, technicians set the time-tolerance window for the dynamic watermarking mechanism to 850ms, which is the 99th percentile latency value. In actual operation, approximately 99% of the data arrives within the time-tolerance window and is sent to the cleaning process after being sorted by timestamp. When late data with a latency exceeding 850ms is detected, the system triggers an incremental backtracking process. Through summation operations in a commutative semi-cell algebraic structure, it recalculates only the aggregation state of the affected 5-minute time window, updates the cleaning results, and maintains eventual consistency without reprocessing the entire data stream.

[0110] Technicians identified data skew issues using lightweight sampling, setting the sampling ratio to 3%, and sampling 3 records out of every 100 records processed for statistical analysis. The sampling results showed that the best-selling product with the identifier P20241156 appeared in the data stream with a frequency as high as 17%, far exceeding the preset frequency threshold of 8%, and was therefore identified as a hot-selling product identifier. As shown in Table 1, there were significant differences in the frequency of occurrence among multiple product identifiers. Technicians added random suffixes ranging from 0 to 15 to the hot-selling product identifiers to implement secondary hashing and partitioning, distributing the hot data originally concentrated in a single partition across 16 partitions. Simultaneously, a work-stealing queue mechanism was deployed. When the double-ended queues of certain threads are empty, lock-free comparison and swap operations are used to steal tasks from the tail of the double-ended queues of busy threads, improving the overall load balancing level.

[0111] Table 1. Statistics on the frequency of product logos

[0112]

[0113] The technical staff established a hierarchical storage architecture, storing state data accessed more than 500 times per second in off-heap memory and pushing state data accessed less than 100 times per second down to the log structure merge tree. During operation, the current off-heap memory utilization rate was 62%, the log structure merge tree utilization rate was 38%, and the average update frequency of state data was 14,000 times per second. The technical staff calculated the optimal storage redundancy using a storage redundancy optimization function. The normalized off-heap memory utilization rate was 0.62, the normalized log structure merge tree utilization rate was 0.38, the normalized data update intensity was 1.53, the normalized total memory capacity was 128, and the adjustment coefficient was set to 1.6, ultimately yielding an optimal storage redundancy of 1.23. Since the optimal storage redundancy falls between 1.0 and 1.5, the technical staff maintained the default configuration of 55% off-heap memory capacity and 45% log structure merge tree capacity.

[0114] Technicians monitored the load variance of each parallel processing thread. Initially, the load variance of the 32 threads was 847, exceeding the preset variance threshold of 500. Technicians dynamically adjusted the random suffix generation strategy for the secondary hash partition, expanding the random suffix range of the hot product identifier P20241156 from 0 to 15 to 0 to 31, increasing the granularity of the split. For example... Figure 3As shown, after adjustment, the load variance gradually decreased to 423, which is lower than the preset variance threshold, and the amount of data processed by each thread tended to be balanced. When the off-heap memory utilization rate reached 83%, the system triggered an asynchronous checkpoint mechanism. The background thread performed a state persistence operation, and the foreground thread continued to process the newly arriving data stream through multi-version concurrency control. The persistence operation took about 1.8 seconds, during which the foreground thread processed more than 41,000 new data records. After persistence was completed, the state version identifier was updated.

[0115] Technicians employ copy-on-write technology to handle status update operations. When modifying product inventory status, they do not directly overwrite the original data. Instead, they create a data copy and perform the modifications on the copy. After the modifications are complete, they update the pointer to point to the new data copy. This mechanism allows read and write operations to be executed concurrently, avoiding thread blocking issues caused by traditional locking mechanisms. The log structure merge tree transforms random writes into sequential writes. Data is first written to a memory table. When the memory table reaches an 8MB capacity threshold, it is flushed to disk to form an ordered file. The ordered files on disk undergo a merge and compression operation every 30 minutes, achieving an overall write throughput of 187MB per second.

[0116] This invention breaks through the long-chain dependency blocking in traditional serial cleaning by using a graph-theoretic minimum cut-driven data dependency decoupling algorithm. It transforms the original single-threaded processing mode, constrained by the critical path, into an architecture where multiple subgraph sets execute in full parallel, eliminating the fundamental constraint of data dependencies on streaming throughput. The Hamiltonian path-inspired operator fusion optimization algorithm merges scattered cleaning operators into a super operator, allowing data to be directly transferred between registers without passing through the memory bus and cache levels, significantly reducing memory access latency and bandwidth consumption. The vectorized machine code generated by the just-in-time compiler fully utilizes the pipelined execution characteristics of modern processors. The multi-level cache affinity scheduling mechanism monitors data access patterns through hardware performance counters and binds data to processor-local memory nodes, significantly reducing the latency overhead of cross-node memory access. The dynamic watermarking mechanism and incremental backtracking processing flow, based on a commutative semi-lattice algebraic structure, only recalculate the affected time window, avoiding the huge overhead of rerunning the entire process. The two-stage dynamic repartitioning strategy, combined with lightweight sampling and a work-stealing queue, effectively solves the load unevenness problem caused by data skew, allowing the processing power of each parallel thread to be fully utilized. The tiered storage architecture dynamically adjusts the capacity allocation ratio between off-heap memory and the log structure merging tree through a storage redundancy optimization function, achieving an optimal balance between access performance and storage cost. Asynchronous checkpointing and multi-version concurrency control completely decouple state persistence from data processing, eliminating processing pauses caused by traditional synchronous persistence. The overall architecture presents a hierarchical pipeline topology, fully leveraging the parallel computing potential of multi-core processors and fundamentally overcoming the performance bottlenecks of traditional streaming processing systems in multiple dimensions, including data dependency, operator overhead, memory access, load balancing, and state management.

[0117] It should be noted that the variables involved in this invention are explained in detail in Table 2.

[0118] Table 2 Variable Explanation Table

[0119]

[0120] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention.

Claims

1. A streaming processing method for real-time cleaning of agricultural product marketing data, characterized in that, Includes the following steps: Construct a field dependency directed graph for agricultural product marketing data flow, with each field in the data flow as a node and the cleaning rule dependency relationship between fields as a directed edge. Quantify the dependency strength by calculating the mutual information between fields and assign it as the edge weight. A graph theory-driven minimum cut data dependency decoupling algorithm is applied to the field-dependent directed graph. The minimum cut algorithm is used to calculate the optimal partitioning scheme, decomposing the directed graph into a set of subgraphs with the minimum dependency strength. Strong dependency rules corresponding to cut edges are inserted into intermediate materialized nodes to achieve dependency decoupling. A Hamiltonian path-inspired operator fusion optimization algorithm is applied to the cleaning operator sequence in each subgraph set. The operators are used as vertices to construct a fusion candidate graph. Dynamic programming combined with pruning strategy is used to solve for the optimal fusion path, generating a fused super operator and converting it into machine code through a just-in-time compiler.

2. The streaming processing method for real-time cleaning of agricultural product marketing data according to claim 1, characterized in that, The mutual information calculation in the field-dependent directed graph involves calculating the joint probability distribution and marginal probability distribution of fields X and Y. The mutual information represents the degree of correlation between the two fields. The larger the value, the stronger the dependency. The result is normalized to the interval between 0 and 1 as the edge weight.

3. The streaming processing method for real-time cleaning of agricultural product marketing data according to claim 2, characterized in that, The execution process of the minimum cut algorithm is as follows: initialize all nodes as independent sets, iteratively perform node merging operations, select the node with the largest connection weight to the current set in each iteration and add it, record the cut set weight in each iteration, and after the number of nodes is reduced by 1 iteration, select the cut set with the smallest weight as the optimal partitioning scheme.

4. The streaming processing method for real-time cleaning of agricultural product marketing data according to claim 3, characterized in that, Intermediate materialized nodes refer to cache nodes inserted during data stream processing. They are used to temporarily store the output results of preceding cleaning operators, breaking the original direct dependencies between fields. This allows subsequent operators to read data from intermediate materialized nodes instead of waiting for preceding operators to complete, thus decoupling the dependency chain.

5. The streaming processing method for real-time cleaning of agricultural product marketing data according to claim 4, characterized in that, The Hamiltonian path-inspired operator fusion optimization algorithm defines a fusion benefit function that comprehensively considers the benefits of eliminating serialization overhead, improving cache locality, and reducing function call overhead. At the same time, it constrains the size of the fused code to not exceed the processor instruction cache capacity and optimizes the search space through state compression.

6. The streaming processing method for real-time cleaning of agricultural product marketing data according to claim 5, characterized in that, A super operator is a composite operator formed by fusing multiple lightweight cleaning operators. It includes filtering operations, mapping transformation operations, and data verification operations. Data transfer within the fused super operator does not require serialization and deserialization; it is directly transferred through registers or level 1 cache, reducing the number of memory accesses.

7. The streaming processing method for real-time cleaning of agricultural product marketing data according to claim 6, characterized in that, The just-in-time compiler uses runtime code generation technology to convert the fused superoperator logic into machine code optimized for the current processor architecture, including vectorized instruction optimization, loop unrolling optimization, and branch prediction optimization. The compiled machine code resides in the instruction cache for repeated use.

8. The streaming processing method for real-time cleaning of agricultural product marketing data according to claim 7, characterized in that, After the superoperator is generated, a multi-level cache affinity scheduling mechanism is also established. This mechanism monitors data access patterns through hardware performance counters, binds relevant data streams to the same processor cache line, and allocates data to the local memory node of the processor where the thread resides under a non-uniform memory access architecture.

9. The streaming processing method for real-time cleaning of agricultural product marketing data according to claim 8, characterized in that, After establishing a multi-level cache affinity scheduling mechanism, it also includes setting a dynamic watermarking mechanism based on quantile statistics. The timing tolerance window is determined according to the 99th percentile delay value of the historical data delay distribution. Data arriving within the timing tolerance window is sorted by timestamp and then sent to the cleaning process for processing.

10. The streaming processing method for real-time cleaning of agricultural product marketing data according to claim 9, characterized in that, The quantile statistical method of the dynamic watermarking mechanism specifically involves collecting data delay samples within a certain time window, arranging the data delay samples in ascending order of delay duration, and the 99th percentile delay value indicating that 99% of the data delay is less than the 99th percentile delay value. The 99th percentile delay value serves as the upper bound of the time series tolerance window, and data exceeding the upper bound is judged as late data.

Images