Asynchronous task processing method based on persistent queue and business rule conflict detection
By constructing a directed acyclic dependency topology graph and database CAS commands, the conflict dependency problem of high-concurrency writes in e-commerce data processing is solved, realizing an efficient and secure data import method that ensures data consistency and integrity.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- GUANGZHOU BATURU INFORMATION TECH CO LTD
- Filing Date
- 2026-03-23
- Publication Date
- 2026-05-26
AI Technical Summary
Existing technologies lack the ability to intelligently identify conflicting dependencies between data in high-concurrency scenarios during e-commerce data processing, leading to frequent lock contention, performance bottlenecks, and data conflicts during concurrent writes, thus failing to meet real-time requirements.
By constructing an asynchronous task processing method based on persistent queues, a resource lock identifier set is generated using business primary keys and association keys, a directed acyclic dependency topology graph is constructed, tasks are decomposed into batches using a topology hierarchical algorithm, and concurrent conflicts are handled by combining database CAS commands and exponential backoff algorithms.
It enables conflict-free parallel processing of e-commerce data import in high-concurrency scenarios, ensuring the consistency and integrity of data writing, avoiding deadlocks and performance bottlenecks, and improving data import efficiency.
Smart Images
Figure CN121880039B_ABST
Abstract
Description
Technical Field
[0001] This application belongs to the field of data processing, and in particular relates to an asynchronous task processing method based on persistent queues and business rule conflict detection. Background Technology
[0002] In e-commerce data processing, a serialized processing method is used to ensure data consistency and accuracy, i.e., parsing and persisting data sequentially. However, this method suffers from low throughput when facing massive, high-concurrency data streams, and latency leads to delayed business data updates, failing to meet the real-time requirements of e-commerce operations. Existing technical solutions often employ multi-threaded concurrent processing mechanisms, splitting large data streams and writing them to the database in parallel. However, when processing operations involving the same business primary key, the concurrent model is prone to data contention, leading to "dirty writes," data overwriting, or logical errors. Although pessimistic locking at the database level or distributed locks at the application layer can force synchronization, frequent lock contention in high-concurrency write scenarios can cause performance bottlenecks, resulting in the blocking or crashing of the entire data import service. While some improved asynchronous processing solutions utilize message queues or hash-based sharding mechanisms to alleviate concurrency pressure, they lack analysis and preprocessing of the internal dependencies within the data stream. When multiple threads simultaneously attempt to modify related data, database-level transaction conflicts or optimistic locking failures can still be triggered. When dealing with concurrent write failures, conventional solutions employ simple immediate or interval retry mechanisms. However, under high load, these mechanisms are prone to the "thundering herd effect," causing failed tasks to repeatedly impact the database. Therefore, there is currently a lack of a technical solution that can intelligently identify conflicting dependencies between data, achieve conflict-free parallelism in advance at the application layer through topology orchestration, and combine lightweight version verification and backoff strategies to handle large-scale data imports from e-commerce. Summary of the Invention
[0003] This invention proposes an asynchronous task processing method based on persistent queues and business rule conflict detection. It addresses the lack of existing technologies that can intelligently identify conflict dependencies between data, achieve conflict-free parallelism in advance at the application layer through topology orchestration, and combine lightweight version verification and backoff strategies to handle large-scale e-commerce data imports. The method includes:
[0004] Receive data import requests from e-commerce platforms, segment and parse the data stream, and generate multiple atomic data row objects containing business primary keys and association keys; extract all resource dimensions to be operated on in each atomic data row object, and construct a resource locking identifier set composed of business primary key identifiers and association key identifiers;
[0005] By detecting whether there is an intersection of resource lock identifier sets of different atomic data row objects, the write operation conflict dependency relationship between data is established; based on the conflict dependency relationship, a directed acyclic dependency topology graph is constructed, and a topology layering algorithm is used to decompose the atomic data row objects into several batches to be executed sequentially, so that the resource lock identifier sets of any two atomic data row objects in the same batch are mutually exclusive;
[0006] Batch submissions are made to the asynchronous thread pool in topological order. For each atomic data row object, the current data version number of the target write position is read, and a write instruction containing the expected version is constructed.
[0007] The database compares and exchanges CAS commands to verify whether the real-time version number of the target record is consistent with the expected version. If they are consistent, the data persistence is completed. If they are inconsistent, it is determined to be a concurrent version conflict, triggering the conflict avoidance mechanism. The exponential backoff algorithm is used to calculate the rollback delay, and the conflicting atomic data row object is re-injected into the subsequent batch of dependency building or execution process.
[0008] Furthermore, this invention also relates to an asynchronous task processing system based on persistent queues and business rule conflict detection, comprising the following modules:
[0009] The first construction module is used to receive data import requests from e-commerce platforms, parse the data stream into segments, generate multiple atomic data row objects containing business primary keys and association keys, extract all the resource dimensions to be operated on in each atomic data row object, and construct a resource locking identifier set composed of business primary key identifiers and association key identifiers;
[0010] The decomposition module is used to establish write operation conflict dependencies between data by detecting whether there is an intersection of resource lock identifier sets of different atomic data row objects; based on the conflict dependency relationship, a directed acyclic dependency topology graph is constructed, and a topology layering algorithm is used to decompose atomic data row objects into several batches of sequential execution, so that the resource lock identifier sets of any two atomic data row objects in the same batch are mutually exclusive;
[0011] The second building module is used to submit batches to the asynchronous thread pool in topological hierarchy order. For each atomic data row object, it reads the current data version number of the target write position and constructs a write instruction containing the expected version.
[0012] The calculation module is used to verify whether the real-time version number of the target record is consistent with the expected version by comparing and exchanging CAS commands in the database. If they are consistent, the data persistence is completed; if they are inconsistent, it is determined to be a concurrent version conflict, triggering the conflict avoidance mechanism, using the exponential backoff algorithm to calculate the rollback delay, and re-injecting the conflicting atomic data row object into the subsequent batch dependency building or execution process.
[0013] This invention parses data import requests from e-commerce platforms in segments, transforming the data stream into atomic data row objects. By constructing a resource locking identifier set using business primary keys and association keys, it can identify write operation conflicts and dependencies between different data sets, and build a dependency topology graph accordingly. Through a topology layering algorithm, data objects are planned into disjoint batches, ensuring that tasks within the same batch do not compete for resources. Simultaneously, optimistic concurrency control is implemented by combining database comparison and exchange instructions with version number mechanisms, guaranteeing the consistency and integrity of data writes. For occasional version conflicts, an exponential backoff algorithm is used to calculate the rollback latency and re-inject the process, alleviating database pressure while ensuring that all business data is accurately and securely persisted, solving the problems of low data import efficiency and frequent deadlocks in high-concurrency scenarios. Attached Figure Description
[0014] Figure 1 A flowchart of the first embodiment;
[0015] Figure 2 This is a schematic diagram of data sharding parsing and flattening mapping;
[0016] Figure 3 This is a diagram illustrating topology-based batch execution. Detailed Implementation
[0017] The features and exemplary embodiments of various aspects of this application will be described in detail below. To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only intended to explain this application and not to limit it. For those skilled in the art, this application can be implemented without some of these specific details. The following description of the embodiments is merely to provide a better understanding of this application by illustrating examples.
[0018] In the first embodiment, the present invention proposes an asynchronous task processing method based on persistent queues and business rule conflict detection, such as... Figure 1 ,include:
[0019] S1 receives data import requests from e-commerce platforms, segments and parses the data stream, and generates multiple atomic data row objects containing business primary keys and association keys; it extracts all resource dimensions to be operated on from each atomic data row object and constructs a resource locking identifier set composed of business primary key identifiers and association key identifiers;
[0020] Specifically, an application programming interface (API) listener endpoint is configured to receive POST requests from the Hypertext Transfer Protocol (HTTP) and read the JSON-formatted data stream from the request body. A streaming parser is used to read the data line by line to avoid memory overflow caused by loading all at once. According to a preset batch size threshold, the read data stream is divided into multiple processing units. Each record in the processing unit is traversed, and the record is mapped to an entity object instance in memory. The unique order number in the record is identified as the business primary key, and the user ID and product SKU are encapsulated as association keys into atomic data row objects with unique identifiers.
[0021] Iterate through each atomic data row object, identify all database table names and corresponding target primary key values that need to be modified for that object according to the preset business logic rules, concatenate the table names and primary key values according to a specific string format to generate a unique resource lock identifier, and store all the resource lock identifiers generated for a single object into a deduplicated hash set. This set is the resource lock identifier set for that atomic data row object.
[0022] In an optional embodiment, the step of fragmenting and parsing the data stream to generate multiple atomic data row objects containing business primary keys and association keys includes:
[0023] Parse the JSON data stream in the data import request and flatten the nested order structure;
[0024] Identify and extract the unique order number from each atomic operation record as the business primary key, and extract the user ID and product SKU as the association key;
[0025] Instantiate an atomic data row object, encapsulate the business primary key, association key and corresponding business load data into the attribute fields of the object, and store all instantiated objects in the pending queue.
[0026] A JSON parsing library is used to read large amounts of JSON data streams in a streaming manner, avoiding memory overflow caused by loading all at once. The parser locates the `orders` array node in the JSON structure and iterates through each order object. For nested order structures containing multiple product details, the parsing engine performs flattening mapping logic, such as... Figure 2 For N product SKUs under the same order, clone N copies containing the same order number and user ID. Each copy contains only a unique product SKU and the corresponding quantity and price information, thereby converting the hierarchical structure into N flat atomic operation records.
[0027] Specifically, a data transfer object named `AtomicRow` is defined, containing a `primaryKey`, a list of `associationKeys`, and a `payload`. During instantiation, the program performs necessary data cleaning, such as removing leading and trailing spaces and validating the UUID format. After instantiation, the objects are sequentially pushed into a bounded blocking queue, serving as a buffer in the producer-consumer model to ensure decoupling between parsing and processing speeds.
[0028] To transform business semantics into a unified mutual exclusion resource lock, in an optional embodiment, the step of extracting all resource dimensions to be operated on from each atomic data row object and constructing a resource lock identifier set composed of business primary key identifiers and association key identifiers includes:
[0029] Iterate through the business primary key, user ID association key, and product SKU association key in the atomic data row object;
[0030] Add preset resource type prefixes to keys of different types to form a globally unique resource identifier string;
[0031] All resource identifier strings generated from the same atomic data row object are aggregated to form the resource lock identifier set of the object.
[0032] Iterate through the key fields of each AtomicRow object and define a set of global static constant prefixes, such as PREFIX_ORDER="ORD#", PREFIX_USER="USR#", and PREFIX_SKU="ITM#". For the business primary key "1001", concatenate to generate "ORD#1001"; for the user ID "U500", concatenate to generate "USR#U500"; and for the product SKU "S_A", concatenate to generate "ITM#S_A". This prefix mechanism avoids logical confusion caused by different dimension IDs being exactly the same.
[0033] The generated resource lock identifier will be added to a HashSet. <string>Within the set, the properties of hash sets are used for deduplication to ensure that the same resource appears only once in the lock set of a single object. Each AtomicRow object will carry a read-only resourceLockSet property, which represents all critical resources involved in the data operation of that row. For example, the lock set of an atomic row object for an order X that purchased product A is {"ORD#X", "USR#User1", "ITM#A"}.
[0034] S2, by detecting whether there is an intersection of the resource lock identifier sets of different atomic data row objects, the write operation conflict dependency relationship between data is established; based on the conflict dependency relationship, a directed acyclic dependency topology graph is constructed, and a topology layering algorithm is used to decompose the atomic data row objects into several batches of sequential execution, so that the resource lock identifier sets of any two atomic data row objects in the same batch are mutually exclusive;
[0035] Specifically, a global inverted index mapping table is established, where the key is the resource lock and the identifier value is the atomic data row holding the identifier. The object ID list traverses the resource lock identifier set of all objects in the current batch and adds the object ID to the list corresponding to each resource identifier held. After the inverted index is built, each entry in the mapping table is traversed. If the list corresponding to a certain resource lock identifier contains two or more object IDs, the order of input of the objects in the original data stream is determined. It is stipulated that the later input object depends on the earlier input object, thereby establishing a directed conflict dependency edge between the later input object and the earlier input object.
[0036] Using atomic data rows as nodes in a graph, and established conflicting dependency edges as directed edges to construct a dependency graph, we calculate the in-degree of all nodes in the graph, create a zero-in-degree queue, and add all nodes with an in-degree of zero to this queue to form the first-level batch. We then logically remove all nodes from the first-level batch from the graph and traverse the outgoing edges of these nodes, reducing the in-degree of all adjacent nodes. We check the updated in-degree of adjacent nodes; if it is zero, we add them to the queue for the next-level batch. Figure 3 The process described above is repeated until all nodes in the graph are assigned to the corresponding level batch, ensuring that there are no connecting edges between nodes within the same level.
[0037] In an optional embodiment, the step of establishing write operation conflict dependencies between data by detecting whether there is an intersection of resource lock identifier sets of different atomic data row objects; and constructing a directed acyclic dependency topology graph based on the conflict dependencies, includes:
[0038] Create a resource tail pointer mapping table to record which atomic data row object used the most recent resource identifier strings;
[0039] Traverse the list of atomic data row objects in the order of receipt. For each resource identifier string in the resource lock identifier set of the current object, query the resource tail pointer mapping table.
[0040] If a record of the resource identifier string exists in the mapping table, a directed dependency edge is established between the atomic data row object pointed to by the record and the current object, with the direction pointing from the record object to the current object;
[0041] After processing all resource identifier strings of the current object, update the resource tail pointer mapping table, and update the mapping values of all resource identifier strings in the identifier set to point to the current atomic data row object.
[0042] Maintain an in-memory hash map, denoted as resourceTailMap, where the keys are resource lock identifiers and the values are references to GraphNode objects. Process atomic data rows in the queue according to their original FIFO order. Assume the current object being processed is node N, and the resource lock set is {R1, R2}. Query R1 in resourceTailMap; if a mapping relationship R1-->Node already exists... A This indicates that node A has previously operated on resource R1 and must execute before N. Therefore, a directed edge Node is created in the dependency graph. A -->Node B At the same time, increment the in-degree of node N by 1 and add N to the out-degree list of node A. Process R2 in the same way.
[0043] After completing all resource dependency checks and establishing the corresponding edges for the current node N, a tail pointer update operation is performed. The values of R1 and R2 in resourceTailMap are updated to reflect the current node N. If any new nodes also need to operate on R1 or R2, they will depend on N, not the earlier A. Through this planned construction process, a directed acyclic graph representing the write operation order is constructed in O(N×M), where N is the number of data rows and M is the average number of resources. This ensures that modifications to the same resource are executed in the order the data arrives, eliminating concurrency race conditions.
[0044] In an optional embodiment, the step of using a topology hierarchical algorithm to decompose atomic data row objects into several sequentially executed batches, such that the resource lock identifier sets of any two atomic data row objects within the same batch are mutually exclusive, includes:
[0045] Count the in-degree of each atomic data row object in the dependency topology graph;
[0046] Establish a zero-in-degree queue, and add atomic data row objects with an initial in-degree of zero to the queue as the first-level batch;
[0047] When a batch is completed or a logic is removed, traverse the successor nodes pointed to by the objects in the batch and decrement the in-degree of the successor nodes by one.
[0048] Detect nodes whose in-degree becomes zero after the update, add the nodes to the batch queue for the next round, and repeat this process until all objects in the graph have been processed.
[0049] During the initialization phase, the constructed DAG graph is scanned, and all nodes with an in-degree counter of 0 are extracted and placed into a thread-safe current batch list. Nodes with an in-degree of 0 represent nodes with no prior dependencies, therefore they do not conflict with each other and can be safely submitted to the thread pool for parallel execution, forming the first-level batch.
[0050] Once all tasks in the first batch have been completed, the hierarchical advancement logic is triggered: traverse all successor adjacent nodes of each node in the batch, atomically decrementing the in-degree of each successor node by 1. After the decrement operation, immediately check if the in-degree has become 0; if so, add the successor node to the next batch list. When all successors of the current batch have been processed, the next batch list is promoted to the current batch list, and a new round of concurrent execution begins. This loop continues until all nodes have been processed. This algorithm guarantees that while maximizing concurrency, it adheres to the execution order determined by data conflicts.
[0051] S3 submits batches to the asynchronous thread pool in topological order. For each atomic data row object, it reads the current data version number of the target write position and constructs a write instruction containing the expected version.
[0052] Specifically, CountDownLatch or CompletableFuture mechanisms are used to control the execution order between batches, ensuring that the next batch is submitted to a pre-configured fixed-size thread pool only after all previous batches have been executed. In the task execution logic, a query request is sent to the database based on the object's primary key information to obtain the current version field value of the target record as the current data version number, and an SQL update statement is constructed. The version field value is incremented by one in the SET clause, and the WHERE clause includes both a primary key matching condition and a condition that the version field equals the retrieved current data version number.
[0053] In an optional embodiment, reading the current data version number of the target write position for each atomic data row object includes:
[0054] Send a snapshot read query command to the database based on the business primary key of the atomic data row object;
[0055] If the database returns an existing record, extract the value of the version field from the record as the current data version number;
[0056] If the database does not return a record, it is determined that new data has been inserted, and the current data version number is initialized to the preset initial value.
[0057] In the task execution thread, for each AtomicRow object, a SQL query is constructed, such as SELECT version FROM order_table WHERE order_id=?. This query must be a snapshot read, typically executed under the ReadCommitted or RepeatableRead isolation level, to obtain the state of the database at the start of the current transaction.
[0058] The program receives a ResultSet returned by JDBC or an ORM framework. If the result set is not empty, it reads the integer value from the version column and temporarily stores it as the expectedVersion variable; if the result set is empty, it means that the data corresponding to the primary key does not yet exist, which is a new data creation scenario, and the program sets expectedVersion to the default initial value.
[0059] S4 uses database comparison and CAS commands to verify whether the real-time version number of the target record is consistent with the expected version. If they are consistent, data persistence is completed; if they are inconsistent, it is determined to be a concurrent version conflict, triggering the conflict avoidance mechanism. The exponential backoff algorithm is used to calculate the rollback delay, and the conflicting atomic data row object is re-injected into the dependency building or execution process of subsequent batches.
[0060] Specifically, the constructed SQL update statement is executed, and the number of affected rows returned by the database driver is obtained. If the number of affected rows is greater than zero, it indicates that the data update was successful and no concurrent modifications occurred. If the number of affected rows is zero, it is determined to be a version conflict. At this point, the cumulative number of retries for the current object is obtained, a base wait time is set, and the time interval to be waited is calculated using the truncated binary exponential backoff algorithm formula. After the current thread sleeps for this time interval, the atomic data row object is marked as pending retry and placed in a dedicated retry queue. Objects in this queue will be used as input sources to participate again in the construction of the resource lock identifier set and the analysis of dependencies during the next round of data stream parsing or batch construction.
[0061] In an optional embodiment, the step of using database comparison and CAS command exchange to verify whether the real-time version number of the target record is consistent with the expected version includes:
[0062] Send update or insert commands with conditional statements to the database;
[0063] If it is an update operation, set the SET clause to increment the version number by one, and set the WHERE clause to filter the record primary key equal to the business primary key and the record current version equal to the expected version read;
[0064] If it is an insert operation, use the database unique index constraint as a CAS mechanism, or carry version number verification logic in the insert statement;
[0065] Get the number of affected rows returned after the database executes the command. If the number of affected rows is greater than zero, the verification is considered to be consistent; otherwise, the verification is considered to be inconsistent.
[0066] For update scenarios, construct the following SQL: `UPDATE order_table SET status=?,...,version=version+1 WHERE order_id=? AND version=?`. The last question mark is filled with the `expectedVersion` obtained in the previous step. When the database engine executes this statement, it checks if the current row is still in the expected version. If so, it updates the row and returns `affected_rows` as 1. If other transactions modified the row between the time of reading and the time of update, the version number has changed, the WHERE condition does not match, and `affected_rows` will return 0.
[0067] For insert scenarios, database unique key constraints or the INSERTING RTIGNORE / ONDUPLICATE KEY UPDATE syntax are typically used. If a DuplicateKeyException is thrown during insert execution or the number of affected rows is 0, the CAS operation is considered to have failed. The program checks the updatedRows integer value returned by the JDBC driver to determine this: if updatedRows > 0, the CAS operation is considered successful and data persistence is complete; if updatedRows == 0, a concurrent version conflict has occurred, the current operation is considered a dirty write, and a rollback and retry process must be initiated, rather than overwriting existing data.
[0068] In an optional embodiment, the step of calculating the backoff latency using the exponential backoff algorithm and re-injecting conflicting atomic data row objects into subsequent batches of dependency build or execution processes includes:
[0069] Set a base wait time constant and maintain a conflict retry counter for the current object;
[0070] Calculate a power value with base 2 and exponent of the number of retries, multiply the power value by the base waiting time, and generate a backoff delay using a random jitter factor;
[0071] The task is placed in a delay queue or a timer scheduler, and the execution logic is triggered again after the rollback delay ends.
[0072] When a CAS operation fails and triggers a Concurrency Failure Exception, a retry mechanism is activated. Preset parameters include: a base wait time (BASE_DELAY) of 100ms, a maximum number of retries (MAX_RETRIES) of 5, and a maximum wait limit (MAX_DELAY) of 5000ms. The retry counter (retryCount) for the current object is incremented by 1. A rollback time is calculated using the formula: Delay = min(MAX_DELAY, BASE_DELAY × 2). retryCount )+Random(0,Jitter). The value of Jitter is a random integer between 50ms and 100ms to prevent conflicting tasks from waking up at the same time and causing another collision.
[0073] For example, the computation delay might be 435ms during the second retry. After computation, the task containing the atomic object is packaged into a DelayedTask and placed in a time wheel or delayed message queue. Once the waiting time ends, the scheduler does not insert it into the current topology level but treats it as a new arrival request, pushing it back into the pending buffer queue or the input of the dependency analyzer. It will reread the latest version number of the database and rebuild dependencies based on the current resource state, thereby ensuring data consistency. Those skilled in the art should know that a maximum number of retries can be set to prevent infinite retries. To prevent data corruption or version overwriting caused by child nodes preempting during parent node retry, optionally, all direct and indirect successor nodes of the conflicting parent node in the topology graph are recursively retrieved. Instead of decrementing the in-degree of these successor nodes, these successor nodes are removed from the current topology waiting queue through atomic operations and bound to the parent node, jointly injected into the subsequent batch dependency building process, thereby ensuring that the order of processing business data is still subject to strict topology dependency constraints in asynchronous retry scenarios.
[0074] In the second embodiment, the present invention also proposes an asynchronous task processing system based on persistent queues and business rule conflict detection, comprising the following modules:
[0075] The first construction module is used to receive data import requests from e-commerce platforms, parse the data stream into segments, generate multiple atomic data row objects containing business primary keys and association keys, extract all the resource dimensions to be operated on in each atomic data row object, and construct a resource locking identifier set composed of business primary key identifiers and association key identifiers;
[0076] The decomposition module is used to establish write operation conflict dependencies between data by detecting whether there is an intersection of resource lock identifier sets of different atomic data row objects; based on the conflict dependency relationship, a directed acyclic dependency topology graph is constructed, and a topology layering algorithm is used to decompose atomic data row objects into several batches of sequential execution, so that the resource lock identifier sets of any two atomic data row objects in the same batch are mutually exclusive;
[0077] The second building module is used to submit batches to the asynchronous thread pool in topological hierarchy order. For each atomic data row object, it reads the current data version number of the target write position and constructs a write instruction containing the expected version.
[0078] The calculation module is used to verify whether the real-time version number of the target record is consistent with the expected version by comparing and exchanging CAS commands in the database. If they are consistent, the data persistence is completed; if they are inconsistent, it is determined to be a concurrent version conflict, triggering the conflict avoidance mechanism, using the exponential backoff algorithm to calculate the rollback delay, and re-injecting the conflicting atomic data row object into the subsequent batch dependency building or execution process.
[0079] In an optional embodiment, the step of fragmenting and parsing the data stream to generate multiple atomic data row objects containing business primary keys and association keys includes:
[0080] Parse the JSON data stream in the data import request and flatten the nested order structure;
[0081] Identify and extract the unique order number from each atomic operation record as the business primary key, and extract the user ID and product SKU as the association key;
[0082] Instantiate an atomic data row object, encapsulate the business primary key, association key and corresponding business load data into the attribute fields of the object, and store all instantiated objects in the pending queue.
[0083] In an optional embodiment, the step of extracting all the resource dimensions to be operated on from each atomic data row object and constructing a resource locking identifier set composed of business primary key identifier and association key identifier includes:
[0084] Iterate through the business primary key, user ID association key, and product SKU association key in the atomic data row object;
[0085] Add preset resource type prefixes to keys of different types to form a globally unique resource identifier string;
[0086] All resource identifier strings generated from the same atomic data row object are aggregated to form the resource lock identifier set of the object.
[0087] In an optional embodiment, the step of establishing write operation conflict dependencies between data by detecting whether there is an intersection of resource lock identifier sets of different atomic data row objects; and constructing a directed acyclic dependency topology graph based on the conflict dependencies, includes:
[0088] Create a resource tail pointer mapping table to record which atomic data row object used the most recent resource identifier strings;
[0089] Traverse the list of atomic data row objects in the order of receipt. For each resource identifier string in the resource lock identifier set of the current object, query the resource tail pointer mapping table.
[0090] If a record of the resource identifier string exists in the mapping table, a directed dependency edge is established between the atomic data row object pointed to by the record and the current object, with the direction pointing from the record object to the current object;
[0091] After processing all resource identifier strings of the current object, update the resource tail pointer mapping table, and update the mapping values of all resource identifier strings in the identifier set to point to the current atomic data row object.
[0092] In an optional embodiment, the step of using a topology hierarchical algorithm to decompose atomic data row objects into several sequentially executed batches, such that the resource lock identifier sets of any two atomic data row objects within the same batch are mutually exclusive, includes:
[0093] Count the in-degree of each atomic data row object in the dependency topology graph;
[0094] Establish a zero-in-degree queue, and add atomic data row objects with an initial in-degree of zero to the queue as the first-level batch;
[0095] When a batch is completed or a logic is removed, traverse the successor nodes pointed to by the objects in the batch and decrement the in-degree of the successor nodes by one.
[0096] Detect nodes whose in-degree becomes zero after the update, add the nodes to the batch queue for the next round, and repeat this process until all objects in the graph have been processed.
[0097] In an optional embodiment, reading the current data version number of the target write position for each atomic data row object includes:
[0098] Send a snapshot read query command to the database based on the business primary key of the atomic data row object;
[0099] If the database returns an existing record, extract the value of the version field from the record as the current data version number;
[0100] If the database does not return a record, it is determined that new data has been inserted, and the current data version number is initialized to the preset initial value.
[0101] In an optional embodiment, the step of using database comparison and CAS command exchange to verify whether the real-time version number of the target record is consistent with the expected version includes:
[0102] Send update or insert commands with conditional statements to the database;
[0103] If it is an update operation, set the SET clause to increment the version number by one, and set the WHERE clause to filter the record primary key equal to the business primary key and the record current version equal to the expected version read;
[0104] If it is an insert operation, use the database unique index constraint as a CAS mechanism, or carry version number verification logic in the insert statement;
[0105] Get the number of affected rows returned after the database executes the command. If the number of affected rows is greater than zero, the verification is considered to be consistent; otherwise, the verification is considered to be inconsistent.
[0106] In an optional embodiment, the step of calculating the backoff latency using the exponential backoff algorithm and re-injecting conflicting atomic data row objects into subsequent batches of dependency build or execution processes includes:
[0107] Set a base wait time constant and maintain a conflict retry counter for the current object;
[0108] Calculate a power value with base 2 and exponent of the number of retries, multiply the power value by the base waiting time, and generate a backoff delay using a random jitter factor;
[0109] The task is placed in a delay queue or a timer scheduler, and the execution logic is triggered again after the rollback delay ends.
[0110] It should be clarified that this application is not limited to the specific configurations and processes described above and shown in the figures. For the sake of brevity, detailed descriptions of known methods are omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of this application is not limited to the specific steps described and shown. Those skilled in the art can make various changes, modifications, and additions, or change the order of steps, after understanding the spirit of this application.
[0111] The above description is merely a specific implementation of this application. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, modules, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here. It should be understood that the protection scope of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the protection scope of this application.< / string>
Claims
1. A method for asynchronous task processing based on persistent queue and business rule conflict detection, characterized in that, include: Receive data import requests from e-commerce platforms, parse the data stream into segments, and generate multiple atomic data row objects containing business primary keys and related keys; Extract all resource dimensions to be operated on from each atomic data row object, and construct a resource locking identifier set consisting of business primary key identifier and association key identifier; By detecting whether there is an intersection of resource lock identifier sets of different atomic data row objects, the write operation conflict dependency relationship between data is established; based on the conflict dependency relationship, a directed acyclic dependency topology graph is constructed, and a topology layering algorithm is used to decompose the atomic data row objects into several batches to be executed sequentially, so that the resource lock identifier sets of any two atomic data row objects in the same batch are mutually exclusive; Batch submissions are made to the asynchronous thread pool in topological order. For each atomic data row object, the current data version number of the target write position is read, and a write instruction containing the expected version is constructed. The database is used to compare and exchange CAS commands to verify whether the real-time version number of the target record is consistent with the expected version. If they are consistent, the data persistence is completed. If they are inconsistent, it is determined to be a concurrent version conflict, triggering the conflict avoidance mechanism. The exponential backoff algorithm is used to calculate the rollback delay, and the conflicting atomic data row object is re-injected into the subsequent batch of dependency building or execution process. The step involves extracting all resource dimensions to be operated on from each atomic data row object and constructing a resource locking identifier set composed of business primary key identifiers and association key identifiers, including: Iterate through the business primary key, user ID association key, and product SKU association key in the atomic data row object; Add preset resource type prefixes to keys of different types to form a globally unique resource identifier string; Aggregate all resource identifier strings generated from the same atomic data row object to form the resource lock identifier set of the object; The step involves establishing write operation conflict dependencies between data by detecting whether there is an intersection in the resource lock identifier sets of different atomic data row objects; and constructing a directed acyclic dependency topology graph based on the conflict dependencies, including: Create a resource tail pointer mapping table to record which atomic data row object used the most recent resource identifier strings; Traverse the list of atomic data row objects in the order of receipt. For each resource identifier string in the resource lock identifier set of the current object, query the resource tail pointer mapping table. If a record of the resource identifier string exists in the mapping table, a directed dependency edge is established between the atomic data row object pointed to by the record and the current object, with the direction pointing from the record object to the current object; After processing all resource identifier strings of the current object, update the resource tail pointer mapping table, and update the mapping values of all resource identifier strings in the identifier set to point to the current atomic data row object.
2. The method of claim 1, wherein, The process of segmenting and parsing the data stream generates multiple atomic data row objects containing business primary keys and association keys, including: Parse the JSON data stream in the data import request and flatten the nested order structure; Identify and extract the unique order number from each atomic operation record as the business primary key, and extract the user ID and product SKU as the association key; Instantiate an atomic data row object, encapsulate the business primary key, association key and corresponding business load data into the attribute fields of the object, and store all instantiated objects in the pending queue.
3. The method of claim 1, wherein, The method employs a topological hierarchical algorithm to decompose atomic data row objects into several sequentially executed batches, ensuring that the resource lock identifier sets of any two atomic data row objects within the same batch are mutually exclusive. This includes: Count the in-degree of each atomic data row object in the dependency topology graph; Establish a zero-in-degree queue, and add atomic data row objects with an initial in-degree of zero to the queue as the first-level batch; When a batch is completed or a logic is removed, traverse the successor nodes pointed to by the objects in the batch and decrement the in-degree of the successor nodes by one. Detect nodes whose in-degree becomes zero after the update, add the nodes to the batch queue for the next round, and repeat this process until all objects in the graph have been processed.
4. The method of claim 1, wherein, The step of reading the current data version number of the target write position for each atomic data row object includes: Send a snapshot read query command to the database based on the business primary key of the atomic data row object; If the database returns an existing record, extract the value of the version field from the record as the current data version number; If the database does not return a record, it is determined that new data has been inserted, and the current data version number is initialized to the preset initial value.
5. The method according to claim 1, characterized in that, The method of using database comparison and CAS command exchange to verify whether the real-time version number of the target record is consistent with the expected version includes: Send update or insert commands with conditional statements to the database; If it is an update operation, set the SET clause to increment the version number by one, and set the WHERE clause to filter the record primary key equal to the business primary key and the record current version equal to the expected version read; If it is an insert operation, use the database unique index constraint as a CAS mechanism, or carry version number verification logic in the insert statement; Get the number of affected rows returned after the database executes the command. If the number of affected rows is greater than zero, the verification is considered to be consistent; otherwise, the verification is considered to be inconsistent.
6. The method according to any one of claims 1-5, characterized in that, The step of calculating the backoff latency using the exponential backoff algorithm and re-injecting conflicting atomic data row objects into subsequent batch dependency build or execution flows includes: Set a base wait time constant and maintain a conflict retry counter for the current object; Calculate a power value with base 2 and exponent of the number of retries, multiply the power value by the base waiting time, and generate a backoff delay using a random jitter factor; The task is placed in a delay queue or a timer scheduler, and the execution logic is triggered again after the rollback delay ends.