Distributed asynchronous data processing method and device, computer equipment and storage medium

Abstract

The invention discloses a distributed asynchronous data processing method and device, computer equipment and a storage medium, belongs to the technical field of big data, and is applied to account checking processing scenes of insurance departure spectrums. By introducing the distributed task scheduling center and combining a parameter configuration function of the distributed task scheduling center with a dynamic adjustment mechanism of the configuration center, efficient timing scheduling and flexible management of the asynchronous tasks are realized. Firstly, by adopting a mode of querying but not frequently updating the database, the write-in pressure and the lock competition problem of the database are remarkably reduced, and the overall stability and the response speed of the system are improved. And secondly, a dynamic configuration function enables the system to adjust data processing parameters, such as the fragment quantity, the single query quantity and the processing rate, according to real-time service requirements, so that the processing capability can be quickly expanded when the data quantity suddenly increases, and the flexibility and adaptability of task scheduling are guaranteed. Meanwhile, the to-be-processed data is accurately divided by using the fragment identifiers.

Description

Technical Field

[0001] This application belongs to the field of big data technology, specifically relating to a distributed asynchronous data processing method, apparatus, computer equipment, and storage medium. Background Technology

[0002] In insurance claims automation scenarios, asynchronous processing is often used to improve response speed and decouple business processes. For large volumes of data, the data is first written to a database table, and then processed asynchronously by scanning the table. Examples include batch advance payment of medical expenses in auto insurance claims, asynchronous remote sensing image analysis in agricultural insurance claims, and asynchronous reconciliation of medical institution data in health insurance claims. However, when processing such large volumes of data, processing speed needs to be considered, requiring multiple threads or instances to process tasks simultaneously. However, simultaneous processing by multiple threads or instances may lead to data duplication.

[0003] To address the above issues, a common solution is to add an identifier field when processing the same table using multiple threads. This involves first locking a data segment (e.g., 1000 rows) by updating this field to a unique identifier, then retrieving the segment by querying the unique identifier before processing. However, if the table contains a large amount of data, numerous lock operations on data segments can lead to longer processing times, impacting data processing efficiency. In severe cases, persistent connection pool usage can prevent the database from handling normal table operations, affecting business processes. Furthermore, if the amount of data to be processed suddenly increases, limitations in data processing logic and database performance often make it difficult to achieve rapid improvements in processing speed. For example, in auto insurance claims settlement scenarios, real-time processing of massive amounts of vehicle location and damage data uploaded by IoT devices is required. Traditional asynchronous processing solutions suffer from claims delays due to database lock contention when handling millions of concurrent tasks, impacting customer experience. Summary of the Invention

[0004] The purpose of this application is to propose a distributed asynchronous data processing method, apparatus, computer equipment, and storage medium to solve the technical problems of long processing time and slow efficiency in existing data processing methods.

[0005] To address the aforementioned technical problems, this application provides a distributed asynchronous data processing method, employing the following technical solution:

[0006] A distributed asynchronous data processing method, comprising:

[0007] Obtain the configuration parameters of the preset distributed task scheduling center. The configuration parameters include a first configuration parameter and a second configuration parameter. The first configuration parameter defines the data sharding configuration, and the second configuration parameter defines the distributed data sharding processing logic.

[0008] In response to the data processing command, the system acquires the data to be processed and performs data fragmentation based on the first configuration parameters to obtain several data segments.

[0009] Obtain the business key hash value corresponding to the data to be processed, and calculate the fragment identifier of each data segment based on the business key hash value;

[0010] Each shard identifier is associated with its corresponding data segment, and the associated data segments are stored in a preset target table to be processed.

[0011] The target data fragment to be processed is determined based on the second configuration parameter, and the upper limit of the data processing volume of the target data fragment is obtained. The upper limit of the data processing volume is the number of data segments that the target data fragment can process at the same time.

[0012] Based on the upper limit of the data processing volume of the target data segment, the target data fragment corresponding to the target data segment is determined in the target table to be processed based on the first modulo operation, and the target data fragment is sent to the target data segment for processing.

[0013] To address the aforementioned technical problems, this application also provides a distributed asynchronous data processing device, which employs the following technical solution:

[0014] A distributed asynchronous data processing device, comprising:

[0015] The configuration parameter module is used to obtain the configuration parameters of the preset distributed task scheduling center. The configuration parameters include a first configuration parameter and a second configuration parameter. The first configuration parameter defines the data sharding configuration, and the second configuration parameter defines the distributed data sharding processing logic.

[0016] The data sharding module is used to respond to data processing instructions, obtain the data to be processed, and perform data sharding processing on the data to be processed based on the first configuration parameters to obtain several data fragments;

[0017] The fragment identification module is used to obtain the business key hash value corresponding to the data to be processed, and calculate the fragment identification of each data fragment based on the business key hash value.

[0018] The data storage module is used to associate each shard identifier with its corresponding data fragment and store the associated data fragments in a preset target table to be processed.

[0019] The shard matching module is used to determine the target data shard to be processed according to the second configuration parameters, and to obtain the upper limit of the data processing volume of the target data shard, wherein the upper limit of the data processing volume is the number of data segments that the target data shard can process at the same time.

[0020] The distributed processing module is used to determine the target data fragment corresponding to the target data fragment in the target data table based on the upper limit of the data processing volume of the target data fragment and the first modulo operation, and send the target data fragment to the target data fragment for processing.

[0021] To address the aforementioned technical problems, this application also provides a computer device that employs the following technical solution:

[0022] A computer device includes a memory and a processor, the memory storing computer-readable instructions, the processor executing the computer-readable instructions to implement the steps of the distributed asynchronous data processing method as described in any of the preceding claims.

[0023] To address the aforementioned technical problems, this application also provides a computer-readable storage medium, employing the technical solution described below:

[0024] A computer-readable storage medium storing computer-readable instructions, which, when executed by a processor, implement the steps of the distributed asynchronous data processing method as described in any one of the preceding descriptions.

[0025] Compared with the prior art, the embodiments of this application have the following main advantages:

[0026] This application discloses a distributed asynchronous data processing method, apparatus, computer equipment, and storage medium, belonging to the field of big data technology, and applied to the scenario of absurd insurance reconciliation processing. This application achieves efficient timed scheduling and flexible management of asynchronous tasks by introducing a distributed task scheduling center, combined with its parameter configuration function and the dynamic adjustment mechanism of the configuration center. First, by using a query-based approach instead of frequent database updates, the write pressure and lock contention issues of the database are significantly reduced, avoiding database bottlenecks and response delays caused by numerous update operations in traditional solutions, thus improving the overall stability and response speed of the system. Second, the dynamic configuration function allows the system to adjust data processing parameters according to real-time business needs, such as the number of shards, the amount of data queried per query, and the processing rate, ensuring rapid expansion of processing capacity when data volume surges, guaranteeing the flexibility and adaptability of task scheduling. Simultaneously, shard identifiers are used to accurately divide the data to be processed. This application avoids duplicate data consumption and task conflicts during multi-threaded or multi-instance concurrent processing, ensuring the accuracy and idempotency of data processing, improving the throughput efficiency of large-scale asynchronous data processing, enhancing the system's scalability and stability, effectively improving customer experience, and reducing system maintenance costs. Attached Figure Description

[0027] To more clearly illustrate the solutions in this application, the accompanying drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0028] Figure 1 An exemplary system architecture diagram is shown, in which this application can be applied;

[0029] Figure 2 A flowchart of one embodiment of the distributed asynchronous data processing method according to this application is shown;

[0030] Figure 3 A schematic diagram of the structure of one embodiment of the distributed asynchronous data processing apparatus according to this application is shown;

[0031] Figure 4 A schematic diagram of the structure of one embodiment of a computer device according to this application is shown. Detailed Implementation

[0032] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains; the terminology used herein in the specification of the application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application; the terms "comprising" and "having," and any variations thereof, in the specification, claims, and foregoing drawings of this application, are intended to cover non-exclusive inclusion. The terms "first," "second," etc., in the specification, claims, or foregoing drawings of this application are used to distinguish different objects, not to describe a particular order.

[0033] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0034] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings.

[0035] like Figure 1 As shown, system architecture 100 may include terminal device 101, network 102, and server 103. Terminal device 101 may be a laptop 1011, tablet 1012, or mobile phone 1013. Network 102 is used as a medium to provide a communication link between terminal device 101 and server 103. Network 102 may include various connection types, such as wired, wireless communication links, or fiber optic cables.

[0036] Users can use terminal device 101 to interact with server 103 via network 102 to receive or send messages, etc. Various communication client applications can be installed on terminal device 101, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social media platform software, etc.

[0037] Terminal device 101 can be various electronic devices with a display screen and support web browsing. In addition to laptops 1011, tablets 1012, or mobile phones 1013, terminal device 101 can also be an e-book reader, an MP3 player (Moving Picture Experts Group Audio Layer III), an MP4 player (Moving Picture Experts Group Audio Layer IV), a laptop computer, and a desktop computer, etc.

[0038] Server 103 can be a server that provides various services, such as a backend server that provides support for the pages displayed on terminal device 101.

[0039] It should be noted that the distributed asynchronous data processing method provided in the embodiments of this application is generally executed by a server / terminal device, and correspondingly, the distributed asynchronous data processing device is generally set in the server / terminal device.

[0040] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative; the system can have any number of terminal devices, networks, and servers depending on implementation needs.

[0041] Continue to refer to Figure 2 A flowchart illustrating an embodiment of a distributed asynchronous data processing method according to this application is shown. The distributed asynchronous data processing method includes the following steps:

[0042] S201, obtain the configuration parameters of the preset distributed task scheduling center, wherein the configuration parameters include a first configuration parameter and a second configuration parameter, the first configuration parameter defines the data sharding configuration, and the second configuration parameter defines the distributed data sharding processing logic.

[0043] Specifically, the system obtains runtime configuration parameters from a pre-defined distributed task scheduling center (such as Saturn, ElasticJob, XXL-Job, etc.) to achieve dynamic control and logic injection for task scheduling. The first configuration parameter primarily defines the data sharding strategy, typically including the maximum number of shards, the shard number of the current task, the data range or data identifier mapping rules for each shard, etc. The maximum number of shards can be distributed and hot-updated through the configuration center, allowing the system to rapidly increase its concurrent processing capacity when the task volume surges. The second configuration parameter defines the sharding processing logic, specifically including query window parameters (start time, end time, interval), the upper limit of the amount of data processed each time, the task processing retry strategy, the error tolerance threshold, and the degradation path after processing failure, etc. These parameters are uniformly managed through the configuration center, supporting task-level or business-level parameter template configuration and taking effect in real time. In technical implementation, remote configuration services (such as Apoll, Nacos, Zookeeper, etc.) or task parameter reading interfaces provided by scheduling platforms are typically used to dynamically inject these parameters into the job execution context, thereby enabling asynchronous processing tasks to have greater flexibility and controllability.

[0044] S202, respond to the data processing instruction, obtain the data to be processed, and perform data fragmentation processing on the data to be processed based on the first configuration parameters to obtain several data segments;

[0045] Specifically, after receiving data processing instructions from external sources or a timer (such as Cron triggers), the system performs initial partitioning of the batch data based on the sharding information defined in the first configuration parameter. Data acquisition generally falls into two categories: real-time incoming data, such as data pushed via message queues (Kafka, RabbitMQ, RocketMQ); and data retrieved in batches from the business database or cache. In this step, the system performs a hash operation using a business key (such as case number, user ID, order number, etc.) based on the configured maximum number of shards and the shard number of the current instance, ensuring that the data is evenly distributed across different shards. This process needs to guarantee idempotency and statelessness to avoid duplicate processing issues caused by multi-instance contention. The typical implementation involves first pulling the data into memory, performing a hash operation based on the business key and taking the modulo, and then dividing the data into several logical segments, each segment corresponding to a shard identifier. This approach significantly improves concurrency compared to traditional database locking mechanisms and offers good scalability.

[0046] S203, obtain the business key hash value corresponding to the data to be processed, and calculate the fragment identifier of each data segment based on the business key hash value;

[0047] Specifically, each piece of data is hashed based on its unique business key, and its shard identifier is calculated using the first configuration parameter (i.e., the maximum number of shards). Typically, this process uses Java's native `hashCode()` method, or a more stable hash algorithm (such as MurmurHash, CRC32, SHA-1, etc.) to ensure uniformity and consistency of distribution. The hash value is then used to determine which logical shard each piece of data belongs to through a modulo operation (`hashValue % maxShardCount`). For example, if the maximum number of shards is 8, the hash value of an order number modulo 8 might result in a shard identifier of 3. This shard identifier is not a system-level physical shard, but rather a marker for logical sharding, used for subsequent task scheduling and query condition filtering. This approach avoids row-level lock contention at the database level, achieving a static binding between data fragments and shard processors through pure computation, thus improving the system's horizontal scalability. In high-concurrency environments, consistent hashing or distributed hash ring designs can be combined to further improve the uniformity of data distribution and the fault tolerance of processing.

[0048] S204, associate each fragment identifier with its corresponding data segment, and store the associated data segments in a preset target table to be processed;

[0049] Specifically, the calculated shard identifiers are associated with each piece of data to be processed, and the data is written to the target table in a structured manner. This target table can reside in a traditional relational database (such as MySQL or PostgreSQL) or a distributed storage system (such as HBase or ClickHouse), and its structure must include metadata such as business fields, shard identifier fields, and insertion timestamps. Insertion typically uses batch insertion (such as JDBC's batchInsert) to improve efficiency, and can further reduce the pressure on a single database through database sharding strategies or primary key routing. In a distributed environment, to avoid write hotspots, the target table can be horizontally partitioned (e.g., partitioned by month), and asynchronous writing or buffer pooling techniques (such as Disruptor or RingBuffer) can be used to improve write throughput. The key to the association operation is maintaining data consistency, ensuring that the shard identifier of each piece of data is correct, and preventing omissions or duplicate processing in subsequent scheduling. In addition, basic validation, deduplication, and preprocessing operations such as format standardization and invalid field cleaning can be performed on the data before insertion.

[0050] S205, determine the target data segment to be processed according to the second configuration parameter, and obtain the upper limit of the data processing volume of the target data segment, wherein the upper limit of the data processing volume is the number of data segments that the target data segment can process at the same time;

[0051] Specifically, the task scheduling system reads the data sharding rules defined in the second configuration parameter to determine the set of shard identifiers that the current instance or thread should process. Assuming the maximum number of shards is 64, but the current configuration only allows processing with 8 shard instances, the system will calculate the range of shards each instance should process based on the correspondence between shard numbers and the maximum number of shards (e.g., shard instance number 0 processes shard identifiers 0, 8, 16,...). Furthermore, the second configuration parameter also defines the upper limit of data processing volume. This value limits the actual amount of data pulled in each scheduling session, preventing excessively long single task processing times or memory overflow due to data explosion. The upper limit of data processing volume can be dynamically adjusted through the configuration center and can be differentiated based on different shards or business types (e.g., higher priority tasks have a larger processing volume). In implementation, the scheduler needs to parse the configuration parameters and bind them to the current task context. Then, it filters the data in the target table according to the shard identifier, limiting each query to no more than the set data volume limit, and records the time window of this query for skipping already processed data in subsequent processing. This mechanism provides a controlled asynchronous processing pace, ensuring that task execution is completed within a controlled scope.

[0052] S206, based on the upper limit of the data processing volume of the target data segment, determine the target data fragment corresponding to the target data segment in the target table to be processed based on the first modulo operation, and send the target data fragment to the target data segment for processing.

[0053] Specifically, mapping logical sharding to the execution actions of actual processing tasks requires the system to filter the data subset to be processed by the shard number of the current task instance and the maximum number of shards by performing modulo matching of the business key hash values ​​of the data already stored in the target table. At the execution level, the scheduling task initiates a SELECT query from the target table, adding shard identifier filtering conditions to the WHERE clause, such as shard_tag ​​IN(0,8,16...), while limiting the query time window and data volume to a preset limit. After obtaining the target data fragment, the system encapsulates it into a processing task and pushes it to the corresponding shard processor. Common implementation methods include concurrent execution using thread pools, consumption using local blocking queues, or relaying using message queues (such as Kafka topic partitioning). Idempotency must be ensured during this process, meaning that regardless of whether the task is repeatedly scheduled or fails and is retried, the processing logic will not have side effects on the same data fragment. A task status table can also be introduced to record processing progress and exception information, enabling task tracking and backtracking. To improve efficiency, pagination + cursor scrolling queries can also be used to retrieve data, avoiding a full table scan at once. Finally, after each shard processor completes its task, it needs to write the processing status back to the database or send a processing result message.

[0054] Furthermore, before obtaining the configuration parameters of the preset distributed task scheduling center, the following steps are also included:

[0055] Configure the initial configuration parameters according to the design specifications of the distributed task scheduling center;

[0056] Parse the data processing instructions to obtain the data processing requirements, which include the maximum amount of data queried in a single data segment, the maximum amount of data processed in a data partition, and the maximum processing speed of the data partition.

[0057] The initial configuration parameters are updated according to the data processing requirements to obtain the configuration parameters.

[0058] In this embodiment, before obtaining the configuration parameters of the preset distributed task scheduling center, the system first sets initial configuration parameters according to the design specifications of the scheduling center. These initial parameters typically include the default maximum number of shards, the load balancing strategy between shards, the default value of the query time window, and the upper limit of the number of concurrent processing threads. Subsequently, the system parses the data processing instructions received from the outside, extracting the data processing requirements, such as the upper limit of the data volume in a single query (to prevent excessively large queries from affecting database performance), the upper limit of the data volume that each data shard can process (to control the load of shard tasks), and the maximum processing speed of each shard (such as the number of records processed per minute or throughput rate limit). After parsing, the system merges these dynamic parameters with the initial configuration parameters, forming the final configuration parameters through overwrite updates, merging strategies, or parameter inheritance mechanisms. The final result is stored in a structured format (such as JSON or YAML) in the configuration center or injected into the scheduling task context, preparing for precise and dynamic control of shard task scheduling and data processing.

[0059] By following the steps above, a high degree of alignment between data processing strategies and business requirements can be achieved, enhancing the system's dynamic scheduling capabilities and the configurability of the processing process, and ensuring that asynchronous processing is more efficient, accurate, and stable.

[0060] Furthermore, the data processing requirements also include the steps of single query start time and interval time, responding to data processing instructions, obtaining data to be processed, and performing data fragmentation based on the first configuration parameters to obtain several data segments, specifically including:

[0061] Parse the first configuration parameter to obtain the maximum amount of data that can be queried in a single query, the start time of a single query, and the interval time.

[0062] Calculate the single query time window based on the query start time and interval time;

[0063] Based on the upper limit of the amount of data to be processed in a single query and the time window of a single query, the data to be processed is divided into several data segments.

[0064] In this embodiment, before responding to data processing instructions and executing data sharding, the system first parses key scheduling control information in the first configuration parameters, including the upper limit of the data volume for a single query, the start time, and the query time interval. These parameters are typically issued in real time by the configuration center to ensure that scheduling behavior can adapt to different data growth rates and processing needs. Based on the parsed query start time and interval, the system dynamically calculates the query time window within the current scheduling cycle (e.g., from T to T+5 minutes), which defines the time range of data to be retrieved. Based on this, the system controls the number of data records retrieved in each round in conjunction with the upper limit of the data volume for a single query, avoiding problems such as data explosion or query timeouts. Subsequently, the system performs hash calculations and modulo operations on the retrieved data based on the business key, and maps the data to several data segments based on the maximum number of shards. Finally, each data segment has a clear time boundary, a quantity upper limit, and a shard identifier, allowing asynchronous tasks to process it in parallel according to shards, thereby realizing dual control logic in both the time and shard dimensions.

[0065] The entire asynchronous task processing flow includes core stages such as data retrieval, processing, and scheduling status maintenance. First, the system reads the dynamic parameters required for the current task execution from the asynchronous task configuration table. These parameters include the start time of the last processing, the time interval for this scheduling (e.g., 5 or 10 minutes), and the deadline. By performing logical operations on these parameters, the system determines a time window to limit the range of data to be processed in the current round. This window defines the time boundary of data generation (e.g., T to T+5min), ensuring that task scheduling has linear progression capabilities over time. Simultaneously, the system extracts the maximum number of queries per session from the configuration table to limit the upper limit of data that can be retrieved within the current scheduling cycle, preventing database overload caused by reading too much data at once.

[0066] Next, the system performs a precise query on the target table (i.e., the data table to be processed) based on a combination of three conditions: the data shard identifier (generated by hashing the business key), the time window, and the maximum number of queries, to obtain the data set that needs to be processed in this scheduling. The obtained data is then divided according to shard dimensions and handed over to the corresponding shard task instances for processing, achieving high-concurrency asynchronous processing. It is worth noting that this process is typically a loop—starting from the configured start time, each round advances by an interval until the deadline is exceeded. During this period, each round retrieves and processes data according to the above logic, ensuring integrity in the time dimension.

[0067] After completing data processing for the entire time interval, the system updates the "Start Time" and "End Time" fields in the asynchronous task configuration table to record the progress of this task execution. This ensures that when the next asynchronous task scheduling starts, the system can automatically resume processing from where it left off, guaranteeing continuous and complete task processing.

[0068] The above steps ensure that the granularity of data retrieval is controllable and the processing rhythm is adjustable, effectively alleviating database pressure and improving the time sensitivity of task scheduling.

[0069] Further, the steps of obtaining the business key hash value corresponding to the data to be processed, and calculating the fragment identifier of each data segment based on the business key hash value, specifically include:

[0070] Retrieve business key-value pairs from the data to be processed;

[0071] The business key hash value is obtained by performing a hash operation on the business key value using a hash algorithm.

[0072] Perform a second modulo operation on the business key hash value and each data fragment respectively, and obtain the result of the second modulo operation;

[0073] The fragment identifier corresponding to each data segment is determined by the result of the second modulo operation.

[0074] In this embodiment, to achieve efficient and accurate data sharding scheduling, the system first extracts key fields from the dataset to be processed to identify the uniqueness of the business, referred to as business keys (such as policy numbers, case numbers, user IDs, etc.). Next, the system performs hash calculations on the business keys using a preset hash algorithm (such as MurmurHash, FNV, or CRC32, which are lightweight and highly hashable algorithms), obtaining the corresponding business key hash value. This hash value has strong consistency and uniformity, effectively avoiding data skew issues. Subsequently, based on the configured maximum number of shards, the system performs a second modulo operation on each business key hash value (e.g., hash value % maximum number of shards) to obtain the shard number to which the data belongs, i.e., the shard identifier. Each piece of data to be processed thus clearly identifies the shard task instance to which it belongs, forming a one-to-one shard mapping relationship. This mapping relationship is completed before data is entered into the database or before scheduling processing, helping tasks to quickly and conflict-free locate processing targets in a distributed environment, avoiding runtime duplicate calculations or inconsistent sharding. Meanwhile, this logic can be extended to scenarios with dynamic addition and removal of shards, supporting on-demand expansion of processing instances and improving the overall processing capacity and flexibility of the system.

[0075] For example, in a health insurance claims system, assuming the maximum number of shards configured in the current system is 16, and the business key of a piece of data to be processed is "USER123456", its hash value is "24895327". Therefore, performing the operation 24895327%16 yields 15, so this data should be assigned to the processing unit with shard number 15. Similarly, another piece of data has a business key "USER7891011", its hash value is "13472923", and its corresponding shard number is 13472923%16 = 11. Therefore, this data should be assigned to shard 11.

[0076] This hash + modulo approach has two advantages: firstly, it ensures that identical business keys are always assigned to the same shard, which is beneficial for state consistency and idempotency; secondly, because hash values ​​are naturally relatively evenly distributed, it effectively prevents data skew and avoids overloading some shards, thus affecting the overall system throughput. Furthermore, when the system is horizontally scaled (e.g., increasing the number of shards from 16 to 32), only the modulo factor needs to be updated and recalculated, without modifying the overall logic, demonstrating good dynamic scalability and maintainability. Through this mechanism, the system can maintain high parallel processing capabilities and consistency guarantees when facing large-scale, heterogeneous business data.

[0077] By following the steps above, precise binding between data and sharding tasks can be achieved, effectively avoiding duplicate data processing and sharding conflicts, and improving concurrent processing efficiency.

[0078] Furthermore, the steps of associating each shard identifier with its corresponding data segment and storing the associated data segments in a preset target table to be processed specifically include:

[0079] Identify the data segments corresponding to each shard identifier, and associate and match each shard identifier with its corresponding data segments;

[0080] A consistent hash ring structure is used to construct the hash ring of the target table to be processed;

[0081] Each shard identifier is mapped to a hash ring, and the storage location of the associated data fragment is determined based on the node position of each shard identifier in the hash ring.

[0082] Based on the storage location of the associated data fragments, the associated data fragments are stored in a preset target table to be processed.

[0083] In this embodiment, after calculating the business key hash and shard identifier, the system needs to associate each shard identifier with its corresponding data fragment and store them appropriately in the target table to be processed. To improve the balance and flexibility of shard mapping, the system introduces a consistent hash ring structure as the core of the data distribution strategy. Specifically, firstly, the set of data fragments corresponding to each shard identifier is identified, and a mapping relationship is established between the identifier value and the data fragment. Subsequently, the system constructs a virtual hash ring, mapping all possible shard identifiers to fixed node positions on the hash ring. For example, when a maximum of 64 shards are supported, 64 virtual nodes will be evenly distributed on the hash ring, with each node corresponding to a shard number. By performing a secondary hash mapping of the shard identifier to the hash ring, the system can accurately determine the data storage area or logical partition to which the shard identifier should belong based on its position on the hash ring, thereby ensuring that data fragments are stored reasonably and evenly in the target table to be processed. Compared with the traditional static partitioning strategy, this method has higher adaptability in terms of scalability and load balancing, and is especially suitable for scenarios with uneven data distribution or dynamically adjusted shard numbers. At the same time, this design also provides structural compatibility and migration convenience for the future introduction of distributed databases (such as ShardingSphere, TiDB, etc.).

[0084] Through the above steps, accurate location and efficient storage of sharded data are achieved, enhancing the system's scalability and load balancing capabilities in a distributed environment.

[0085] Furthermore, before the step of storing the associated data fragment to a preset target table based on the storage location of the associated data fragment, the method further includes:

[0086] Monitor the capacity threshold status of each storage node on the hash ring;

[0087] If the node's storage capacity exceeds a preset threshold, a load balancing mechanism is triggered to obtain a backup node with a lower load.

[0088] The routing rule table is updated based on the information of the candidate nodes, and the storage location of the data fragments is reallocated using the least connection number algorithm to obtain the new storage location;

[0089] Based on the new storage location, the associated data fragments are stored in the preset target table to be processed.

[0090] In this embodiment, to ensure the stability and sustainability of data sharding under high concurrency and massive write scenarios, the system adds a dynamic load balancing mechanism before formally writing associated data fragments to the target table to be processed. Specifically, the system monitors the capacity status of each storage node (logical partition or physical database instance) in the hash ring in real time, including the current amount of stored data, write rate, and number of connections. When the storage capacity or number of access connections of a node exceeds a preset load threshold (e.g., CPU, I / O, or connection pool resource usage exceeds 80%), the system triggers the load balancing module to automatically execute a node transfer strategy. This strategy prioritizes selecting nodes with lower loads from the available candidate nodes as new candidate write targets. Subsequently, the system updates the internal routing rule table according to the minimum connection count algorithm or response latency metric, dynamically adjusting the hash mapping relationship so that data fragments that should have fallen on high-load nodes are redistributed to low-load nodes. This process does not require stopping the current write task, has real-time performance and transparency, and avoids problems such as write failures and processing blockages caused by node overload. Finally, the system writes the data fragments to the corresponding nodes according to the updated storage location, ensuring reasonable data distribution and stable processing, while providing accurate routing information for the system's data query and processing of fragments.

[0091] Through the above steps, the system achieves real-time load awareness and dynamic adjustment capabilities during the data writing process, effectively alleviating the pressure on a single node and improving the overall throughput and availability of the system.

[0092] Furthermore, the steps of determining the target data segment corresponding to the target data segment in the target table based on the first modulo operation, according to the upper limit of the data processing volume of the target data segment, and sending the target data segment to the target data segment for processing, specifically include:

[0093] Calculate the number of target data fragments, and configure the first modulo operation rule based on the number of target data fragments and the upper limit of data processing volume;

[0094] According to the first modulo operation rule, the data fragments in the target table to be processed are subjected to modulo operation to obtain the first modulo operation result, wherein the first modulo operation result is the combination of data fragments in the target table to be processed;

[0095] Based on the result of the first modulo operation, the combination of data fragments in the target table to be processed is assigned to the matching target data fragment.

[0096] In this embodiment, to achieve efficient distribution and balanced execution of data processing tasks, the system needs to dynamically configure reasonable modulo operation rules based on the number of target data shards and the processing capacity of each shard (i.e., the upper limit of data processing volume), thereby determining the precise mapping relationship between data fragments and shard instances. First, the system formulates a first modulo operation rule based on the number of currently active target shards (e.g., 8, 16, or 64) obtained from the configuration center or task scheduling platform (e.g., Saturday), combined with the upper limit of the amount of data each shard can process per unit time (e.g., a maximum of 1000 data entries per round). This rule is generally based on the business key hash value or the allocated shard identifier field for modulo operation (e.g., shard identifier % active shard count), thereby dividing the data to be processed into multiple data fragment combinations.

[0097] Subsequently, the system iterates through the target table to be processed, performing the first modulo operation on each piece of data according to the aforementioned modulo rules to obtain the target data shard number corresponding to each piece of data. Data with consistent modulo results are aggregated into a group, forming a data fragment combination, and this combination is assigned to the corresponding target processing instance. For example, if 8 shards are currently enabled, data identified as 0, 8, 16, ... are processed by shard 0, data identified as 1, 9, 17, ... are processed by shard 1, and so on. This strategy not only ensures a reasonable and balanced distribution of data among multiple shard tasks but also avoids data duplication and contention during concurrent processing. Furthermore, the system supports dynamic scaling; when the number of target shards changes, only the modulo factor needs to be adjusted for a quick response, without altering the original logic, maintaining high flexibility and maintainability.

[0098] By following the steps above, dynamic and balanced distribution of data tasks can be achieved, improving system throughput while reducing the risk of resource contention between threads / instances.

[0099] In the above embodiments, this application discloses a distributed asynchronous data processing method, belonging to the field of big data technology, and applied to the scenario of absurd insurance reconciliation processing. This application introduces a distributed task scheduling center, combining its parameter configuration function with the dynamic adjustment mechanism of the configuration center, to achieve efficient timed scheduling and flexible management of asynchronous tasks. First, by using a query-based approach instead of frequent database updates, the write pressure and lock contention issues of the database are significantly reduced, avoiding database bottlenecks and response delays caused by numerous update operations in traditional solutions, thus improving the overall stability and response speed of the system. Second, the dynamic configuration function allows the system to adjust data processing parameters according to real-time business needs, such as the number of shards, the amount of data queried per query, and the processing rate, ensuring rapid expansion of processing capacity when data volume surges, guaranteeing the flexibility and adaptability of task scheduling. Simultaneously, shard identifiers are used to accurately divide the data to be processed. This application avoids duplicate data consumption and task conflicts during multi-threaded or multi-instance concurrent processing, ensuring the accuracy and idempotency of data processing, improving the throughput efficiency of large-scale asynchronous data processing, enhancing the system's scalability and stability, effectively improving customer experience, and reducing system maintenance costs.

[0100] In this embodiment, the distributed asynchronous data processing method runs on an electronic device (e.g., Figure 1 The server shown can receive instructions or acquire data via wired or wireless connection. It should be noted that the aforementioned wireless connection methods may include, but are not limited to, 3G / 4G connections, WiFi connections, Bluetooth connections, WiMAX connections, Zigbee connections, UWB (ultra-wideband) connections, and other currently known or future wireless connection methods.

[0101] It should be emphasized that, in order to further ensure the privacy and security of the aforementioned data to be processed, the aforementioned data to be processed can also be stored in a node of a blockchain.

[0102] The blockchain referred to in this application is a novel application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanisms, and encryption algorithms. Essentially, a blockchain is a decentralized database, a chain of data blocks linked together using cryptographic methods. Each data block contains information about a batch of network transactions, used to verify the validity of the information (anti-counterfeiting) and generate the next block. A blockchain can include an underlying blockchain platform, a platform product service layer, and an application service layer.

[0103] The embodiments of this application can acquire and process relevant data based on artificial intelligence technology. Artificial intelligence (AI) is the theory, method, technology, and application system that uses digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results.

[0104] Foundational technologies for artificial intelligence generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies mainly encompass computer vision, robotics, biometrics, speech processing, natural language processing, and machine learning / deep learning.

[0105] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by instructing related hardware through computer-readable instructions. These computer-readable instructions can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the methods described above. The aforementioned storage medium can be a non-volatile storage medium such as a magnetic disk, optical disk, or read-only memory (ROM), or random access memory (RAM).

[0106] It should be understood that although the steps in the flowcharts of the accompanying figures are shown sequentially as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the accompanying figures may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0107] Further reference Figure 3 As a response to the above Figure 2 The implementation of the method shown in this application provides an embodiment of a distributed asynchronous data processing device, which is similar to... Figure 2 Corresponding to the method embodiments shown, this device can be specifically applied to various electronic devices.

[0108] like Figure 3 As shown, the distributed asynchronous data processing device 300 described in this embodiment includes:

[0109] The configuration parameter module 301 is used to obtain the configuration parameters of the preset distributed task scheduling center. The configuration parameters include a first configuration parameter and a second configuration parameter. The first configuration parameter defines the data sharding configuration, and the second configuration parameter defines the distributed data sharding processing logic.

[0110] The data sharding module 302 is used to respond to data processing instructions, obtain the data to be processed, and perform data sharding processing on the data to be processed based on the first configuration parameters to obtain several data fragments;

[0111] The fragment identification module 303 is used to obtain the business key hash value corresponding to the data to be processed, and calculate the fragment identification of each data fragment based on the business key hash value.

[0112] The data storage module 304 is used to associate each fragment identifier with its corresponding data segment and store the associated data segments in a preset target table to be processed.

[0113] The shard matching module 305 is used to determine the target data shard to be processed according to the second configuration parameters, and to obtain the upper limit of the data processing volume of the target data shard, wherein the upper limit of the data processing volume is the number of data segments that the target data shard can process at the same time.

[0114] The distributed processing module 306 is used to determine the target data segment corresponding to the target data segment in the target table based on the upper limit of the data processing volume of the target data segment and the first modulo operation, and send the target data segment to the target data segment for processing.

[0115] Furthermore, the distributed asynchronous data processing device 300 also includes:

[0116] The initial configuration module is used to configure initial configuration parameters according to the design specifications of the distributed task scheduling center.

[0117] The instruction parsing module is used to parse data processing instructions and obtain data processing requirements. These requirements include the maximum amount of data that can be queried in a single data segment, the maximum amount of data that can be processed in a data piece, and the maximum processing speed of the data piece.

[0118] The configuration update module is used to update the initial configuration parameters according to data processing requirements to obtain the configuration parameters.

[0119] Furthermore, the data processing requirements also include the start time and interval of a single query. The data sharding module 302 specifically includes:

[0120] The configuration parsing unit is used to parse the first configuration parameters and obtain the maximum amount of data in a single query, the start time of a single query, and the interval time of the data segment.

[0121] The time window unit is used to calculate the time window for a single query based on the query start time and the interval time.

[0122] The data sharding unit is used to shard the data to be processed according to the upper limit of the amount of data in a single query and the time window of a single query, so as to obtain several data fragments.

[0123] Furthermore, the fragment identification module 303 specifically includes:

[0124] Business key-value unit, used to retrieve business key values ​​from the data to be processed;

[0125] The hash operation unit is used to perform hash operations on the business key value using a hash algorithm to obtain the business key hash value;

[0126] The second modulo operation unit is used to perform a second modulo operation on the business key hash value and each data fragment respectively, and obtain the second modulo operation result;

[0127] The fragment identifier is determined by the result of the second modulo operation, which identifies the fragment identifier corresponding to each data segment.

[0128] Furthermore, the data storage module 304 specifically includes:

[0129] The association matching unit is used to identify the data segments corresponding to each shard identifier and to associate and match each shard identifier with its corresponding data segments;

[0130] Hash ring building unit, used to construct a hash ring for the target table to be processed using a consistent hash ring structure;

[0131] The identifier mapping unit is used to map each fragment identifier to the hash ring, and determine the storage location of the associated data fragment based on the node position of each fragment identifier in the hash ring.

[0132] The data storage unit is used to store associated data fragments into a preset target table to be processed based on the storage location of the associated data fragments.

[0133] Furthermore, the data storage module 304 also includes:

[0134] The capacity monitoring unit is used to monitor the capacity threshold status of each storage node on the hash ring.

[0135] The load balancing unit is used to trigger the load balancing mechanism and obtain alternative nodes with lower load if the storage capacity of a node exceeds a preset threshold.

[0136] The reallocation unit is used to update the routing rule table based on the information of the candidate nodes and to reallocate the storage location of the data fragments using the least connection number algorithm to obtain the new storage location.

[0137] The restorage unit is used to store associated data fragments to a preset target table for processing based on a new storage location.

[0138] Furthermore, the distributed processing module 306 specifically includes:

[0139] The modulo operation rule configuration unit is used to calculate the number of target data fragments and configure the first modulo operation rule according to the number of target data fragments and the upper limit of data processing volume.

[0140] The first modulo operation unit is used to perform modulo operation on the data fragments in the target table to be processed according to the first modulo operation rule, and obtain the first modulo operation result, wherein the first modulo operation result is a combination of the data fragments in the target table to be processed;

[0141] The combination allocation unit is used to allocate the combination of data fragments in the target table to the matching target data fragments according to the result of the first modulo operation.

[0142] In the above embodiments, this application discloses a distributed asynchronous data processing device, belonging to the field of big data technology, and applied to the scenario of reconciliation processing in insurance companies with outrageous claims. This application introduces a distributed task scheduling center, combining its parameter configuration function with the dynamic adjustment mechanism of the configuration center, to achieve efficient timed scheduling and flexible management of asynchronous tasks. First, by using a query-based approach instead of frequent database updates, the write pressure and lock contention issues of the database are significantly reduced, avoiding database bottlenecks and response delays caused by numerous update operations in traditional solutions, thus improving the overall stability and response speed of the system. Second, the dynamic configuration function allows the system to adjust data processing parameters according to real-time business needs, such as the number of shards, the amount of data queried per query, and the processing rate, ensuring rapid expansion of processing capacity when data volume surges, guaranteeing the flexibility and adaptability of task scheduling. Simultaneously, shard identifiers are used to accurately divide the data to be processed. This application avoids duplicate data consumption and task conflicts during multi-threaded or multi-instance concurrent processing, ensuring the accuracy and idempotency of data processing, improving the throughput efficiency of large-scale asynchronous data processing, enhancing the system's scalability and stability, effectively improving customer experience, and reducing system maintenance costs.

[0143] To address the aforementioned technical problems, embodiments of this application also provide a computer device. Please refer to [link / reference needed] for details. Figure 4 , Figure 4 This is a basic structural block diagram of the computer device in this embodiment.

[0144] The computer device 4 includes a memory 41, a processor 42, and a network interface 43 that are interconnected via a system bus. It should be noted that only the computer device 4 with memory 41, processor 42, and network interface 43 is shown in the figure; however, it should be understood that it is not required to implement all the components shown, and more or fewer components can be implemented alternatively. Those skilled in the art will understand that the computer device described here is a device capable of automatically performing numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes, but is not limited to, microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital signal processors (DSPs), embedded devices, etc.

[0145] The computer device can be a desktop computer, laptop, handheld computer, or cloud server, etc. The computer device can interact with the user via a keyboard, mouse, remote control, touchpad, or voice control.

[0146] The memory 41 includes at least one type of readable storage medium, including flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 41 may be an internal storage unit of the computer device 4, such as the hard disk or memory of the computer device 4. In other embodiments, the memory 41 may also be an external storage device of the computer device 4, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the computer device 4. Of course, the memory 41 may also include both the internal storage unit and its external storage device of the computer device 4. In this embodiment, the memory 41 is typically used to store the operating system and various application software installed on the computer device 4, such as computer-readable instructions for distributed asynchronous data processing methods. In addition, the memory 41 can also be used to temporarily store various types of data that have been output or will be output.

[0147] In some embodiments, the processor 42 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chip. The processor 42 is typically used to control the overall operation of the computer device 4. In this embodiment, the processor 42 is used to execute computer-readable instructions stored in the memory 41 or to process data, for example, to execute computer-readable instructions of the distributed asynchronous data processing method.

[0148] The network interface 43 may include a wireless network interface or a wired network interface, which is typically used to establish communication connections between the computer device 4 and other electronic devices.

[0149] This application also provides an implementation method, namely, a computer device including a memory and a processor. The memory stores computer-readable instructions, and when the processor executes the computer-readable instructions, it implements the steps of the distributed asynchronous data processing method described above, that is, it implements:

[0150] A distributed asynchronous data processing method, comprising:

[0151] Obtain the configuration parameters of the preset distributed task scheduling center. The configuration parameters include a first configuration parameter and a second configuration parameter. The first configuration parameter defines the data sharding configuration, and the second configuration parameter defines the distributed data sharding processing logic.

[0152] In response to the data processing command, the system acquires the data to be processed and performs data fragmentation based on the first configuration parameters to obtain several data segments.

[0153] Obtain the business key hash value corresponding to the data to be processed, and calculate the fragment identifier of each data segment based on the business key hash value;

[0154] Each shard identifier is associated with its corresponding data segment, and the associated data segments are stored in a preset target table to be processed.

[0155] The target data fragment to be processed is determined based on the second configuration parameter, and the upper limit of the data processing volume of the target data fragment is obtained. The upper limit of the data processing volume is the number of data segments that the target data fragment can process at the same time.

[0156] Based on the upper limit of the data processing volume of the target data segment, the target data fragment corresponding to the target data segment is determined in the target table to be processed based on the first modulo operation, and the target data fragment is sent to the target data segment for processing.

[0157] This application also provides another implementation, namely, a computer-readable storage medium storing computer-readable instructions that can be executed by at least one processor to cause the at least one processor to perform the steps of the distributed asynchronous data processing method described above, i.e., to implement:

[0158] A distributed asynchronous data processing method, comprising:

[0159] Obtain the configuration parameters of the preset distributed task scheduling center. The configuration parameters include a first configuration parameter and a second configuration parameter. The first configuration parameter defines the data sharding configuration, and the second configuration parameter defines the distributed data sharding processing logic.

[0160] In response to the data processing command, the system acquires the data to be processed and performs data fragmentation based on the first configuration parameters to obtain several data segments.

[0161] Obtain the business key hash value corresponding to the data to be processed, and calculate the fragment identifier of each data segment based on the business key hash value;

[0162] Each shard identifier is associated with its corresponding data segment, and the associated data segments are stored in a preset target table to be processed.

[0163] The target data fragment to be processed is determined based on the second configuration parameter, and the upper limit of the data processing volume of the target data fragment is obtained. The upper limit of the data processing volume is the number of data segments that the target data fragment can process at the same time.

[0164] Based on the upper limit of the data processing volume of the target data segment, the target data fragment corresponding to the target data segment is determined in the target table to be processed based on the first modulo operation, and the target data fragment is sent to the target data segment for processing.

[0165] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0166] This application can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics devices, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0167] It should be noted that the software tools or components not belonging to this company that appear in the various embodiments of this application are merely illustrative examples and do not represent actual use.

[0168] Obviously, the embodiments described above are only some embodiments of this application, not all embodiments. The accompanying drawings show preferred embodiments of this application, but do not limit the patent scope of this application. This application can be implemented in many different forms; rather, the purpose of providing these embodiments is to provide a more thorough and comprehensive understanding of the disclosure of this application. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing specific embodiments, or make equivalent substitutions for some of the technical features. Any equivalent structures made using the content of this application's specification and drawings, directly or indirectly applied to other related technical fields, are similarly within the scope of patent protection of this application.

Claims

1. A distributed asynchronous data processing method, characterized in that, include: Obtain the configuration parameters of the preset distributed task scheduling center, wherein the configuration parameters include a first configuration parameter and a second configuration parameter, the first configuration parameter defines the data sharding configuration, and the second configuration parameter defines the distributed data sharding processing logic; In response to a data processing instruction, the system acquires the data to be processed and performs data segmentation processing on the data to be processed based on the first configuration parameters to obtain several data fragments. Obtain the business key hash value corresponding to the data to be processed, and calculate the fragment identifier of each data segment based on the business key hash value; Each of the aforementioned fragment identifiers is associated with its corresponding data segment, and the associated data segments are stored in a preset target table to be processed. The target data segment to be processed is determined according to the second configuration parameters, and the upper limit of the data processing volume of the target data segment is obtained, wherein the upper limit of the data processing volume is the number of data segments that the target data segment can process simultaneously; Based on the upper limit of the data processing volume of the target data segment, the target data fragment corresponding to the target data segment is determined in the target table to be processed based on the first modulo operation, and the target data fragment is sent to the target data segment for processing.

2. The distributed asynchronous data processing method as described in claim 1, characterized in that, Before obtaining the configuration parameters of the preset distributed task scheduling center, the method further includes: Configure the initial configuration parameters according to the design specifications of the distributed task scheduling center; Parse the data processing instructions to obtain data processing requirements, wherein the data processing requirements include the upper limit of the amount of data queried in a single data segment, the upper limit of the amount of data processed in a data shard, and the maximum processing speed of the data shard; The initial configuration parameters are updated according to the data processing requirements to obtain the configuration parameters.

3. The distributed asynchronous data processing method as described in claim 2, characterized in that, The data processing requirements also include a single query start time and interval time. The steps of responding to data processing instructions, obtaining the data to be processed, and performing data fragmentation processing on the data to be processed based on the first configuration parameters to obtain several data segments specifically include: Parse the first configuration parameters to obtain the maximum amount of data that can be queried in a single query, the start time of a single query, and the interval time; Calculate the single query time window based on the query start time and the interval time; The data to be processed is divided into several data segments based on the upper limit of the data volume in a single query and the time window of a single query.

4. The distributed asynchronous data processing method as described in claim 1, characterized in that, The step of obtaining the business key hash value corresponding to the data to be processed, and calculating the fragment identifier of each data segment based on the business key hash value, specifically includes: Obtain the business key value from the data to be processed; The business key value is hashed using a hash algorithm to obtain the business key hash value; A second modulo operation is performed on the hash value of the business key and each of the data segments to obtain the result of the second modulo operation. The fragment identifier corresponding to each data segment is determined by the result of the second modulo operation.

5. The distributed asynchronous data processing method as described in claim 1, characterized in that, The step of associating each of the fragment identifiers with the corresponding data segments and storing the associated data segments in a preset target table to be processed specifically includes: Identify the data segments corresponding to each of the aforementioned fragment identifiers, and associate and match each of the aforementioned fragment identifiers with the corresponding data segments; A consistent hash ring structure is used to construct the hash ring of the target table to be processed; Each of the fragment identifiers is mapped to the hash ring, and the storage location of the associated data fragment is determined based on the node position of each fragment identifier in the hash ring. Based on the storage location of the associated data fragment, the associated data fragment is stored in a preset target table to be processed.

6. The distributed asynchronous data processing method as described in claim 5, characterized in that, Before the step of storing the associated data fragment to a preset target table based on the storage location of the associated data fragment, the method further includes: Monitor the capacity threshold status of each storage node on the hash ring; If the node's storage capacity exceeds a preset threshold, a load balancing mechanism is triggered to obtain a backup node with a lower load. The routing rule table is updated based on the information of the candidate nodes, and the storage location of the data fragments is reallocated using the least connection number algorithm to obtain the new storage location; Based on the new storage location, the associated data fragments are stored in the preset target table to be processed.

7. The distributed asynchronous data processing method as described in claim 1, characterized in that, The step of determining the target data segment corresponding to the target data segment in the target table based on the upper limit of the data processing volume of the target data segment and the first modulo operation, and sending the target data segment to the target data segment for processing, specifically includes: Calculate the number of target data fragments, and configure the first modulo operation rule based on the number of target data fragments and the upper limit of data processing volume; The data fragments in the target table to be processed are subjected to modulo operation according to the first modulo operation rule to obtain the first modulo operation result, wherein the first modulo operation result is a combination of the data fragments in the target table to be processed; Based on the result of the first modulo operation, the combination of data fragments in the target table to be processed is assigned to the matching target data fragment.

8. A distributed asynchronous data processing device, characterized in that, include: The configuration parameter module is used to obtain the configuration parameters of the preset distributed task scheduling center. The configuration parameters include a first configuration parameter and a second configuration parameter. The first configuration parameter defines the data sharding configuration, and the second configuration parameter defines the distributed data sharding processing logic. The data sharding module is used to respond to data processing instructions, obtain data to be processed, and perform data sharding processing on the data to be processed based on the first configuration parameters to obtain several data fragments. The fragment identification module is used to obtain the business key hash value corresponding to the data to be processed, and calculate the fragment identification of each data fragment based on the business key hash value. The data storage module is used to associate each of the fragment identifiers with the corresponding data segments, and store the associated data segments in a preset target table to be processed; The shard matching module is used to determine the target data shard to be processed according to the second configuration parameters, and to obtain the upper limit of the data processing volume of the target data shard, wherein the upper limit of the data processing volume is the number of data segments that the target data shard can process at the same time; The distributed processing module is used to determine the target data segment corresponding to the target data segment in the target target table based on the upper limit of the data processing volume of the target data segment and the first modulo operation, and send the target data segment to the target data segment for processing.

9. A computer device, characterized in that, The method includes a memory and a processor, wherein the memory stores computer-readable instructions, and the processor executes the computer-readable instructions to implement the steps of the distributed asynchronous data processing method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-readable instructions, which, when executed by a processor, implement the steps of the distributed asynchronous data processing method as described in any one of claims 1 to 7.

Images