Task fragmentation processing method, apparatus, device, medium and product

Abstract

Embodiments of the present application provide a task fragmentation processing method, device, equipment, medium and product, relating to the field of big data. The method periodically acquires a batch task containing multiple merchant archive change information, extracts a merchant identifier and an archive change type, forms a joint hash key based on the two, disperses the task to multiple shard libraries, and combines a preset task table to perform parallel scheduling processing on each shard data, and outputs the processing result corresponding to each merchant. The scheme can effectively disperse the processing pressure, reduce the task backlog, improve the archive change linkage response timeliness, overall processing efficiency and system stability in high-load scenarios.

Description

Technical Field

[0001] This application relates to the field of big data, and in particular to a task fragmentation processing method, apparatus, equipment, medium, and product. Background Technology

[0002] In the financial sector, banks and merchants are increasingly collaborating digitally. By connecting to the bank's system, merchants can access exclusive functions (such as transaction details inquiry, statement management, refund processing, store information maintenance, and promotional activity configuration) to improve operational efficiency.

[0003] With the widespread adoption of banking services, the number of merchants connected has grown exponentially, leading to a surge in merchant profiles and dedicated user data. When a merchant profile changes (such as an update to their communication method or an adjustment to the activation status of their dedicated features), the system must automatically update the login information or permission configurations of the corresponding dedicated user.

[0004] However, existing systems suffer from inefficiency and response delays when handling such high-frequency, massive data changes, which seriously affects merchants' need for immediate access to bank-specific functions. Summary of the Invention

[0005] This application provides a task fragmentation processing method, apparatus, device, medium, and product for scenarios involving changes to bank merchant profiles. It uniformly identifies, classifies, and distributes acquired batch tasks, and collaboratively processes change tasks corresponding to different bank merchants in accordance with task processing rules, thereby improving processing efficiency, responsiveness, and operational stability in scenarios with massive tasks.

[0006] Firstly, this application provides a task fragmentation processing method, including:

[0007] Batch task data is acquired according to a preset cycle. The batch task data includes file change information for multiple bank merchants.

[0008] Determine the corresponding merchant identifier and file change type in the batch task data;

[0009] Using the merchant identifier and file change type as the combined hash key, batch task data is distributed to multiple sharded databases through a preset hash algorithm;

[0010] The data in multiple sharded databases is processed in parallel according to the preset task table, and the task data processing results corresponding to each bank merchant are obtained. The preset task table stores the processing process of various file change tasks.

[0011] Secondly, this application provides a task fragmentation processing apparatus, comprising:

[0012] The acquisition module is used to acquire batch task data according to a preset period. The batch task data includes file change information for multiple bank merchants.

[0013] The determination module is used to determine the corresponding merchant identifier and file change type in the batch task data;

[0014] The processing module is used to distribute batch task data to multiple sharded databases using a preset hash algorithm, with the merchant identifier and file change type as the combined hash key.

[0015] The processing module is used to process sharded data from multiple sharded databases in parallel according to a preset task table, and obtain the task data processing results for each bank merchant. The preset task table stores the processing procedures for various file change tasks.

[0016] Thirdly, this application provides an electronic device, including: a processor, and a memory communicatively connected to the processor;

[0017] The memory stores computer-executed instructions;

[0018] The processor executes computer execution instructions stored in the memory to implement the task fragmentation processing method as described in the first aspect and various possible implementations of the first aspect above.

[0019] Fourthly, this application provides a computer-readable storage medium storing computer-executable instructions thereon, which, when executed by a processor, are used to implement the task fragmentation processing method as described in the first aspect and various possible implementations of the first aspect.

[0020] Fifthly, this application provides a program product, including a computer program, which, when executed by a processor, implements the task fragmentation processing method described above.

[0021] The task sharding processing method, apparatus, equipment, medium, and product provided in this application acquire batch task data containing multiple bank merchant file change information according to a preset cycle, determine the merchant identifier and file change type corresponding to the batch task data, and distribute the batch task data to multiple sharding databases using a preset hash algorithm with the merchant identifier and file change type as a joint hash key. Furthermore, the sharded data in multiple sharding databases are processed in parallel according to a preset task table. This enables the orderly distribution and parallel processing of batch change tasks, reduces task backlog and response latency caused by centralized processing, and thus improves the linkage processing efficiency, resource adaptation capability, and system operation stability in scenarios involving massive merchant file changes. Attached Figure Description

[0022] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0023] Figure 1 A flowchart illustrating the task fragmentation processing method provided in the embodiments of this application. Figure 1 ;

[0024] Figure 2 A flowchart illustrating the task fragmentation processing method provided in the embodiments of this application. Figure 2 ;

[0025] Figure 3 A flowchart illustrating the task fragmentation processing method provided in the embodiments of this application. Figure 3 ;

[0026] Figure 4 A flowchart illustrating the task fragmentation processing method provided in the embodiments of this application. Figure 4 ;

[0027] Figure 5 A schematic diagram of the task fragmentation processing device provided in this application;

[0028] Figure 6 This is a schematic diagram of the structure of an electronic device provided in this application.

[0029] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0030] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0031] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, storage, use, processing, transmission, provision, disclosure, and application of the relevant data all comply with relevant laws, regulations, and standards, necessary confidentiality measures have been taken, they do not violate public order and good morals, and corresponding operation portals are provided for users to choose to authorize or refuse.

[0032] Furthermore, the technical solution involved in this application, which involves big data analysis of user information (including but not limited to personal biometrics, identity data, consumption data, asset data, electronic terminal operation data, etc.) and the use of artificial intelligence technology for automated decision-making, and makes decisions that have a significant impact on personal rights based on the results of automated decision-making, provides users with corresponding operation entry points for users to choose to agree to or reject the results of automated decision-making; if the user chooses to reject, the process will proceed to the expert decision-making process.

[0033] It should be noted that the task fragmentation processing method, apparatus, equipment, medium and product provided in this application can be used in the field of big data, or in any field other than big data. The application field of the task fragmentation processing method, apparatus, equipment, medium and product in this application is not limited.

[0034] In financial services, banks and merchants are increasingly collaborating digitally. By connecting to the bank's system, merchants can access exclusive functions (such as transaction details inquiry, account management, refund processing, store information maintenance, and promotional activity configuration) to improve operational efficiency. When connecting to the bank's system, merchants need to create dedicated administrator and operator users. The administrator user is responsible for managing the merchant's overall functions, such as creating / deleting operators and setting permission scopes; the operator user executes specific operations according to the permissions assigned by the administrator.

[0035] With the widespread adoption of banking services, the number of merchants connected has grown exponentially, leading to a surge in merchant profiles and dedicated user data. For example, a bank's system may contain tens of millions of merchant profiles, each requiring a dedicated administrator user and multiple operator users, resulting in a total user data volume reaching hundreds of millions. When a merchant profile changes (such as updated communication methods or adjustments to the activation status of dedicated functions), the system must simultaneously modify the login information or permission configurations of the corresponding dedicated users.

[0036] In existing solutions, after receiving requests for merchant profile creation, information modification, or functional status adjustment, banks typically write the profile change information for multiple merchants into a unified task storage, and then the backend processing program executes these requests one by one according to a predetermined process. During the processing, the system generally first identifies the merchant and the changes, and then completes the corresponding exclusive user information updates, functional status linkage, or other related processing according to preset business rules.

[0037] However, as the number of merchants continues to expand and the types of file changes increase, centralized or low-concurrency processing methods can easily lead to a large number of tasks converging on the same processing chain, resulting in increased task queuing time. Especially when multiple merchants initiate requests for contact information modification, function adjustments, etc., at similar times, different types of tasks compete for processing resources, leading to a decrease in overall throughput.

[0038] The aforementioned defects will further lead to problems with untimely linkage processing, resulting in changes to merchant files but the relevant task results not yet being completed and effective. This will affect merchants' continuous use of bank-specific functions and increase the system's operational pressure and stability risks under high load scenarios.

[0039] In light of this, improving task processing efficiency and reducing response latency in scenarios involving massive merchant profile changes has become a pressing technical challenge. To address this, this application provides a task sharding method. After acquiring batch task data containing merchant profile change information from multiple banks at a preset cycle, the method determines the merchant identifier and profile change type. Using the merchant identifier and profile change type as a combined hash key, the batch task data is distributed to multiple sharding databases using a preset hash algorithm. The sharded data in these databases is then processed in parallel according to a preset task table, thereby obtaining the task data processing results for each bank merchant. By distributing batch tasks across multiple sharding databases using a combined hash key and processing them in parallel, the backlog problem caused by centralized processing can be alleviated, improving processing efficiency and system stability in scenarios involving massive merchant profile changes.

[0040] The task sharding method provided in this application is applicable to high-frequency, interconnected maintenance scenarios involving merchant profiles and dedicated user information within banking systems. Banks need to establish profiles for tens of millions of merchants and dynamically manage each merchant's dedicated administrator and operator users. When a merchant profile changes (such as a modification of communication methods or an adjustment to the activation status of dedicated functions), the system needs to synchronously update the corresponding user's login information, permission configuration, or send notifications. This scenario faces the pressure of processing hundreds of thousands of task requests daily, requiring task response times to be within seconds, while also supporting elastic scaling of business volume.

[0041] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.

[0042] Figure 1 A flowchart illustrating the task fragmentation processing method provided in the embodiments of this application. Figure 1 .like Figure 1 As shown, the task sharding processing method provided in this embodiment is applied to a data processing system, and the method includes:

[0043] S101. Obtain batch task data according to the preset cycle.

[0044] The batch task data includes file change information for multiple bank merchants.

[0045] Understandably, the preset period is used to limit the time interval for acquiring batch task data. This batch task data carries the file change information for multiple bank merchants and serves as input data for subsequent sharding and scheduling processing. In practice, when each preset period arrives, newly added or pending file change records within the current period can be acquired, and multiple task records collected within that period can be assembled into a batch of batch task data.

[0046] In one possible embodiment, a single record in the batch task data may include information such as merchant identifier, change time, change field content, change source, and processing status marker. In another possible embodiment, file change information may be periodically collected based on a preset data source to generate batch task data.

[0047] To ensure clear data boundaries for each cycle, the data processing system can record the start and end times of the current cycle when the scheduling task is initiated, and use a time window approach to filter archive change information that meets the criteria. For duplicate reports or duplicate records generated during cross-system synchronization, deduplication markers can be generated during the acquisition process, and deduplication processing can be completed during the batch task data construction phase, ensuring the determinism of data entering subsequent segmentation processing. For archive change information from different sources, it can be uniformly converted into structured task records after collection to reduce inconsistencies in subsequent processing caused by differences in field formats from different sources.

[0048] S102. Determine the corresponding merchant identifier and file change type in the batch task data.

[0049] Understandably, the merchant identifier is used to uniquely identify the corresponding bank merchant, and the file change type is used to identify the change category to which the corresponding merchant's file change information belongs. Together, they form the foundational fields for subsequent joint hash allocation. In specific implementation, the system parses the batch task data generated in S101 record by record, extracting the merchant identifier field from each task record as the merchant identifier, and identifying the corresponding file change type from the change item field, change source field, or business event information. If the original record already contains a clear type identifier, the system can read it directly; if the original record only records the change field content, the system can classify different field changes according to pre-established type mapping rules. To ensure that file change information from different sources enters the unified fragmentation process, the system can also perform normalization processing after extracting the merchant identifier and file change type.

[0050] In one possible implementation, if there are abnormal records in the batch task data where the merchant identifier is empty, the type field is missing, or the field content does not meet the mapping rules, the system can mark the abnormal records and allow the valid tasks to continue to participate in subsequent sharding processing.

[0051] Furthermore, after determining the merchant identifier and the file change type, the system can write both into the current task record, forming a standard task item for subsequent use. In this way, when entering the joint hash calculation, task records from different sources and in different formats can be calculated according to a consistent data structure, avoiding the impact of differences in field naming, encoding rules, or content expression on the allocation results.

[0052] S103. Using the merchant identifier and file change type as the combined hash key, batch task data is distributed to multiple sharded databases through a preset hash algorithm.

[0053] Understandably, the combined hash key is used to combine the merchant identifier and the file change type into a single hash calculation input. The default hash algorithm is used to map batch task data to multiple shard libraries based on the combined hash key. The shard libraries are used to hold the allocated task records and serve as independent storage units for parallel scheduling processing.

[0054] In its implementation, the system first generates a combined hash key for the standard task items obtained in S102. This can be achieved by concatenating the standardized merchant identifier and the standardized file change type in a predetermined order to form a key-value pair, or by forming a hash input according to a fixed format. Subsequently, the system calls a preset hash algorithm to perform a hash calculation on the combined hash key, obtains the hash result, and determines the target shard library based on the currently configured multiple shard libraries.

[0055] Specifically, writing refers to batch writing the corresponding task records into the corresponding shard database. Each write record can include at least the task number, merchant identifier, file change type, change content, original timestamp, shard identifier, and pending status, so that the scheduler can directly retrieve the corresponding shard data in each shard database.

[0056] Furthermore, the sharded library can use independent storage units or a logical sharding structure within the same storage system, as long as different shards can form relatively independent storage and access channels.

[0057] S104. Process the sharded data in multiple sharded databases in parallel according to the preset task table, and obtain the task data processing results corresponding to each bank merchant.

[0058] The preset task table stores the processing procedures for various file change tasks and can serve as the execution basis for parallel scheduling of fragmented data in multiple fragmented databases. Fragmented data refers to the set of task records written to each fragmented database after allocation by S103. The task data processing results are used to represent the result records or status information of the corresponding bank merchant after task execution. The preset task table may include type identifier, processing order, execution action identifier, associated business rules, status update rules, and result output fields.

[0059] Understandably, the system can process fragmented data from multiple fragmented databases in parallel. During processing, it queries a preset task table based on the file change type in the task record to locate the corresponding processing procedure definition. When processing a single fragmented data entry, the system can first locate the corresponding bank merchant's file record based on the merchant identifier, then call the processing procedure defined in the preset task table according to the file change type, complete the corresponding processing, and output the result. For successfully processed tasks, the system can generate task data processing results including task number, merchant identifier, file change type, processing completion time, processing status, and result summary; for tasks that are not successfully processed, the system can also record the corresponding result information for subsequent processing.

[0060] In another possible embodiment, after tasks in multiple shard libraries are processed in parallel, the processing results can be aggregated to form a set of task data processing results corresponding to each bank merchant. Since the processing procedures for various file change tasks are pre-stored in the preset task table, the system does not need to temporarily assemble logic during parallel execution. It only needs to locate the corresponding rule according to the file change type and execute it, so that tasks in different shard libraries are processed according to a unified and repeatable process.

[0061] Based on the above analysis, this step involves distributing sharded data from multiple sharded databases into different processing channels, using a pre-defined task table as the unified processing basis. This ensures that tasks, after being distributed, can continue to be executed concurrently under consistent rules. Since each sharded database carries a subset of tasks distributed via joint hashing, the system can process files and output results in parallel across different sharded databases. Therefore, in scenarios involving massive merchant changes, it no longer relies on a single processing link for sequential processing. Consequently, each bank merchant can obtain task data processing results matching their file change information, reducing task queuing time and processing latency.

[0062] One possible implementation is that, after obtaining the task data processing results, the data processing system can submit the notification sending task in the task data processing results to the message queue, and an independent asynchronous process will execute the notification sending operation so that bank merchants can be aware of the task data processing results.

[0063] Understandably, after the task data is processed, the data processing system can write the generated result information into a notification sending task and submit the notification sending task to a message queue. An independent asynchronous process periodically polls or subscribes to the message queue, reads the task content upon detecting a message to be processed, and completes the notification sending. If the notification is sent successfully, a sending receipt is written to the notification result record; if the sending fails, it is re-enqueued according to a preset retry strategy or compensation processing is triggered. By separating notification sending from the main processing chain, the task data processing result can be asynchronously notified to bank merchants upon completion, allowing them to know the completion status, failure reason, or subsequent processing requirements of the corresponding task.

[0064] This approach decouples the processing results of task data from the notification sending operation at the execution level, with message queues and independent asynchronous processes handling the notification delivery load. This ensures that result notifications can continue to execute even in high-concurrency scenarios. Because each notification task is associated with its processing result, bank merchants can promptly obtain the task status and thus confirm the final results of file changes, function adjustments, or other linked processes.

[0065] The task sharding processing method provided in this embodiment acquires batch task data containing multiple bank merchant profile change information according to a preset period, determines the corresponding merchant identifier and profile change type in the batch task data, uses the merchant identifier and profile change type as a joint hash key, and distributes the batch task data to multiple sharding databases through a preset hash algorithm. The sharded data in multiple sharding databases are then processed in parallel according to a preset task table, yielding the task data processing result for each bank merchant. In this method, the preset period completes task aggregation, the merchant identifier and profile change type complete task standardization identification, the joint hash key and preset hash algorithm achieve stable distribution across sharding databases, and the preset task table drives the concurrent execution of tasks in multiple sharding databases, thus forming a complete implementation chain from batch acquisition, field determination, sharding writing to parallel processing.

[0066] By splitting the massive number of merchant profile change tasks into multiple independent sharded databases and simultaneously entering multiple processing channels, the single-link backlog caused by centralized task storage is alleviated, and the bank's merchant profile change processing can be completed and output results in a shorter period of time.

[0067] Figure 2 A flowchart illustrating the task fragmentation processing method provided in the embodiments of this application. Figure 2 .like Figure 2 As shown, in Figure 1 Based on the implementation examples, the process of allocating batch task data to multiple sharded libraries specifically includes:

[0068] S201. Perform a hash operation on the merchant identifier and the file change type to obtain the corresponding hash value.

[0069] Understandably, after receiving batch task data, the data processing unit can first read the merchant identifier and file change type from each task data entry. The merchant identifier is a unique identification field for the merchant entity, possessing global uniqueness. The merchant file change type is used to distinguish different file operation behaviors such as addition, modification, cancellation, and information supplementation. The combination of the two can uniquely identify a merchant file change business record. During the calculation, the data processing system can concatenate the merchant identifier string with the merchant file change type code to generate a fixed-format composite business key value, avoiding the probability of hash collisions caused by duplicate single fields. Then, a standardized cryptographic hash algorithm is used to perform irreversible one-way mapping calculation on the concatenated composite key value, outputting a fixed-length, irregular, and uniformly distributed digital digest, which is the unique hash value of the business data.

[0070] Hash operations have a one-way irreversible characteristic, making it impossible to deduce the original merchant identifier and change type from the hash value; at the same time, the same composite key value will output a completely consistent hash value after cloud calculation, while different business records will likely generate differentiated hash results, which can provide a uniformly discrete numerical basis for subsequent sharding routing.

[0071] S202. Determine the candidate fragment identifier based on the preset number of bits in the hash value.

[0072] Understandably, the hash value obtained after hash calculation is usually an extremely long hexadecimal string. To simplify calculations, a pre-configured hash truncation rule can be used as a globally unified parameter. That is, the data processing system can truncate a specified number of bits from the beginning or end of the complete hash digest, convert the truncated bit into a decimal integer, and this integer becomes the candidate shard identifier. The preset bit length is a globally configured item for distributed sharding routing and can be dynamically adjusted based on shard library expansion plans and business data volume. The longer the truncation bit length, the wider the range of candidate shard identifier values, the higher the data dispersion, and the lower the probability of shard data skew. For example, the last eight bits of the hash value can be used as the candidate shard identifier.

[0073] Candidate shard identifiers are merely intermediate values ​​and are not directly used as the basis for final shard routing. Their core function is to compress the length of the original hash value, narrow the value range, simplify the subsequent modulo operation logic, and at the same time retain the uniform discrete distribution characteristics inherent in hashing to ensure the randomness of data routing.

[0074] S203. Divide the candidate shard identifier by the number of shards in the shard library and take the remainder to obtain the shard identifier.

[0075] Understandably, the number of sharded databases refers to the total number of sharded storage nodes for all available merchant data in the current cluster, which is a static parameter that can be configured for cluster operation and maintenance. The modulo operation is a standard modulo operation.

[0076] Modular operation can map candidate shard identifiers with a very large value range to a continuous integer range from 0 to N-1. Each integer in the range uniquely corresponds to an independent shard library, realizing a one-to-one binding of massive discrete values ​​with a limited number of shard nodes. This operation logic is deterministic. For the same candidate shard identifier and the same total number of shard libraries, the output shard identifier is exactly the same in each operation, ensuring that the same merchant change record will be routed to the same shard library in a fixed way.

[0077] Modular operations have extremely low computational complexity. Distributed gateways and data synchronization middleware can complete the calculation in milliseconds without causing performance loss to the data writing and querying process.

[0078] One possible implementation is that the number of sharded libraries is 2 to the power of N, where N is a positive integer.

[0079] Understandably, when deploying a data processing system, the value of N can be determined based on the merchant scale, concurrent task volume, and storage node capacity. Based on this, a corresponding number of sharded database instances are generated. Each sharded database has independent write and read addresses, each corresponding one-to-one with a shard identifier. Since the number of sharded databases satisfies a power of two, and with the preset truncation of hash values ​​and the mapping method using the remainder of the number of sharded databases, the calculation and routing rules for shard identifiers can remain consistent. This allows the system to add sharded databases exponentially during expansion or reconfiguration without altering existing allocation logic.

[0080] In this embodiment, the sharded database can be implemented using independent tablespaces, independent database instances, or logically isolated storage partitions in a relational database. In practical applications, the sharded database can also choose other storage formats, which are not limited in this application.

[0081] Under the above configuration, after task data enters the sharding library, it is processed independently by the processing unit corresponding to that sharding library. After generating the task data processing results, the notification sending tasks that need to notify bank merchants are submitted to the message queue. The message queue is used to temporarily store notification sending tasks. Independent asynchronous processes retrieve the corresponding tasks from the message queue and execute the notification sending operation, thus separating the notification sending process from the main processing process within the sharding library. When the number of sharding libraries is set to a power of 2, the sharding routing can stably correspond to a fixed number of processing channels. Notification sending tasks can also be evenly submitted to subsequent asynchronous processing links based on this stable channel, thereby forming a clear decoupling relationship between task allocation, result generation, and notification sending.

[0082] With the above-mentioned quantity configuration, the sharding library expansion method remains consistent with the hash mapping rules. The sharding location process can be directly completed based on bitwise operations and modulo operations, reducing the additional calculations introduced by the non-exponential number of shards. Since the notification sending task is transferred to the message queue and processed by an independent asynchronous process, the main processing chain does not need to wait for the notification to be sent to complete before continuing to execute subsequent tasks, thus maintaining the continuity of the overall processing chain. Therefore, when facing scenarios involving batch changes to merchant profiles, the system can simultaneously maintain the stability of sharding processing and the independent execution capability of notification sending.

[0083] S204. Based on the shard identifier, distribute batch task data to multiple shard libraries.

[0084] Understandably, there's a predefined mapping relationship between shard identifiers and shard databases. After determining the shard identifier, the system can match it with the target shard database. Once the matching is complete, the system can establish a database connection with the target shard database and persistently write all business data, including complete merchant identifiers, file change content, operation time, and change type, to the corresponding shard database. In batch synchronization scenarios, the system can group data based on the respective shard identifiers of multiple data entries, and batch distribute data with the same shard identifier to the corresponding shard, reducing the number of cross-database connections and improving batch processing throughput.

[0085] After sharding, the same hash sharding logic is reused for querying, updating, and deleting merchant files. The sharding identifier is reproduced by the merchant identifier and change type, and the target shard library is directly located, avoiding full shard library scanning, greatly reducing database query overhead, and realizing targeted data read and write under distributed storage.

[0086] For example, the following specific example will be used to illustrate the implementation process of allocating batch task data to multiple sharded libraries.

[0087] If a bank has 30 million merchant files and 100 million dedicated users, and the files are updated at a rate of 1% per day, there will be 300,000 file changes per day, involving synchronous modifications for 1 million users. Currently, each shard database task table has a daily storage capacity of 10,000 records. Since there are 300,000 changes per day, and the number of shard databases is based on the principle of 2 to the power of N, 32 shard databases are created, numbered from SET0 to SET31.

[0088] After concatenating and hashing the merchant identifier and change type, a hexadecimal hash string is obtained. The last 8 hexadecimal characters of the hash string are extracted and converted into a decimal integer as the candidate shard identifier. The remainder of the candidate shard identifier divided by 32 is the target shard number. Assuming the last eight characters of the hash value are hexadecimal characters, the converted candidate shard identifier is 234578693. Dividing the candidate shard identifier by the total number of shards (32) and taking the remainder is 234578693 mod 32 = 5. Therefore, the shard number corresponding to this merchant profile change data is 5, and the system will write this data to the shard database with the number SET5.

[0089] Assuming the last eight bits of the hash value are hexadecimal characters, the candidate shard identifier after conversion is 192061377. Dividing the candidate shard identifier by the total number of shards, 32, and taking the remainder, 192061377 mod 32 = 1. Therefore, the shard number corresponding to this merchant profile change data is 1, and the system will write this data to the shard database with the number SET1.

[0090] The task sharding method provided in this embodiment forms a stable allocation relationship through a joint mapping between merchant identifiers and file change types. This allows the same merchant to fall into different sharding databases under different file change types, while data with the same business attributes can maintain consistent allocation rules. Since the hash value is extracted with a preset number of bits and then mapped to the number of sharding databases by taking the remainder, the data distribution is more likely to form a relatively balanced load among the sharding databases, thus providing a foundation for subsequent independent processing and parallel scheduling.

[0091] After adopting this implementation method, batch task data can be stably distributed to multiple sharded databases according to the joint hash key, reducing the situation of concentrated accumulation in a single database and reducing the probability of clustering of similar or related tasks in the same processing link. This makes the storage and subsequent processing of sharded tasks more balanced, thereby improving the task allocation efficiency and system operation stability in high-concurrency file change scenarios.

[0092] Figure 3 A flowchart illustrating the task fragmentation processing method provided in the embodiments of this application. Figure 3 .like Figure 3 As shown, in Figure 1 Based on the implementation examples, the process of batch processing fragmented data specifically includes:

[0093] S301. Configure at least one independent batch processing process for each shard library.

[0094] Understandably, configuring at least one independent batch processing process for each shard library is essentially based on the idea of ​​process-level resource isolation to achieve one-to-one binding of sharded computation and storage, physically isolating the data batch processing computation logic of each shard, and eliminating the competition for computing resources between shards.

[0095] Each independent batch processing process has its own dedicated memory space, processor scheduling time slice, input / output channels, and task execution context. Its lifecycle is strongly bound to the corresponding shard library, and it only receives batch data tasks from the current shard library, without participating in data processing in other shards. When a sudden increase in data volume in a shard library causes excessive load on the batch process, memory overflow, or program crashes, the fault is limited to the independent process corresponding to that shard and will not affect the batch processing chain of other shard libraries. This achieves fault domain segmentation and significantly reduces the overall downtime risk of distributed batch tasks.

[0096] S302. The batch processing process processes the fragmented data in each fragmented library in batches according to the preset processing rules.

[0097] The preset processing rules include task type classification and resource allocation strategies.

[0098] Understandably, once the batch processing process starts, it establishes a dedicated data access session with the corresponding sharded database, and traverses all the data to be processed within the shard based on the shard identifier. The process can divide the raw sharded data into several data sub-batches according to a fixed number of data entries, data capacity threshold, or time window, and complete the reading, verification, calculation, database writing, or synchronization operations batch by batch, ensuring the smooth progress of batch operations on massive amounts of sharded data.

[0099] Each segmented database has its own independent batch processing process, specifically responsible for handling batch change tasks of bank merchant profiles stored in its respective segmented database. The batch processing process can automatically divide the entire segment of merchant profile change data into multiple smaller batches, and complete the data processing operations in a cyclical manner. All processing actions are strictly executed according to the unified rules pre-configured by the system.

[0100] The system's predefined processing rules comprise two main parts. The first is task type classification rules. Changes to bank merchant profiles are categorized into specific business types, such as adding a merchant profile, modifying a merchant profile, canceling a merchant profile, and supplementing profile information. When processing each batch of segmented profile data, the batch processing process can first identify the task type corresponding to each piece of data, and then match the standard processing flow specific to that type. Different profile change types execute corresponding verification, data entry, and synchronization logic to ensure consistent processing standards for similar merchant profile change business.

[0101] When a merchant applies to access the bank's system, a new merchant profile is created. This profile records the merchant's contact information, such as mobile phone number and email address, as well as whether they have activated exclusive features. Once the merchant profile is complete, the system sends a notification to the merchant to confirm that the profile is ready and they can use the corresponding exclusive merchant features.

[0102] Modifying a merchant's profile, which involves changing their contact information and whether they have enabled exclusive features, will also allow the system to send a notification to the merchant after the modification, informing them of the result.

[0103] The second category of supporting rules is the resource allocation strategy. For different types of batch tasks involving merchant profiles, the system presets differentiated resource limit standards. For example, adding or massively modifying merchant profiles are high-load tasks, so the number of data entries processed in each batch and the concurrency of database queries will be limited. For lightweight tasks such as a small number of profile updates, the processing concurrency will be appropriately relaxed to speed up execution. The batch process, combined with the identified profile task type, automatically matches the corresponding resource quota to avoid excessive server and database resource consumption during batch processing of merchant profiles, which could affect the operation of other online banking services.

[0104] The task sharding processing method provided in this embodiment configures independent batch processing processes for each shard library and performs batch processing in combination with task type classification and resource allocation strategies. This enables different types of tasks to be executed in parallel in an orderly manner within the corresponding shard library and matches processing resources according to task load, thereby achieving stable concurrent processing of sharded data and shortening the result generation time.

[0105] Figure 4 A flowchart illustrating the task fragmentation processing method provided in the embodiments of this application. Figure 4 .like Figure 4 As shown, in Figure 1 Based on the implementation examples, further expansion can be achieved by adjusting the load on the sharded library. The specific process includes:

[0106] S401. For any sharded database, determine the actual processing time of the subtask corresponding to the sharded data in the sharded database.

[0107] Understandably, the data processing system can embed full-link time-consuming data collection points for merchant profile change subtasks under all sharded databases. The batch processing process records a timestamp when starting a single subtask, and collects the end timestamp again after the entire subtask process is completed. The time difference is used to calculate the complete actual processing time of this subtask. The time-consuming statistics cover the entire lifecycle of the subtask. Additional time consumed by abnormal retries, data lock waits, and database input / output interface blocking in merchant profile batch change scenarios will also be fully included in the actual processing time, ensuring that the statistical results accurately reflect the real execution load of the sharded subtasks in the current cluster environment.

[0108] S402. Retrieve the standard processing time of the corresponding subtask in the preset task table.

[0109] Understandably, the pre-defined task table can be indexed by sub-task business type, pre-entering the baseline processing time (i.e., standard processing time) for various merchant profile change sub-tasks under standard hardware and no resource contention environment. Different types of sub-tasks, such as adding merchant profiles, modifying merchant profiles, canceling merchant profiles, and supplementing profiles, are configured with independent standard times. These values ​​have been verified through business load testing and represent the normal execution time of the sub-tasks when there is no resource congestion.

[0110] After the actual time consumption of each subtask is statistically analyzed, the data processing system can use the business type of the current subtask as the query condition to read the matching standard processing time from the preset task table in real time. The configuration table supports dynamic updates, allowing operations and maintenance personnel to adjust the baseline duration of various subtasks based on business iterations and system performance upgrades. Updates take effect immediately without restarting the batch processing process, ensuring that the baseline reference value matches the current business processing capacity.

[0111] S403. Determine the backlog of tasks based on the actual processing time and the standard processing time.

[0112] The backlog of tasks refers to the number of subtasks whose actual processing time exceeds the standard processing time.

[0113] Understandably, the data processing system can compare the actual processing time with the standard processing time for each completed or currently executing subtask in the sharded database. If the actual processing time of a subtask exceeds the standard processing time for the same type of task in the preset task table, it is determined that the subtask is experiencing execution delays due to factors such as insufficient resources, a sudden increase in data volume, or database pressure, and is classified as a backlog of subtasks.

[0114] The task backlog is the total number of subtasks in the current shard library that are determined to be delayed in execution. This metric is used to intuitively quantify the congestion level of the shard batch processing link. Unlike simply unexecuted tasks waiting in the queue, the backlog focuses on subtasks that have timed out, accurately reflecting the current situation where the existing computing power is insufficient to meet the current shard merchant profile data processing needs. It can serve as a core basis for determining resource expansion.

[0115] S404. When the backlog of tasks exceeds the preset threshold, perform an expansion operation.

[0116] The expansion operations include: increasing the number of sharded libraries and redistributing batch task data based on the federated hash key, and / or increasing the number of batch processing processes.

[0117] Understandably, the data processing system is pre-configured with alarm thresholds for shard backlog, which can be calculated based on peak merchant profile business volume and cluster capacity. The system continuously compares the current shard backlog with the preset threshold in real time. Once the shard backlog exceeds the threshold, it indicates a persistent bottleneck in current shard processing resources. The system can then automatically trigger a batch task expansion process to proactively alleviate task execution delays.

[0118] The first expansion solution is horizontal splitting of the sharded database: New sharded storage units are added, using a combined hash key composed of merchant identifier and file change type as the sharding routing basis. Existing pending data and newly added batch change data for merchant files are re-hash-routed. By splitting the original massive data in a single shard, the total data volume and subtask base of a single sharded database are reduced, distributing the processing pressure at the data source level and fundamentally reducing the generation of timeout subtasks in a single shard.

[0119] The second expansion scheme is a vertical scaling computing example: adding independent batch processing processes to the corresponding sharded database, leveraging process-level resource isolation mechanisms to enhance the parallel processing capabilities of the shards. The new processes can synchronously execute subtasks split from the sharded database, increasing the total number of merchant profile change tasks that can be processed concurrently per unit time, shortening the overall processing cycle for a single batch of data, and reducing the probability of subtask timeouts.

[0120] Both expansion methods can be executed individually or in combination. If the sharding is only mildly congested, simply adding batch processing processes can quickly increase concurrent computing power. If the original data volume of the sharding is too large and there are frequent high backlogs, the sharding library can be added simultaneously and the sharding can be rehashed and the processing processes expanded. Storage and computing are expanded simultaneously, completely solving the problem of long-term backlog and timeout of merchant file batch change subtasks.

[0121] For example, assuming the processing times for the four steps corresponding to a certain data shard are 0.05 seconds, 0.05 seconds, 0.1 seconds, and 0.1 seconds, the sum of these times gives the fixed standard processing time for a single subtask: 0.05 + 0.05 + 0.1 + 0.1 = 0.3 seconds. Considering the current limits on processor, memory, and database connection resources, performance testing reveals the following hardware constraints: the maximum concurrent capacity of a single independent batch processing process is 5 subtasks processed simultaneously.

[0122] The system imposes a hard latency constraint on merchant profile modification services, requiring all subtask end-to-end response times to be controlled within 1 second. Based on a standard single-task time of 0.3 seconds and a maximum latency of 1 second, the maximum number of batches that can be executed in a single process within a given timeframe can be calculated: 1 ÷ 0.3 ≈ 3 execution batches. Combining this with process concurrency capabilities, the theoretical maximum throughput per second for a single process is calculated as: 5 concurrent tasks per batch × 3 batches per second = 15 tasks / second. This throughput represents the theoretical upper limit of processing power under current single-process conditions without resource bottlenecks, serving as a benchmark for scaling up sharded resources.

[0123] Assuming that the daily average number of merchant change tasks is 250 and is divided into various shards (SETs), there is a surge in traffic during peak business hours, with a single shard SET receiving up to 25 pending sub-tasks at any given time.

[0124] Compared to the processing capacity of 15 tasks per second for a single process, when only one batch process is deployed, the shard SET can only process 15 tasks per second. The remaining 10 sub-tasks cannot be processed within the 1-second latency window. These timed-out tasks will be counted as task backlog. When the backlog exceeds the system's preset threshold, it is determined that the current shard computing power resources are insufficient, triggering the automatic expansion logic.

[0125] To meet the peak processing requirement of 25 messages / second and eliminate task backlog, the current shard SET batch computing resources can be expanded. The current maximum throughput of a single process is 15 messages / second. After deploying two independent batch processes, the total shard processing capacity can reach 30 messages / second, which is greater than the peak instantaneous task volume of 25 messages. This ensures that all subtasks are completed within the 1-second latency requirement, and no more timeout backlog subtasks are generated.

[0126] This embodiment can trigger expansion control in a timely manner when the load on the sharded library or processing process increases. After redistributing batch task data based on the combined hash key, the task distribution can be rebalanced; after adding batch processing processes, the concurrent processing capacity of the same sharded library is improved, thereby ensuring that the processing capacity of file change tasks matches the load changes.

[0127] In some embodiments, when batch processing data, the data processing system can also prioritize more urgent tasks based on task priority. Specifically, the batch task data includes a task priority field. The task priority field is used to identify the urgency of the task data. The data processing system can adjust the task priority for parallel processing of data chunks and allocate independent processes to handle high-priority tasks based on the task priority field and the current task backlog status.

[0128] Understandably, the task priority field can be written when generating batch task data. The field value can be an integer level, an enumerated identifier, or a flag, with higher values ​​corresponding to more urgent tasks. The current task backlog status can be determined by a combination of the shard library's pending queue length, average waiting time, or the number of assigned but incomplete tasks, reflecting the congestion level of the current shard batch processing chain. The combination of these two factors can serve as the basis for determining the scheduling strategy.

[0129] During parallel processing, the data processing system can read batch task data from each shard in real time, perform hierarchical sorting of tasks based on the priority field carried by each task, and dynamically adjust the parallel processing execution order of data within each shard. In multi-task parallel consumption scenarios, high-priority merchant profile change tasks are executed first, while low-priority ordinary profile tasks are scheduled later, avoiding a large number of low-priority tasks occupying computing resources and causing processing delays and backlogs for urgent business tasks.

[0130] For high-priority sharded tasks, the data processing system supports allocating dedicated, independent batch processing processes to isolate and handle these tasks. These independent processes do not participate in the computation of low-priority tasks, possess dedicated processing and database connections, and are unaffected by concurrent contention or backlogs from ordinary tasks. Even if a large backlog of ordinary tasks occurs across the corresponding shard, high-priority file change data can still be processed stably and quickly in batches through this dedicated process, ensuring that urgent business operations strictly meet time constraints.

[0131] This scheduling mechanism complements the aforementioned expansion mechanism: if only low-priority tasks are backlogged in a shard, simply adjusting the task execution order can alleviate congestion without triggering expansion. If high-priority tasks also experience timeouts and backlogs, the system, in addition to increasing task scheduling priority and allocating independent processes, will simultaneously determine the backlog threshold and perform expansion operations such as adding batch processes and splitting the shard database as needed, balancing the timeliness of urgent tasks with the overall load balancing of the shard batch processing cluster. With this approach, high-urgency tasks can be completed first in backlog scenarios, the processing order of sharded data is more stable, and the blocking level of the task queue is reduced.

[0132] Figure 5 A schematic diagram of the task fragmentation processing device provided in this application. Figure 5 As shown, this application provides a task fragmentation processing apparatus 50, which includes:

[0133] The acquisition module 501 is used to acquire batch task data according to a preset period. The batch task data includes file change information corresponding to multiple bank merchants.

[0134] The determination module 502 is used to determine the corresponding merchant identifier and file change type in the batch task data;

[0135] Processing module 503 is used to distribute batch task data to multiple sharded databases using a preset hash algorithm with merchant identifier and file change type as the combined hash key.

[0136] The processing module 503 is used to process the sharded data in multiple sharded libraries in parallel according to a preset task table, and obtain the task data processing results corresponding to each bank merchant. The preset task table stores the processing process of various file change tasks.

[0137] Optionally, the processing module 503 is specifically used to perform hash operations on the merchant identifier and the file change type to obtain the corresponding hash value;

[0138] The determination module 502 is specifically used to determine the candidate fragment identifier based on a preset number of bits in the hash value;

[0139] Processing module 503 is specifically used to divide the candidate shard identifier by the number of shards in the shard library and take the remainder to obtain the shard identifier;

[0140] The processing module 503 is specifically used to distribute batch task data to multiple shard libraries based on the shard identifier.

[0141] Optionally, the number of sharded libraries is 2 to the power of N, where N is a positive integer.

[0142] Optionally, the processing module 503 is specifically configured to configure at least one independent batch processing process for each shard library;

[0143] The processing module 503 is specifically used to process the sharded data in each sharded library in batches according to preset processing rules through a batch processing process. The preset processing rules include task type classification and resource allocation strategy.

[0144] Optionally, the preset task table also stores the standard processing time for various file change tasks. The determination module 502 is also used to determine the actual processing time of the subtask corresponding to the fragment data in any fragment library.

[0145] The determination module 502 is also used to retrieve the standard processing time of the corresponding subtask in the preset task table;

[0146] The determination module 502 is also used to determine the task backlog based on the actual processing time and the standard processing time. The task backlog is the number of subtasks whose actual processing time is greater than the standard processing time.

[0147] The processing module 503 is also used to perform expansion operations when the backlog of tasks exceeds a preset threshold. The expansion operations include: increasing the number of shard libraries and redistributing batch task data based on the combined hash key, and / or increasing the number of batch processing processes.

[0148] Optionally, the batch task data includes a task priority field. The processing module 503 is also used to adjust the task priority of parallel processing of the sharded data and allocate independent processes to handle high-priority tasks based on the task priority field and the current task backlog status. The task priority field is used to identify the urgency of the task data.

[0149] Optionally, the processing module 503 is also used to submit the notification sending task in the task data processing result to the message queue, and the notification sending operation is executed by an independent asynchronous process so that the bank merchant can be clear about the processing result of the task data.

[0150] The task fragmentation processing apparatus provided in this application embodiment has a similar implementation principle and technical effect to the implementation of each part of the aforementioned task fragmentation processing method, and will not be described again here.

[0151] Figure 6 This is a schematic diagram of the structure of an electronic device provided in this application. Figure 6 As shown, this application provides an electronic device 60, which includes: a receiver 601, a transmitter 602, a processor 603, and a memory 604.

[0152] Receiver 601 is used to receive instructions and data;

[0153] Transmitter 602 is used to send commands and data;

[0154] Memory 604 is used to store instructions executed by the computer;

[0155] The processor 603 is used to execute computer execution instructions stored in the memory 604 to implement the various steps of the task fragmentation processing method in the above embodiments. For details, please refer to the relevant descriptions in the foregoing embodiments of the task fragmentation processing method.

[0156] Optionally, the memory 604 can be either standalone or integrated with the processor 603.

[0157] When the memory 604 is set up independently, the electronic device also includes a bus for connecting the memory 604 and the processor 603.

[0158] The implementation principle and technical effects of the electronic device provided in this embodiment can be found in the foregoing embodiments, and will not be repeated here.

[0159] This application also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the method of any of the foregoing embodiments.

[0160] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the method of any of the foregoing embodiments.

[0161] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily essential to this application.

[0162] It should be further noted that although the steps in the flowchart are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowchart may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0163] It should be understood that the above-described device embodiments are merely illustrative, and the device of this application can also be implemented in other ways. For example, the division of units / modules in the above embodiments is only a logical functional division, and there may be other division methods in actual implementation. For example, multiple units, modules, or components may be combined, or integrated into another system, or some features may be ignored or not executed.

[0164] Furthermore, unless otherwise specified, the functional units / modules in the various embodiments of this application can be integrated into one unit / module, or each unit / module can exist physically separately, or two or more units / modules can be integrated together. The integrated units / modules described above can be implemented in hardware or as software program modules.

[0165] When integrated units / modules are implemented in hardware, the hardware can be digital circuits, analog circuits, etc. The physical implementation of the hardware structure includes, but is not limited to, transistors, memristors, etc. Unless otherwise specified, the processor can be any suitable hardware processor, such as a CPU, GPU, FPGA, DSP, and ASIC, etc. Unless otherwise specified, the storage unit can be any suitable magnetic or magneto-optical storage medium, such as Resistive Random Access Memory (RRAM), Dynamic Random Access Memory (DRAM), Static Random Access Memory (SRAM), Enhanced Dynamic Random Access Memory (EDRAM), High-Bandwidth Memory (HBM), Hybrid Memory Cube (HMC), etc.

[0166] If the integrated unit / module is implemented as a software program module and sold or used as an independent product, it can be stored in a computer-readable storage device (CMD). Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a memory and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned memory includes various media capable of storing program code, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard drive, magnetic disk, or optical disk.

[0167] In the above embodiments, the descriptions of each embodiment have their own emphasis. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments. The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as the combination of these technical features does not contradict each other, it should be considered within the scope of this specification.

[0168] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this application are indicated by the following claims.

[0169] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. A task fragmentation processing method, characterized in that, include: Batch task data is acquired according to a preset cycle, and the batch task data includes file change information corresponding to multiple bank merchants; Determine the corresponding merchant identifier and file change type in the batch task data; Using the merchant identifier and the file change type as a combined hash key, the batch task data is distributed to multiple sharded databases through a preset hash algorithm; The data in the multiple sharded databases are processed in parallel according to the preset task table, and the task data processing results corresponding to each bank merchant are obtained. The preset task table stores the processing process of various file change tasks.

2. The method according to claim 1, characterized in that, The step of using the merchant identifier and the file change type as a combined hash key, and distributing the batch task data to multiple sharded databases through a preset hash algorithm, includes: Perform a hash operation on the merchant identifier and the file change type to obtain the corresponding hash value; Based on the preset number of bits in the hash value, the candidate fragment identifier is determined; The candidate fragment identifier is divided by the number of fragments in the fragment library and the remainder is taken to obtain the fragment identifier; Based on the shard identifier, the batch task data is distributed to multiple shard libraries.

3. The method according to claim 2, characterized in that, The number of the fragmented libraries is 2 to the power of N, where N is a positive integer.

4. The method according to claim 1, characterized in that, The parallel processing of sharded data from multiple sharded databases according to a preset task table includes: Configure at least one independent batch processing process for each of the aforementioned shard libraries; The batch processing process processes the sharded data in each sharded library in batches according to preset processing rules, which include task type classification and resource allocation strategies.

5. The method according to claim 4, characterized in that, The preset task table also stores standard processing times for various file change tasks. After processing the fragmented data in multiple fragmented databases in parallel according to the preset task table, the method further includes: For any given sharded database, determine the actual processing time of the subtask corresponding to the sharded data in the sharded database; Retrieve the standard processing time for the corresponding subtask from the preset task table; Based on the actual processing time and the standard processing time, the task backlog is determined, where the task backlog is the number of subtasks whose actual processing time is greater than the standard processing time. When the backlog of tasks exceeds a preset threshold, an expansion operation is performed. The expansion operation includes: increasing the number of shard libraries and redistributing the batch task data based on the combined hash key, and / or increasing the number of batch processing processes.

6. The method according to claim 1, characterized in that, The batch task data includes a task priority field, and the method further includes: Based on the task priority field and the current task backlog status, the task priority for parallel processing of the sharded data is adjusted and an independent process is assigned to handle high-priority tasks. The task priority field is used to identify the urgency of the task data.

7. The method according to claim 1, characterized in that, The method further includes: The notification sending task from the task data processing result is submitted to the message queue, and the notification sending operation is executed by an independent asynchronous process so that the bank merchant is aware of the processing result of the task data.

8. A task fragmentation processing device, characterized in that, include: The acquisition module is used to acquire batch task data according to a preset period, the batch task data including file change information corresponding to multiple bank merchants; The determination module is used to determine the corresponding merchant identifier and file change type in the batch task data; The processing module is used to distribute the batch task data to multiple sharded libraries using the merchant identifier and the file change type as a combined hash key and a preset hash algorithm. The processing module is used to process the sharded data in the multiple sharded libraries in parallel according to a preset task table, and to obtain the task data processing results corresponding to each bank merchant. The preset task table stores the processing process of multiple file change tasks.

9. An electronic device, characterized in that, include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory to implement the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of claims 1 to 7.

11. A computer program product, characterized in that, Includes a computer program that, when executed by a processor, implements the method of any one of claims 1 to 7.

Images