Data processing methods based on multiple data sources, computer equipment, and storage media.

Abstract

This application provides a data processing method, computer device, and storage medium based on multiple data sources. The method includes: adapting multiple data sources according to a configuration file of a target business to determine the target data source corresponding to the target business; obtaining multiple business records of the target business from the target data source, where each business record is a record generated after processing multiple business requests of the target business; grouping the multiple business records according to the requester information in the multiple business records to obtain multiple business groups; and using multiple consumer threads corresponding to the multiple business groups to call the pre-configured business consumption logic of the target data source to sequentially consume each business record in the multiple business groups. Adapting data sources based on the business configuration file reduces the complexity of database source adaptation, and the parallel consumption of business records in different business groups by multiple threads balances efficiency and sequentiality.

Description

Technical Field

[0001] This application relates to the field of computer technology, and more specifically, to a data processing method, computer device, and storage medium based on multiple data sources. Background Technology

[0002] In actual software development and system architecture, different business lines typically have their own characteristics, data processing requirements, and business processes. To better manage and scale these business lines, design principles such as modularity and microservice architecture are often adopted to decouple and independently develop different business lines.

[0003] In related technologies, when implementing business consumption logic, it is necessary to develop separate adaptation code for different data sources. Data source adaptation is relatively complex. Furthermore, the current single-threaded approach can guarantee the order but is inefficient, while multi-threaded consumption can improve efficiency but is difficult to guarantee the order. Therefore, business consumption logic cannot balance efficiency and order. Summary of the Invention

[0004] In view of this, embodiments of this application provide a data processing method, computer device, and storage medium based on multiple data sources to solve the problems of complex data source adaptation and the inability of business consumption logic to balance efficiency and order.

[0005] In a first aspect, embodiments of this application provide a data processing method based on multiple data sources, including:

[0006] Based on the configuration file of the target business, adapt multiple data sources to determine the target data source corresponding to the target business;

[0007] Multiple business records of the target business are obtained from the target data source. The multiple business records are records generated after processing multiple business requests of the target business. Each business record includes: information of the requester of the corresponding business request.

[0008] Based on the requester information in the multiple business records, the multiple business records are grouped to obtain multiple business groups;

[0009] Multiple consumption threads corresponding to the multiple business groups are used to call the pre-configured business consumption logic of the target data source to sequentially consume each business record in the multiple business groups.

[0010] Secondly, embodiments of this application also provide an electronic device, including: a processor, a memory, and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor communicates with the memory via the bus, and the processor executes the machine-readable instructions to perform the method described in any of the first aspects.

[0011] Thirdly, embodiments of this application also provide a computer-readable storage medium storing a computer program, which, when executed by a processor, performs the method described in any of the first aspects.

[0012] This application provides a data processing method, computer device, and storage medium based on multiple data sources. The method includes: adapting multiple data sources according to a configuration file of a target business to determine the target data source corresponding to the target business; obtaining multiple business records of the target business from the target data source, where each business record is a record generated after processing multiple business requests of the target business; grouping the multiple business records according to the requester information in the multiple business records to obtain multiple business groups; and using multiple consumer threads corresponding to the multiple business groups to call the pre-configured business consumption logic of the target data source to sequentially consume each business record in the multiple business groups. Adapting data sources based on the business configuration file reduces the complexity of database source adaptation, and the parallel consumption of business records in different business groups by multiple threads balances efficiency and sequentiality. Attached Figure Description

[0013] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0014] Figure 1 A flowchart illustrating the data processing method based on multiple data sources provided in this application embodiment. Figure 1 ;

[0015] Figure 2 A flowchart illustrating the data processing method based on multiple data sources provided in this application embodiment. Figure 2 ;

[0016] Figure 3 A flowchart illustrating the data processing method based on multiple data sources provided in this application embodiment. Figure 3 ;

[0017] Figure 4A flowchart illustrating the data processing method based on multiple data sources provided in this application embodiment. Figure 4 ;

[0018] Figure 5 A flowchart illustrating the data processing method based on multiple data sources provided in this application embodiment. Figure 5 ;

[0019] Figure 6 This is a schematic diagram of the architecture of a data processing system based on multiple data sources provided in an embodiment of this application;

[0020] Figure 7 A schematic diagram of service grouping provided in the embodiments of this application;

[0021] Figure 8 This is a schematic diagram of the structure of a data processing device based on multiple data sources provided in an embodiment of this application;

[0022] Figure 9 A schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0023] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. The components of the embodiments of this application described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely represents selected embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0024] To address the current issues of complex data source adaptation and the inability of business consumption logic to balance efficiency and order, this application provides a data processing method based on multiple data sources. It adapts data sources based on business configuration files, reducing the complexity of database source adaptation, and consumes business records in multiple business groups in parallel to balance efficiency and order.

[0025] Before introducing the technical solution of this application, the technical terms involved in this application will be explained first:

[0026] Local retries: If a business transaction fails, it will immediately retries a limited number of times within the current process, without relying on external middleware.

[0027] Delayed retry: If a local retry fails, the message is delivered to the middleware, and asynchronous retry is performed at preset time intervals.

[0028] Sequential message multi-threaded batch consumption: For messages that need to be ordered, different batches of messages are processed in parallel by multiple threads, while ensuring the order of messages within the same batch.

[0029] Batch sharding consumption: Messages are grouped based on the sharding key, and multiple threads consume messages from different groups in parallel, improving processing efficiency.

[0030] Multi-data source adaptation: Supports access and message consumption from multiple data sources such as Canal, Kafka, Elastic Job, and ZMQ.

[0031] Software Development Kit (SDK) integration: Standardized interfaces are provided in the form of SDKs, so users do not need to worry about the underlying retry and consumption logic, but only need to implement the business data processing logic, i.e., business consumption logic.

[0032] Message queue: A middleware based on asynchronous communication mode, used to store, transmit and distribute messages in distributed systems.

[0033] The data processing method based on multiple data sources provided in this application is described below.

[0034] Figure 1 A flowchart illustrating the data processing method based on multiple data sources provided in this application embodiment. Figure 1 In this embodiment, the executing entity can be a computer device, such as a server.

[0035] like Figure 1 As shown, the method may include:

[0036] S101. Based on the configuration file of the target business, adapt multiple data sources to determine the target data source corresponding to the target business.

[0037] The target business could be, for example, a product purchase, a product search, or an order payment. Multiple data sources are already connected to a multi-data source-based data processing system, and each data source has its own pre-configured business consumption logic.

[0038] The configuration file of the target business includes information about the data source corresponding to the target business. Therefore, the target data source corresponding to the target business can be adapted from multiple data sources based on the configuration file of the target business. The target data source corresponding to the target business refers to the data source that stores the business records of the target business. The business records are generated after the business requests of the target business are processed.

[0039] For example, if the target business is order payment, the target data source is Canal, and the business request is an order payment request, then the business records generated after processing the order payment request could be: "Order 1001: Amount 500 yuan, User ID 101, Payment time 2025-07-01 10:00:00" and "Order 1002: Amount 300 yuan, User ID 102, Payment time 2025-07-01 10:05:00". Here, "1001" and "1002" are the order IDs.

[0040] S102. Obtain multiple business records of the target business from the target data source.

[0041] The target data source is used to listen to and store multiple business records of the target business. These multiple business records are generated after processing multiple business requests of the target business, and each business record corresponds to one business request.

[0042] Taking Canal as an example, Canal is used to continuously monitor the binlog of the MSQL database. When new business records are generated, Canal is used to encapsulate these business records in a specific format (such as JSON format).

[0043] Each business record includes: information about the requester of the corresponding business request, which is the user ID. The requester of the business request is used to initiate the business request. For example, the requester's information can be user 101, indicating that the requester with user ID 101 initiated the business request.

[0044] S103. Based on the requester information in multiple business records, group the multiple business records to obtain multiple business groups.

[0045] Based on the requester information in multiple business records, the multiple business records are grouped to divide the business records of the same requester into one business group, thus obtaining multiple business groups. Different business groups are business groups corresponding to different requesters.

[0046] In some embodiments, multiple business records are evenly distributed among the business groups by calculating the hash value of the user ID and taking the modulo of the number of groups.

[0047] In some embodiments, based on the requester's information, the getGroupShardingKey method is used to group multiple business records so that multiple threads can consume business records in different groups in parallel. In addition, a LinkedHashMap can be used to ensure the order of business records within each business group, thereby improving processing efficiency.

[0048] S104. Using multiple consumption threads corresponding to multiple business groups, the pre-configured business consumption logic of the target data source is called to consume each business record in the multiple business groups in turn.

[0049] Multiple consumer threads corresponding to multiple business groups are launched, with one consumer thread corresponding to each business group. Each consumer thread in a business group invokes the pre-configured business consumption logic of the target data source to sequentially consume the business records within each business group. The business records within each business group are sequential, meaning their order corresponds to the processing time of the business request for that business record, such as payment time. This means that while performing parallel consumption across different business groups, the consumption order of the business records within each group is also taken into account. This ensures that the consumption order of business records matches the processing order of business requests, preventing business logic errors caused by disordered consumption order (e.g., processing refunds before payments).

[0050] It should be noted that business consumption refers to obtaining business records from a target data source and performing business data processing according to the business consumption logic of the target data source. For example, if the target data source is Canal, Canal's business consumption logic could be to synchronize the amount and user ID in the business records to a distributed search and analysis engine (such as Elasticsearch), so that the distributed search and analysis engine can update the user's balance based on the user ID and amount.

[0051] For example, if the target data source is Kafka, the business consumption logic of Kafka can be to synchronously send a payment success message to the requester. Of course, the target data source can also be Elastic Job, ZMQ, etc. The specific target data source and corresponding business consumption logic can be determined according to business needs. The above is just an exemplary description provided in this embodiment, and this embodiment is not limited thereto.

[0052] In the data processing method based on multiple data sources provided in this embodiment, data source adaptation is performed based on the business configuration file, which reduces the complexity of database source adaptation. Furthermore, by consuming each business record in multiple business groups in parallel, efficiency and order are balanced, which improves the overall throughput and processing efficiency, and satisfies the consumption strategy of balancing data order and efficiency.

[0053] In an alternative implementation, the method may further include:

[0054] If the business consumption of a target business record in any business group fails, the business consumption logic of the target data source will be invoked according to the pre-configured retry strategy abstract class (RetryStrategy abstract class) inherited by multiple data sources, and the business consumption of the target business record will be performed again.

[0055] The retry strategy abstract class is used to define the data source retry strategy. Multiple data sources inherit from the retry strategy abstract class, which means that multiple data sources adopt a unified data source retry strategy.

[0056] The target business record is any business record that failed to be consumed in any of the multiple business groups. If the target business record in any business group fails to be consumed, the business consumption logic is invoked according to the pre-configured unified data source retry strategy to re-consume the target business record.

[0057] It should be noted that a unified retry strategy abstract class indicates the data source retry strategy for different data sources. If the consumption of the target business record fails, the business record will be consumed again according to the unified data source retry strategy. In other words, the unified implementation of retry logic significantly reduces the amount of code, lowers development costs, improves the success rate of business consumption processing, optimizes resource utilization, and shortens fault recovery time.

[0058] Figure 2 A flowchart illustrating the data processing method based on multiple data sources provided in this application embodiment. Figure 2 ,like Figure 2 As shown, in an optional implementation, the data source retry strategy includes: a local retry interval and a local retry count threshold.

[0059] The local retry interval is the time interval for re-consuming business locally, and the local retry threshold is the maximum number of local retries.

[0060] Based on the retry strategy abstract class inherited from multiple pre-configured data sources, the business consumption logic of the target data source is invoked to re-consume the target business records, which may include:

[0061] S201. If the local retry switch is on and the current number of local retries has not exceeded the local retry threshold, then according to the local retry interval, the business consumption logic of the target data source is called to consume the target business record until the local consumption of the target business record is successful or the current number of local retries reaches the local retry threshold.

[0062] It should be noted that step S201 is the implementation process of the local retry mechanism. If the local retry switch is on and the current number of local retries has not exceeded the local retry threshold, then after the local retry interval, the business consumption logic of the target data source is called to re-consume the target business record until the local consumption of the target business record is successful or the current number of local retries reaches the local retry threshold.

[0063] The local retry mechanism has two possible outcomes: one is that the local consumption of the target business record is successful when the current number of local retries is less than or equal to the local retry threshold; the other is that the local consumption of the target business record fails when the current number of local retries is equal to the local retry threshold.

[0064] In some embodiments, the following can be configured in the retry strategy configuration (RetryStrategyConfig): the status of the local retry switch (ENABLE_LOCAL_RETRY), the local retry threshold (MAX_LOCAL_RETRY), and the local retry interval (LOCAL_RETRY_WAIT_MILLS). After the consumption of the target business record fails, the local retry (canLocalRetry) method can be used to determine whether the local retry switch is enabled and whether the current number of local retries exceeds the local retry threshold. If the local retry switch is enabled and the current number of local retries does not exceed the local retry threshold, the local retry condition is met. Then, the local retry mechanism is started after the local retry interval, that is, the business consumption logic of the target data source is called to re-consume the target business record. This process is repeated until the local consumption of the target business record is successful or the current number of local retries reaches the local retry threshold.

[0065] For example, if the local retry threshold is 3 times and the local retry interval is 1 second, the business consumption logic of the target data source is called again after 1 second, and the target business record is consumed again. This process is repeated until the local consumption of the target business record is successful or the current local retry count reaches 3 times.

[0066] For example, if the target data source is Canal, Canal's business consumption logic can be invoked to retrieve the user's latest balance from MySQL based on the user ID in the target business record and synchronize it to the distributed search and analysis engine.

[0067] For example, if the target data source is Kafka, the business consumption logic of Kafka can be invoked, and the target will also record the corresponding payment success message to the requester.

[0068] In an optional implementation, the data source retry strategy further includes a delay retry time configuration. The delay retry time configuration includes multiple delay times, for example, SYSTEM_RETRY_WAIT_TIME_MIN_LIST = [0,1,2,4,...] (unit: minutes), where 0,1,2,4 represent a first delay time of 0 minutes, a second delay time of 1 minute, a third delay time of 2 minutes, and a fourth delay time of 4 minutes, respectively.

[0069] In some embodiments, during local retries, if the consumption of the target service record in any service group fails, the consumption of other service records following the target service record in any group is blocked until the local consumption of the target service record is successful when the current number of local retries is less than or equal to the local retries threshold, or the target service record is delivered to the delayed retry queue, at which point other service records are consumed normally.

[0070] Based on a pre-configured retry strategy abstract class inherited from multiple data sources, the business consumption logic of the target data source is invoked to re-consume the target business records. This may also include:

[0071] S202. If the local consumption of the target business record fails when the current local retry count reaches the local retry count threshold, the target business record will be delivered to the delayed retry queue.

[0072] For example, if the local retry threshold is 3 times, and the local consumption of the target business record fails after 3 local retries, a delayed retry is triggered. The target business record is delivered to the delayed retry queue using an asynchronous retry bus (AsyncRetryBus). The delayed retry queue can be middleware, such as Kafka.

[0073] The delayed retry queue includes multiple target service records where local retries failed when the current number of local retries reached the local retry threshold.

[0074] It should be noted that once a business record is delivered to the delayed retry queue, it will not block the consumption of other business records following that business record in the business group.

[0075] S203. Obtain the current delay time based on the delay retry time configuration.

[0076] For the target business record, if the delay retry is performed for the first time, the first delay time is obtained from the delay retry time configuration as the current delay time. If the delay retry is performed for the second time, the second delay time is obtained from the delay retry time configuration as the current delay time, and so on.

[0077] S204. Based on the current delay time and the current random time, obtain the current delay retry interval of the target service record.

[0078] The current random time is a randomly generated time, such as 5 minutes, 3 minutes, etc. The current delay time and the current random time are added together to obtain the current delay retry interval.

[0079] It should be noted that by superimposing the current random time to obtain the current delay retry interval, the business records in the delay retry queue are avoided from being consumed in a cluster, which further improves the retry efficiency of business consumption.

[0080] S205. Based on the current delay retry interval of the target business record, call the business consumption logic of the target data source to re-consume the target business record in the delay retry queue until the delay consumption of the target business record is successful or the current delay retry count reaches the delay retry count threshold.

[0081] After the current delay retry interval for the target business record, the business consumption logic of the target data source is invoked to re-consume the target business record in the delay retry queue until the delayed consumption of the target business record is successful or the current delay retry count reaches the delay retry count threshold. The delay retry count threshold can be controlled by the length of the wait time list (maxDelayRetryTimes). Furthermore, a single delay retry refers to executing the action of invoking the business consumption logic of the target data source to re-consume the target business record in the delay retry queue once, based on the current delay retry interval.

[0082] It should be noted that steps S202-S205 are the implementation process of the delayed retry mechanism. The delayed retry mechanism has two possible outcomes: one is that the delayed consumption of the target business record is successful when the current number of delayed retries is less than or equal to the delayed retry threshold; the other is that the delayed consumption of the target business record fails when the current number of delayed retries is equal to the delayed retry threshold.

[0083] In some embodiments, if the delayed consumption of the target service record fails when the current number of delayed retries reaches the delayed retries threshold, an alarm message can be pushed to the operations and maintenance personnel so that they can manually investigate and handle the issue. For example, if the system fails after 3 local retries and N delayed retries, it triggers an alarm mechanism to push an alarm message to the operations and maintenance equipment.

[0084] In the data processing method based on multiple data sources provided in this embodiment, after the consumption of the target business record fails, a layered retry mechanism is adopted, combining local retries (instant fault tolerance) and delayed retries (asynchronous compensation). Retry storms are avoided by using random timing, and retry logic is implemented uniformly, which greatly reduces the amount of code, lowers development costs, improves the success rate of business consumption processing, optimizes resource utilization, and shortens fault recovery time.

[0085] In some embodiments, the local retry interval, local retry count threshold, and delayed retry time can be dynamically configured in the configuration center. This enables centralized management of parameters such as the number of retries and the retry interval, supports dynamic adjustment at runtime, and adapts to different business scenarios.

[0086] Figure 3 A flowchart illustrating the data processing method based on multiple data sources provided in this application embodiment. Figure 3 ,like Figure 3 As shown, in an optional implementation, step S205, which involves invoking the business consumption logic of the target data source based on the current delay retry interval of the target business record, and re-consuming the target business record in the delay retry queue, may include:

[0087] S301. Obtain the number of records for each service record in the delayed retry queue.

[0088] The delayed retry queue includes multiple target service records where local retries failed when the current number of local retries reached the local retry threshold.

[0089] S302. If the number of records reaches the preset threshold, the consumption priority of each business record in the delayed retry queue is obtained based on the information of the requester in each business record in the delayed retry queue.

[0090] If the number of records reaches the preset threshold, it means that the delayed retry queue is due to a large backlog of business records.

[0091] Among them, the requester level is determined based on the requester's information. The requester level and consumption priority are mapped. The higher the requester level, the higher the consumption priority, and the lower the requester level, the lower the consumption priority.

[0092] By querying the mapping relationship based on the requester's level corresponding to the requester's information in each business record in the delayed retry queue, the consumption priority of each business record in the delayed retry queue can be determined. For example, if the requester's information indicates that the requester is a VIP user, then the consumption priority of the corresponding business record is higher than that of ordinary user business records.

[0093] S303. If the consumption priority of the target business record is higher than that of other business records in the delayed retry queue, then the current delayed retry interval of the target business record is shortened according to the delayed retry interval of other business records. Based on the shortened delayed retry interval, the business consumption logic of the target data source is called to re-consume the target business record in the delayed retry queue.

[0094] If the consumption priority of the target business record is higher than that of other business records in the delayed retry queue, then the current delayed retry interval of the target business record is shortened according to the delayed retry interval of other business records, so that the current delayed retry interval of the target business record is less than the delayed retry interval of other business records.

[0095] In other words, the waiting time for the target business record is shortened, and after the shortened delay retry time, the business consumption logic of the target data source is invoked to re-consume the target business record in the delay retry queue.

[0096] In some embodiments, the service records in the delayed retry queue are arranged in ascending order according to their current delayed retry time. If the number of records in the delayed retry queue reaches a preset threshold, it indicates that the delayed retry queue is too crowded. In this case, high-priority service records may not be able to be retried for a long time due to queue accumulation. Based on this, by obtaining the consumption priority of each service record, and when the consumption priority of the target service record is higher than the consumption priority of other service records in the delayed retry queue, the current delayed retry interval of the target service record is shortened, so that the high-priority service record can preempt the consumption order of the low-priority service record, thereby ensuring that the high-priority service record is consumed as soon as possible.

[0097] In the delayed retry queue, each business record is arranged in ascending order of its current delayed retry interval, with the smallest current delayed retry interval at the head of the queue and the largest at the tail. For example, if the preset threshold is 800 records, and the delayed retry queue contains 1000 records, the queue will become too cluttered, triggering a preemption of retry resources. This means that by identifying the consumption priority of each business record in the delayed retry queue, a high-priority record, such as "Order 1001: Amount 500 yuan, User ID 101, Payment Time 2025-07-01 10:00:00", is removed from its current position and directly inserted at the head of the queue. This allows it to skip lower-priority records like "Order 1002: Amount 300 yuan, User ID 102, Payment Time 2025-07-01 10:05:00", ensuring that high-priority records are processed first and that timely consumption is guaranteed.

[0098] In the data processing method based on multiple data sources provided in this embodiment, by identifying consumption priorities to preempt resource retry, it is further ensured that high-priority business records are consumed in a timely manner, sacrificing some experience of low-value businesses and minimizing the impact on high-value businesses.

[0099] Figure 4 A flowchart illustrating the data processing method based on multiple data sources provided in this application embodiment. Figure 4 ,like Figure 4 As shown, in an optional implementation, the method may further include:

[0100] S401, Business consumption failure type for retrieving target business records.

[0101] If the consumption of the target business record fails, the business consumption failure type is obtained. The consumption failure type may include network timeout (network jitter), downstream service downtime (downstream overload), and business record format error.

[0102] Downstream overload could be caused by a full downstream service connection pool, resulting in no available connections and thus preventing the establishment of connections with downstream services. For example, a downstream service could be a distributed search and analysis engine, meaning that the amount in the business records cannot be synchronized to the distributed search and analysis engine.

[0103] Incorrect business record format can be a problem, such as the absence of a user ID in the business record. Without a user ID, even if a retrieval is performed, the amount in the business record cannot be synchronized to the distributed search and analysis engine because the user ID cannot be found.

[0104] Network timeouts and downstream service outages are considered preset retry types, while incorrect service record formats are considered preset no-retry types. Service records are only re-consumed under preset retry types. For preset no-retry types, service records are not re-consumed, but alarm information can be pushed to the maintenance equipment so that maintenance personnel can manually troubleshoot and handle the issues.

[0105] S402. If the business consumption failure type is a temporary fault in the preset retry type, then shorten the retry interval according to the historical recovery cycle corresponding to the temporary fault.

[0106] Temporary failures are those that can be recovered after a short wait, and network timeouts are considered temporary failures.

[0107] If the business consumption failure type is a temporary fault in the preset retry type, the historical recovery period corresponding to the temporary fault is obtained, and the retry time is shortened according to the historical recovery period corresponding to the temporary fault. The retry interval here may include the local retry interval and / or the current delayed retry interval. The historical recovery period corresponding to the temporary fault is the time between the occurrence of the temporary fault and the restoration of normal consumption within the historical period.

[0108] For example, to address network jitter, the local retry interval is shortened from 1 second to 500 ms.

[0109] S403. If the business consumption failure type is a systemic failure in the preset retry type, the retry interval shall be extended according to the historical recovery cycle corresponding to the systemic failure.

[0110] Systemic failures are those that require a long wait to recover from, and downstream service outages fall under the category of systemic failures.

[0111] If the business consumption failure type is a systemic failure in the preset retry type, the historical recovery period corresponding to the systemic failure is obtained, and the retry time is shortened according to the historical recovery period corresponding to the systemic failure. The retry interval here may include the local retry interval and / or the current delayed retry interval. The historical recovery period corresponding to the systemic failure is the time between the occurrence of the systemic failure and the restoration of normal consumption within the historical period.

[0112] For example, if the downstream database connection pool is full, the local retry interval will be extended to 1.5 times the connection pool release cycle to avoid cascading failures. In other words, if there is a problem with the downstream interface, it will repeatedly try to retry, which will aggravate the impact of the failure, prolong the failure recovery time and aggravate congestion.

[0113] For example, by analyzing historical network jitter recovery cycle data and finding it to be 2 minutes, the second delay retry interval can be adjusted to 2 minutes to better match the recovery cycle and improve the data retry success rate.

[0114] In some embodiments, a "circuit breaker-recovery" mechanism can be triggered for systemic failures. This means that after a specified number of retries, the retries are paused for a specified duration before gradually resuming. For example, after 10 retries, the retries are paused for 5 minutes before gradually resuming.

[0115] In the data processing method based on multiple data sources provided in this embodiment, by identifying the failure type, the consistency retry strategy is intelligently adjusted to avoid mindless looping retries that could overwhelm downstream services, prolong failure time, and exacerbate the impact of failures, thereby enabling adaptive adjustment of the retry strategy.

[0116] Figure 5 A flowchart illustrating the data processing method based on multiple data sources provided in this application embodiment. Figure 5 ,like Figure 5 As shown, in an optional implementation, step S101 above, which adapts multiple data sources according to the configuration file of the target service to determine the target data source corresponding to the target service, may include:

[0117] S501. Determine the data source type corresponding to the target business based on the data source annotation information in the configuration file.

[0118] S502. Determine the target data source from multiple data sources that corresponds to the data source type.

[0119] The configuration file for the target business includes: data source annotation information. The data source annotation information is extracted from the configuration file, and the data source type corresponding to the target business is determined based on the data source annotation information. The data source corresponding to this data source type among multiple data sources is then selected as the target data source.

[0120] For example, the `@ZMQListener` annotation, when adapted to a ZMQ data source, identifies the ZMQ data source as the target data source among multiple data sources; the `@KafkaConsumer` annotation, when adapted to a Kafka data source, identifies the Kafka data source as the target data source among multiple data sources; the `@CanalConsumer` annotation, when adapted to a Canal data source, identifies the Canal data source as the target data source among multiple data sources; and the `@ElasticJobAutoConf` annotation, when adapted to an Elastic Job data source, identifies the Elastic Job data source as the target data source among multiple data sources.

[0121] In the data processing method based on multiple data sources provided in this embodiment, the data source type is automatically identified based on annotations. When adding a new data source, only the annotation needs to be extended, without modifying the core framework, which has strong scalability.

[0122] In an alternative implementation, the method may further include:

[0123] If a new data source is added, the business consumption logic of the new data source will be configured using a pre-defined standardized interface, and the new data source will inherit the retry strategy abstract class.

[0124] The default standardized interface is the standardized interface provided by the SDK access layer. If a new data source is added, the business consumption logic of the new data source will be configured using the standardized interface, and the new data source will inherit the retry strategy abstract class. The new data source and multiple data sources inherit the retry strategy abstract class and adopt a unified data source retry strategy.

[0125] In other words, through SDK-based design, users only need to implement the business consumption logic of the new data source to quickly access the data processing system based on multiple data sources, making access convenient and significantly reducing the access threshold.

[0126] The following describes the data processing system based on multiple data sources provided in this application.

[0127] Figure 6 This is a schematic diagram of the architecture of a data processing system based on multiple data sources provided in the embodiments of this application, such as... Figure 6 As shown, it includes: SDK access layer, multi-data source adaptation module, retry strategy module, consumption processing module, and configuration module.

[0128] The SDK access layer provides standardized interfaces, allowing data sources to access the system by inheriting the retry strategy abstract class.

[0129] See Figure 6 The parent class of the retry abstract classes (ZMQRetry, JobRetry, KafkaRetry, CanalRetry) of ZMQ, Elastic Job, Kafka, and Canal data sources is the retry strategy abstract class, so that ZMQ, Elastic Job, Kafka, and Canal data sources inherit the retry strategy abstract class.

[0130] The system automatically registers consumer instances (using the postProcessAfterInitialization method of the RetryStrategyBoot class), eliminating the need to concern users with underlying consumption scheduling and retry logic. For example, when connecting to a Canal data source, users only need to implement the business consumption logic, while retry scheduling is automatically handled by the CanalRetry parent class.

[0131] In addition, when adding a new data source, you can integrate it into the system by inheriting the retry strategy abstract class and implementing the corresponding annotation parsing and consumption logic.

[0132] Multi-data source adaptation module: compatible with data sources such as Canal, Kafka, Elastic Job, and ZMQ, and uses data source annotations for data source adaptation.

[0133] Retry strategy module: includes local retry, delayed retry, and adaptive retry mechanisms, and is responsible for executing retry and circuit breaker decisions.

[0134] The consumption processing module is used to implement the logic of multi-threaded batch consumption and batch fragmented consumption of sequential messages. Messages within each business group are processed sequentially (implemented through CountDownLatch). The processing between different business groups is parallel, which improves the overall throughput. The processing results are summarized in a unified manner, and different logic is executed according to success or failure. Failed messages enter the retry process, and after the maximum number of retries is reached, they are transferred to manual processing.

[0135] The configuration module is used to manage retry parameters (number of times, interval), data source parameters, etc., and supports dynamic adjustment. Among them, the data source parameters include the connection information of the data source, such as the IP address and port number of the server where the data source is located.

[0136] Figure 7 A schematic diagram of the service grouping provided in the embodiments of this application, such as Figure 7 As shown, business records are grouped according to user ID, resulting in group 1, group 2, and group 3. Group 1 includes messages (i.e., business records) 1.1, 1.2, and 1.3; group 2 includes messages 2.1, 2.2, and 2.3; and group 3 includes messages 3.1, 3.2, and 3.3. Consumer thread 1, consumer thread 2, and consumer thread 3 respectively consume business records from groups 1, 2, and 3.

[0137] If the processing result is successful, the offset is committed to record the position of the processed messages so that message consumption can continue from the last successfully processed position, instead of starting from the beginning or missing some messages.

[0138] If the processing result is unsuccessful, the retry logic will be entered. If the number of retries is exceeded, manual processing will be initiated. If the number of retries is not exceeded, the data will be consumed again.

[0139] The following example uses the "order payment data synchronization" scenario on an e-commerce platform. Assume the data source is Canal, and the data to be processed consists of 100 order payment records, i.e., business records (order IDs: 1001 - 1100, where order 1001 is a VIP user order with higher priority). The specific process is as follows:

[0140] Step 1: SDK Integration and Data Source Adaptation

[0141] SDK Integration: The Canal data source is integrated into the system via the SDK. In the actual code implementation, a class, such as OrderConsumer, is created that inherits from the CanalRetry abstract class. This class only needs to implement the business logic (consumeLogic). The system automatically registers this consumer instance through the RetryStrategyBoot class. During the registration process, the RetryStrategyBoot class scans all classes in the project that inherit from CanalRetry, identifies the OrderConsumer class, instantiates it, and records it in the system's consumer instance list for subsequent data consumption operations.

[0142] Data Source Adaptation: The multi-data source adaptation module identifies the data source type as Canal using the `@CanalConsumer` annotation. For example, in the corresponding configuration file, add the `@CanalConsumer` annotation to the relevant data source configuration section, and specify the Canal data source connection information in the annotation parameters, such as the IP address and port number of the server where the data source resides. By parsing the annotation parameters, the Canal data source driver is automatically loaded, completing the adaptation process with the Canal data source. This ensures that the system can correctly retrieve data from Canal, and the entire adaptation process does not require modification of the core framework code, guaranteeing the system's stability and scalability.

[0143] Step 2: Data Reception and Consumption Processing

[0144] Data Reception: Canal continuously monitors the binlog of the MySQL database. When new order payment records are generated, for example, order 1001: amount 500 yuan, user ID 101, payment time 2025-07-01 10:00:00; order 1002: amount 300 yuan, user ID 102, payment time 2025-07-01 10:05:00, etc., totaling 100 records, Canal encapsulates this data in a specific format (such as JSON format) and pushes it to the system's data receiving interface.

[0145] Consumption Processing Strategy: The consumption processing module adopts a "sequential message multi-threaded batch consumption" strategy. First, orders are grouped by user ID. For example, orders 1001-1020 are divided into group 1, 1021-1040 into group 2, and so on, for a total of 5 groups. During grouping, the system calculates the hash value of the user ID and then modulo the group number to evenly distribute orders among the groups. Next, 5 threads are started to process different groups in parallel. Within each thread, a queue data structure is used to store the order data for that group, ensuring that orders within each group are processed in ID order. For example, within group 1, order 1001 is retrieved from the queue first for processing, then 1002 is processed. This balances efficiency and sequentiality, preventing business logic errors caused by disordered order processing, such as processing payment follow-up operations before processing refund operations.

[0146] Step 3: Consumption failure and local retry triggering

[0147] Consumption failure scenario: When processing order 1001, if the downstream reconciliation system (such as a distributed search and analysis engine) fails to respond within the specified timeout period (such as 5 seconds) due to network jitter, resulting in consumption failure, a timer will be started after sending the order data to the reconciliation system interface. If the timer does not receive a response from the reconciliation system within 5 seconds, a timeout exception will be captured, and the consumption will be determined to have failed.

[0148] Local retry mechanism: The retry strategy module triggers local retries. First, it checks the RetryStrategyConfig configuration, which shows that the local retry switch is enabled, the maximum number of retries is set to 3, and the retry interval is 1 second. The canLocalRetry method is used to determine if the current failure meets the local retry conditions (i.e., the maximum number of retries has not been exceeded). Then, after waiting 1 second, the business consumption logic is invoked again to attempt to synchronize the data of order 1001 to the downstream reconciliation system, performing a maximum of 3 local retry operations.

[0149] Step 4: Local retry failure and delayed retry triggering

[0150] Delayed retry trigger condition: If three local retries fail due to the downstream network not recovering, the system will trigger a delayed retry. An asynchronous retry bus (AsyncRetryBus) is used to redeliver order 1001 messages to the delayed retry consumer (i.e., the delayed retry queue). The delay time is configured as SYSTEM_RETRY_WAIT_TIME_MIN_LIST = [0,1,2,4,...] (unit: minutes), and a random time is added to avoid message congestion.

[0151] When calculating the delay time, the time values ​​are first retrieved sequentially from the configuration list. The first delay retry time is 0 minutes + a random time. Assuming the randomly generated time is 3 minutes, the first delay retry interval is 3 minutes. The second time, 1 minute is retrieved from the configuration list. Assuming the randomly generated time is 4 minutes, the second delay retry interval is 5 minutes, and so on.

[0152] Step 5: Adaptive retry dynamic adjustment

[0153] Failure data collection and analysis: Collect failure data and determine the failure type.

[0154] Adaptive adjustment strategy implementation: trigger adaptive retry adjustment based on failure type.

[0155] Step 6: Retry resource preemption

[0156] By monitoring the length of the delayed queue, when the queue length exceeds a set threshold (such as 800 messages), it is determined that a queue backlog has occurred. For example, if 1,000 messages are waiting to be processed in the queue, order 1001 (VIP high priority) is blocked by low priority messages and cannot enter the consumption and processing stage in time.

[0157] The retry strategy module triggers resource preemption: Based on preset priority rules (such as VIP user orders having higher priority than regular user orders), it identifies the high priority attribute of order 1001, then removes order 1001 from its current position and inserts it directly into the head of the queue, allowing it to skip low-priority messages in the queue (such as regular user orders 1050-1060) and enter the consumption thread for processing first, ensuring the timely synchronization of high-priority order data.

[0158] Step 7: Dynamic configuration takes effect

[0159] Configuration modification: Operations personnel adjusted the local retry threshold from 3 to 5 in the RetryStrategyConfig configuration item through the configuration center and saved the changes. Subsequent orders (1002-1100) will automatically have their local retries adjusted according to the new configuration without requiring a system restart. This achieves dynamic configuration updates and real-time effectiveness, ensuring the system can quickly adapt to changes in business needs.

[0160] Step 8, Final Processing Result

[0161] Successful processing: During the second delayed retry of order 1001, the downstream network returned to normal, and the retry was successful. The processing result log for order 1001 was recorded, including the synchronization success time and a summary of the order data synchronized to the reconciliation system. Simultaneously, a confirmation message was sent to the Canal data source, submitting the processing result to update the consumption offset and mark the message as processed, preventing duplicate consumption.

[0162] Handling failures: If all retries (3 local attempts + N delayed attempts) fail, an alarm mechanism will be triggered to send an alarm message to the operations and maintenance personnel for manual troubleshooting and handling.

[0163] Through the detailed process described above, a closed-loop process is completed, from data source access, consumption, retry to final processing, achieving efficient and reliable data processing. It is also highly compatible with multiple data sources and can be seamlessly integrated into core business systems such as e-commerce and finance.

[0164] Based on the same inventive concept, this application also provides a data processing device based on multiple data sources, which corresponds to the data processing method based on multiple data sources. Since the principle of the device in this application is similar to the data processing method based on multiple data sources described above, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be described again.

[0165] Figure 8 This is a schematic diagram of the structure of a data processing device based on multiple data sources provided in an embodiment of this application. This device can be integrated into a computer device.

[0166] like Figure 8 As shown, the device may include:

[0167] The determination module 601 is used to adapt multiple data sources according to the configuration file of the target business and determine the target data source corresponding to the target business.

[0168] The acquisition module 602 is used to acquire multiple business records of the target business from the target data source. The multiple business records are records generated after processing multiple business requests of the target business. Each business record includes: information of the requester of the corresponding business request.

[0169] The processing module 603 is used to group multiple business records based on the information of the requester in multiple business records to obtain multiple business groups;

[0170] The processing module 603 is also used to use multiple consumption threads corresponding to multiple business groups to call the pre-configured business consumption logic of the target data source and perform business consumption on each business record in the multiple business groups in sequence.

[0171] In an optional implementation, the processing module 603 is further configured to:

[0172] If the consumption of a target business record in any business group fails, the business consumption logic of the target data source is invoked according to the retry strategy abstract class inherited by multiple pre-configured data sources, and the target business record is consumed again. The retry strategy abstract class is used to indicate the retry strategy of the data source.

[0173] In an optional implementation, the processing module 603 is specifically used for:

[0174] If the local retry switch is enabled and the current number of local retries has not exceeded the local retry threshold, then according to the local retry interval, the business consumption logic of the target data source is invoked to re-consume the target business record until the local consumption of the target business record is successful or the current number of local retries reaches the local retry threshold.

[0175] In an optional implementation, the processing module 603 is specifically used for:

[0176] If the local consumption of the target business record fails when the current local retry count reaches the local retry count threshold, the target business record will be delivered to the delayed retry queue.

[0177] Get the current delay time based on the delay retry time configuration;

[0178] Based on the current delay time and the current random time, obtain the current delay retry interval of the target business record;

[0179] Based on the current delay retry interval of the target business record, the business consumption logic of the target data source is invoked to re-consume the target business record in the delay retry queue until the delay consumption of the target business record is successful or the current delay retry count reaches the delay retry count threshold.

[0180] In an optional implementation, the processing module 603 is specifically used for:

[0181] Get the number of records for each business record in the delayed retry queue;

[0182] If the number of records reaches the preset threshold, the consumption priority of each business record in the delayed retry queue is obtained based on the information of the requester in each business record in the delayed retry queue.

[0183] If the consumption priority of the target business record is higher than that of other business records in the delayed retry queue, then the current delayed retry interval of the target business record is shortened according to the delayed retry interval of other business records. Based on the shortened delayed retry interval, the business consumption logic of the target data source is invoked to re-consume the target business record in the delayed retry queue.

[0184] In an optional implementation, the processing module 603 is further configured to:

[0185] Retrieve the business consumption failure types from the target business records;

[0186] If the business consumption failure type is a temporary fault in the preset retry type, then the retry interval is shortened according to the historical recovery cycle corresponding to the temporary fault.

[0187] If the business consumption failure type is a systemic failure in the preset retry types, the retry interval will be extended according to the historical recovery cycle corresponding to the systemic failure.

[0188] In an optional implementation, the determining module 601 is specifically used for:

[0189] Based on the data source annotation information in the configuration file, determine the data source type corresponding to the target business;

[0190] Determine the target data source from multiple data sources that corresponds to the data source type.

[0191] In an optional implementation, the processing module 603 is further configured to:

[0192] If a new data source is added, the business consumption logic of the new data source will be configured using a pre-defined standardized interface, and the new data source will inherit the retry strategy abstract class.

[0193] The processing flow of each module in the device and the interaction flow between each module can be referred to the relevant descriptions in the above method embodiments, and will not be detailed here.

[0194] Figure 9 A schematic diagram of the structure of the computer device provided in the embodiments of this application, such as... Figure 9 As shown, the device may include a processor 701, a memory 702, and a bus 703. The memory 702 stores machine-readable instructions that can be executed by the processor 701. When the computer device is running, the processor 701 communicates with the memory 702 through the bus 703, and the processor 701 executes the machine-readable instructions to perform the above-described method embodiments.

[0195] This application also provides a computer-readable storage medium storing a computer program, which is executed by a processor to perform the above-described method embodiments.

[0196] In this embodiment, the computer program, when run by the processor, can also execute other machine-readable instructions to perform other methods as described in the embodiments. For details on the specific execution steps and principles, please refer to the description of the embodiments, which will not be repeated here.

[0197] In the embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and there may be other division methods in actual implementation. Furthermore, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Additionally, the coupling or direct coupling or communication connection shown or discussed may be through some communication interface; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0198] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0199] In addition, the functional units in the embodiments provided in this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0200] If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0201] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. In addition, the terms "first", "second", "third", etc. are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0202] Finally, it should be noted that the above embodiments are merely specific implementations of this application, used to illustrate the technical solutions of this application, and not to limit them. The protection scope of this application is not limited thereto. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments, or make equivalent substitutions for some of the technical features, within the scope of the technology disclosed in this application; and these modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application. All should be covered within the protection scope of this application. Therefore, the protection scope of this application should be determined by the protection scope of the claims.

Claims

1. A data processing method based on multiple data sources, characterized in that, include: Based on the configuration file of the target business, adapt multiple data sources to determine the target data source corresponding to the target business; Multiple business records of the target business are obtained from the target data source. The multiple business records are records generated after processing multiple business requests of the target business. Each business record includes: information of the requester of the corresponding business request. Based on the requester information in the multiple business records, the multiple business records are grouped to obtain multiple business groups; Multiple consumption threads corresponding to the multiple business groups are used to call the pre-configured business consumption logic of the target data source to sequentially consume each business record in the multiple business groups; The method further includes: If the business consumption of the target business record in any business group fails, the business consumption logic of the target data source is called according to the retry strategy abstract class inherited by the multiple data sources in the pre-configured retry strategy abstract class, and the business consumption of the target business record is performed again. The retry strategy abstract class is used to indicate the retry strategy of the data source. The data source retry strategy includes: a local retry interval and a local retry count threshold. The step of calling the business consumption logic of the target data source based on the pre-configured retry strategy abstract class inherited by the multiple data sources to re-consume the target business record includes: If the local retry switch is on and the current number of local retries has not exceeded the local retry threshold, then according to the local retry interval, the business consumption logic of the target data source is invoked to re-consume the target business record until the local consumption of the target business record is successful or the current number of local retries reaches the local retry threshold. The data source retry strategy further includes: a delayed retry time configuration; the step of calling the business consumption logic of the target data source according to the pre-configured retry strategy abstract class inherited by the multiple data sources to re-consume the target business record also includes: If the local consumption of the target service record fails when the current local retry count reaches the local retry count threshold, the target service record will be delivered to the delayed retry queue. Based on the configured delay retry time, obtain the current delay time; Based on the current delay time and the current random time, obtain the current delay retry interval of the target service record; Based on the current delay retry interval of the target business record, the business consumption logic of the target data source is invoked to re-consume the target business record in the delay retry queue until the delay consumption of the target business record is successful or the current delay retry count reaches the delay retry count threshold.

2. The method according to claim 1, characterized in that, The step of invoking the business consumption logic of the target data source based on the current delay retry interval of the target business record, and re-consuming the target business record in the delay retry queue, includes: Obtain the number of records for each service record in the delayed retry queue; If the number of records reaches a preset threshold, the consumption priority of each service record in the delayed retry queue is obtained based on the information of the requester in each service record in the delayed retry queue. If the consumption priority of the target business record is higher than that of other business records in the delayed retry queue, then the current delayed retry interval of the target business record is shortened according to the delayed retry interval of the other business records, and the business consumption logic of the target data source is invoked according to the shortened delayed retry interval to re-consume the target business record in the delayed retry queue.

3. The method according to claim 1, characterized in that, The method further includes: Obtain the business consumption failure type of the target business record; If the business consumption failure type is a temporary fault in the preset retry types, then the retry interval is shortened according to the historical recovery cycle corresponding to the temporary fault. If the business consumption failure type is a systemic failure among the preset retry types, the retry interval is extended according to the historical recovery period corresponding to the systemic failure.

4. The method according to claim 1, characterized in that, The step of adapting multiple data sources according to the configuration file of the target service to determine the target data source corresponding to the target service includes: Based on the data source annotation information in the configuration file, determine the data source type corresponding to the target business; The data source corresponding to the data source type is determined from the plurality of data sources as the target data source.

5. The method according to claim 1, characterized in that, The method further includes: If a new data source is added, the business consumption logic of the new data source is configured using a preset standardized interface, and the new data source inherits the retry strategy abstract class.

6. A computer device, characterized in that, include: The device includes a processor, a memory, and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is in operation, the processor communicates with the memory via the bus, and the processor executes the machine-readable instructions to perform the method according to any one of claims 1-5.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, performs the method according to any one of claims 1-5.

Images