Data synchronization method and device and storage medium

Through task thread pool management of concurrent API calls and message queue delivery, the problems of high task management costs, breakpoint continuous running and hot updates in API data synchronization methods are solved, and efficient recovery and update capabilities of data synchronization are realized, improving the system management efficiency and data synchronization accuracy.

CN119088596AInactive Publication Date: 2024-12-06SHENGGUANG MARKETING GRP CO LTD
View PDF 6 Cites 0 Cited by

Patent Information

Application Number
CN202411184602.7
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-27
Publication Date
2024-12-06
Estimated Expiration
Not applicable · inactive patent

AI Technical Summary

Technical Problem

The existing API data synchronization methods have problems such as high task management costs, no breakpoint running, and no hot updates at any time, resulting in increased data synchronization management costs, risk of data loss and low system maintenance efficiency.

Method used

Manage concurrent API calls through the task thread pool and deliver task scheduling parameters to the message queue, ensuring that the task can be resumed after interruption, and supports hot updates at any time.

Benefits of technology

It effectively reduces task management costs, ensures that tasks can be restored and continued to be processed after interruption, and supports hot updates in the system at any time, improving the accuracy of data synchronization and operation and maintenance deployment efficiency.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119088596A_ABST
    Figure CN119088596A_ABST
Patent Text Reader

Abstract

The invention discloses a data synchronization method and device and a storage medium. The method comprises the steps of obtaining a data synchronization task; according to the data synchronization task, obtaining attribute information of target synchronization data and calling information of N target APIs, and packaging the attribute information and the calling information into task scheduling parameters; wherein N is greater than or equal to 1; delivering the task scheduling parameters to a message queue; task scheduling parameters distributed to the task thread pool are obtained from the message queue; and when available threads exist in the task thread pool, concurrently calling all the target APIs based on the distributed task scheduling parameters and the task thread pool, and obtaining target synchronization data defined by the attribute information. According to the method, concurrent API calling is managed through the task thread pool, so that the task management cost is effectively reduced; and meanwhile, task scheduling parameters are delivered to the message queue, so that the tasks can be recovered and continuously processed after being interrupted, and hot updating at any time is supported.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a data synchronization method, device and computer-readable storage medium. Background Art

[0002] Application Program Interface (API) is an important tool that allows different systems or applications to communicate with each other. A system can request data provided by another system through a third-party API to complete the data synchronization task. However, the existing API data synchronization task still has the following technical defects:

[0003] (1) Increased task management costs: Existing methods usually manage tasks at the granularity of a single API synchronization task. If the current data synchronization task requires multiple API data sources, the synchronization task of each data source needs to be processed and monitored separately, which obviously increases the management cost.

[0004] (2) No support for breakpoint resumption: Existing API data synchronization tasks generally lack an effective task interruption protection mechanism. Once a task is abnormally interrupted or restarted, the system cannot resume the task from the breakpoint and can only start from the beginning. This not only increases the risk of data loss, but also reduces the accuracy of data synchronization.

[0005] (3) No support for hot updates at any time: The existing method can only restart at a fixed time and cannot achieve hot updates at any time. This means that each update requires planned downtime for system maintenance. Summary of the invention

[0006] The purpose of the embodiments of the present invention is to provide a data synchronization method, device and computer-readable storage medium, which effectively reduce the cost of task management by managing concurrent API calls through a task thread pool; at the same time, by delivering task scheduling parameters to a message queue, it ensures that the task can be restored and continued after interruption, and supports hot updates at any time.

[0007] A first aspect of the present invention provides a data synchronization method, including:

[0008] Get data synchronization tasks;

[0009] According to the data synchronization task, the attribute information of the target synchronization data and the calling information of N target APIs are obtained, and encapsulated into the task scheduling parameters; wherein N≥1;

[0010] Delivering the task scheduling parameters to a message queue;

[0011] Obtaining task scheduling parameters assigned to the task thread pool from the message queue;

[0012] When there are available threads in the task thread pool, all the target APIs are called concurrently based on the assigned task scheduling parameters and the task thread pool, and the target synchronization data defined by the attribute information is obtained.

[0013] Optionally, the task scheduling parameters include: task message subject, task timestamp, API account information, API authorization information, data time range, data type, data synchronization quantity and MD5 encryption information.

[0014] Optionally, obtaining the task scheduling parameters assigned to the task thread pool from the message queue includes:

[0015] Reading the task scheduling parameters to be assigned from the message queue;

[0016] Determining whether the task scheduling parameters to be assigned meet cache uniqueness;

[0017] If the cache uniqueness is not satisfied, then the data synchronization task is stopped;

[0018] If the cache uniqueness is satisfied, determining whether the number of requests for processing the data synchronization task exceeds a preset task frequency control limit;

[0019] If the preset task frequency control limit is exceeded, the task scheduling parameters to be assigned are returned to the message queue;

[0020] If the preset task frequency control limit is not exceeded, the task scheduling parameters to be allocated are allocated to the task thread pool.

[0021] Optionally, after obtaining the task scheduling parameters assigned to the task thread pool from the message queue, the method further includes:

[0022] The task thread pool is dynamically configured according to the assigned task scheduling parameters so that the task thread pool does not exceed the corresponding API frequency control limit when concurrently calling any one of the target APIs.

[0023] Optionally, the task thread pool does not set a task queue cache.

[0024] Optionally, the method further comprises:

[0025] When there is no available thread in the task thread pool, transmitting the assigned task scheduling parameters to the main thread;

[0026] Determine whether the main thread is in a blocked state;

[0027] If yes, returning the assigned task scheduling parameters to the message queue;

[0028] If not, all the target APIs are called concurrently based on the assigned task scheduling parameters and the main thread to obtain the target synchronization data.

[0029] Optionally, before the concurrently calling of all the target APIs, the allocated task scheduling parameters are cached.

[0030] Optionally, during the concurrent calling of all the target APIs, if any of the target APIs is abnormally interrupted, the abnormal interruption is recorded in an error handling table; based on a preset time period, the error handling table is backtracked, and the target API where the abnormal interruption occurs is called again.

[0031] A second aspect of the present invention provides a data synchronization device, which is used to implement the data synchronization method described in the first aspect. The device includes:

[0032] Synchronization task acquisition module, used to obtain data synchronization tasks;

[0033] A scheduling parameter encapsulation module, used to obtain attribute information of target synchronization data and call information of N target APIs according to the data synchronization task, and encapsulate them into task scheduling parameters; wherein N≥1;

[0034] A scheduling parameter delivery module, used to deliver the task scheduling parameters to the message queue;

[0035] A scheduling parameter allocation module, used for obtaining task scheduling parameters allocated to the task thread pool from the message queue;

[0036] The scheduling parameter execution module is used to concurrently call all the target APIs based on the assigned task scheduling parameters and the task thread pool when there are available threads in the task thread pool, and obtain the target synchronization data defined by the attribute information.

[0037] An embodiment of the third aspect of the present invention further provides a computer-readable storage medium, which includes a stored computer program; wherein, when the computer program is running, the device where the computer-readable storage medium is located controls the data synchronization method described in any one of the first aspects above.

[0038] Compared with the prior art, the embodiments of the present invention provide a data synchronization method, device and computer-readable storage medium, the method comprising: obtaining a data synchronization task; according to the data synchronization task, obtaining the attribute information of the target synchronization data and the call information of N target APIs, and encapsulating them into the task scheduling parameters; wherein N≥1; delivering the task scheduling parameters to the message queue; obtaining the task scheduling parameters assigned to the task thread pool from the message queue; when there are available threads in the task thread pool, based on the assigned task scheduling parameters and the task thread pool, concurrently calling all target APIs, and obtaining the target synchronization data defined by the attribute information. The present invention manages concurrent API calls through the task thread pool, effectively reducing the cost of task management; at the same time, by delivering the task scheduling parameters to the message queue, it ensures that the task can be restored and continued to be processed after interruption, and supports hot updates at any time. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 It is a flowchart of an embodiment of the data synchronization method provided by the present invention;

[0040] Figure 2 is a flow chart of another embodiment of the data synchronization method provided by the present invention;

[0041] Figure 3 It is a structural diagram of an embodiment of a data synchronization device provided by the present invention. DETAILED DESCRIPTION

[0042] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this technical field without creative work are within the scope of protection of the present invention.

[0043] In order to better understand the technical solutions in the embodiments of the present invention, the technical terms involved are described below:

[0044] (1) Data synchronization task: First, extract data from various source systems; then, clean, integrate, calculate, and format the extracted data to make it suitable for the requirements of the target repository; finally, load the converted data into the target repository; the source systems include relational databases, NoSQL databases, files (such as CSV, XML, JSON, etc.), and third-party open interfaces (APIs).

[0045] (2) Message Queue (MQ): is a method for cross-process communication or thread communication within the same process. It allows independent applications to communicate by sending and receiving messages (also called messages or events) without directly connecting the sender and the receiver. This communication method is often used to implement scenarios such as asynchronous processing, application decoupling, traffic peak shaving, and message communication.

[0046] (3) Message Listener: This involves creating a Consumer instance, configuring the Consumer, subscribing to one or more Topics, and registering a message listener to process received messages.

[0047] (4) Message Acknowledgment (ACK): A mechanism that ensures that a message is confirmed after being processed by the consumer. It is mainly used to ensure reliable message transmission and consumer reading progress management.

[0048] (5) Thread Pool: A component used to manage and reuse threads. Using a thread pool can avoid frequent creation and destruction of threads, thereby saving system resources and improving the response speed and throughput of the program.

[0049] (6) Resume from Break Point: This is a technique or method that allows execution to continue from the break point after a task or program is interrupted for some reason. Specifically, when an error, exception, manual pause or other interruption occurs during the execution of a task or program, execution can continue from the interrupted location (breakpoint) instead of starting from the beginning or skipping that part.

[0050] See also Figure 1 , is a flow chart of an embodiment of the data synchronization method provided by the present invention.

[0051] The first aspect of the present invention provides a data synchronization method, including steps S1 to S5, which are specifically as follows:

[0052] Step S1: Get data synchronization tasks;

[0053] Step S2: According to the data synchronization task, the attribute information of the target synchronization data and the calling information of N target APIs are obtained, and encapsulated into the task scheduling parameters; wherein N≥1;

[0054] Step S3: delivering the task scheduling parameters to the message queue;

[0055] Step S4: Obtaining task scheduling parameters assigned to the task thread pool from the message queue;

[0056] Step S5: When there are available threads in the task thread pool, all the target APIs are concurrently called based on the assigned task scheduling parameters and the task thread pool, and the target synchronization data defined by the attribute information is obtained.

[0057] During specific implementation, the embodiment of the present invention will periodically trigger the task of splicing (encapsulating) the task of task scheduling parameters and locate a specific data synchronization task. The splicing task is processed by a designated processor, that is, the processor will assemble the scheduling parameters required for this data synchronization task according to the task conditions of data synchronization (such as the attribute information of the target synchronization data and the call information of N target APIs), and complete the task scheduling parameter splicing (encapsulating) task. Then send the assembled task scheduling parameters to the designated message queue. At this time, the system will cyclically monitor the message queue, consume messages, and obtain the task scheduling parameters assigned to the task thread pool; if there are available threads in the task thread pool, the assigned task scheduling parameters will be asynchronously executed. In the task thread pool, the system will find a specific executor, which is responsible for calling the API and saving the data returned by the API, thereby completing the entire data synchronization task process. The splicing task can also encapsulate and splice the scheduling parameters required by different data synchronization tasks according to a unified data model, generate batches of task scheduling parameters at one time, and send them to the message queue corresponding to the designated task message topic.

[0058] It is worth noting that the embodiment of the present invention makes full use of the message persistence function and message confirmation mechanism of the message queue by delivering the task scheduling parameters to the message queue, so that even if the system fails or restarts, the task scheduling parameters are still in the message queue, and the data synchronization task will not be lost, and the processing can continue after the system is restored; based on this, the embodiment of the present invention can support hot updates at any time (that is, support system updates and restarts at any time) on the one hand, improve the efficiency of system operation and maintenance deployment; on the other hand, it can improve the security of concurrent execution of multi-node tasks and solve the data confusion and conflict problems caused by multi-task recovery. At the same time, through the message confirmation mechanism, that is, the consumer sends a confirmation to the message queue only after successfully processing the message (task scheduling parameters), which ensures that the task can be restored and continued after the interruption, and the breakpoint can be resumed. In addition, the embodiment of the present invention can also manage concurrent API calls through the task thread pool, effectively reducing the cost of task management.

[0059] In an optional embodiment, the task scheduling parameters include: task message subject, task timestamp, API account information, API authorization information, data time range, data type, data synchronization quantity and MD5 encryption information.

[0060] It should be noted that the task scheduling parameters required for splicing and encapsulating data synchronization tasks are the prerequisites for executing data synchronization. In an embodiment of the present invention, the data synchronization tasks are pre-split into before the task, during execution, execution exception, successful execution, and re-execution. Before the synchronization task is executed, a unified execution task data model is abstracted, including basic scheduling parameters (such as task message subject, task timestamp, API account information, API authorization information, data time range, data type, data synchronization quantity, and MD5 encryption information), but in the specific implementation, the data model / task scheduling parameters mainly include the following points:

[0061] (1) API account information: There is a division of account permissions in the API, which contains information such as account level, account classification, and account authorization, which is mainly used to call data within the scope of the API account.

[0062] (2) Data time range (start time - end time): Define the data source time range, which is divided into year, month, day, and hour. According to different time ranges, obtain different data source time range data.

[0063] (3) Time caliber: It is divided into real-time time caliber, batch processing time caliber, and custom time caliber. It is mainly used to perform different scheduling methods according to different time calibers. For example, the real-time caliber will be scheduled once every minute or every second, while the batch processing caliber will be scheduled once a day.

[0064] (4) Data Dimensions: Data is aggregated into corresponding dimensional data according to different data dimensions, which can be divided into primary account dimensional data, primary account sub-account dimensional data, time dimension, other business dimension indicator detailed data, custom aggregated dimensional data, etc. Depending on the data dimensions, the API call request will return different dimensional report data.

[0065] (5) Data type: Identify the data source type, such as file, JSON, CSV, and other data format types.

[0066] (6) Data synchronization quantity: the quantity of data requested each time.

[0067] (7) Whether to retry after interruption: If an exception occurs during data synchronization, whether to retry (not to start from the beginning, but to re-execute from the task breakpoint);

[0068] (8) Task timestamp: records the task generation timestamp (the timestamp of a single API call in a data synchronization task), identifies the synchronization time of the finest-grained data, and is used for real-time calculation after the subsequent data synchronization is completed.

[0069] (9) Task message topic: The specific message queue topic to which the task scheduling parameters need to be sent.

[0070] (10) API authorization information: API authorization token, used for authentication when requesting to call the API.

[0071] (11) MD5 encryption information: The task parameter assembly characters are encrypted with MD5, which is used to determine whether the task is repeated during task execution. In other words, the entire text content of the task splicing parameters is encrypted into MD5 characters, and the uniqueness of MD5 is used to determine whether the data synchronization task is repeated. The specific judgment logic is: add the MD5 characters of the first data synchronization task to the Redis cache, and perform cache uniqueness verification for subsequent tasks, that is, search the cache to see if there is the same MD5 character as the current data synchronization task. If so, it means that this task has been executed before and should not be repeated; if not, it means that this task is a new task and needs to be executed.

[0072] The following is an example of task scheduling parameters provided by an embodiment of the present invention:

[0073]

[0074]

[0075] See also Figure 2 , is a flow chart of another embodiment of the data synchronization method provided by the present invention.

[0076] In an optional embodiment, obtaining the task scheduling parameters assigned to the task thread pool from the message queue includes:

[0077] Reading the task scheduling parameters to be assigned from the message queue;

[0078] Determining whether the task scheduling parameters to be assigned meet cache uniqueness;

[0079] If the cache uniqueness is not satisfied, then the data synchronization task is stopped;

[0080] If the cache uniqueness is satisfied, determining whether the number of requests for processing the data synchronization task exceeds a preset task frequency control limit;

[0081] If the preset task frequency control limit is exceeded, the task scheduling parameters to be assigned are returned to the message queue;

[0082] If the preset task frequency control limit is not exceeded, the task scheduling parameters to be allocated are allocated to the task thread pool.

[0083] During specific implementation, the embodiment of the present invention will pre-set the topic and number of partitions of the good news queue, and set a consumer group (used to listen to messages on a specific topic). For example, the topic of task A is TOPIC_A, the number of partitions is 6, and the consumer group is GROUP_A. Consumer group GROUP_A will listen to TOPIC_A, obtain the message content (task scheduling parameters to be assigned) and deserialize and decode the message content, and can obtain the task code TASK_A, and then find the corresponding processor HANDLER_A according to TASK_A. HANDLER_A is used to perform cache uniqueness judgment and task frequency control limit judgment. The specific processing logic is as follows: First, the cache uniqueness judgment is performed on the task scheduling parameters to be assigned to determine whether the current data synchronization task (task scheduling parameters to be assigned) is executed repeatedly. If the cache uniqueness is not satisfied, it indicates that the task is executed repeatedly and the current data synchronization task should be terminated; if the cache uniqueness is satisfied, it indicates that the task is a new task, and then continue to determine whether the data synchronization task exceeds the preset task frequency control limit; the task frequency control limit determination process is as follows: initialize the current limiter according to the task code TASK_A, and each task corresponds to a different current limiter, for example, TASK_A QPS=100, TASK_B QPS=50; then find the corresponding current limiter for current limiting determination, if overclocked, enter the message response NACK, according to the characteristics of the message queue, at this time it will refuse to process the message and block the queue; if not overclocked, start the task thread pool, and assign the task scheduling parameters to be assigned to the task thread pool.

[0084] The following are examples of current limiter initialization and current limiter judgment provided by an embodiment of the present invention:

[0085] Current limiter initialization example:

[0086]

[0087] In an optional embodiment, after obtaining the task scheduling parameters assigned to the task thread pool from the message queue, the method further includes:

[0088] The task thread pool is dynamically configured according to the assigned task scheduling parameters so that the task thread pool does not exceed the corresponding API frequency control limit when concurrently calling any one of the target APIs.

[0089] During specific implementation, after the task scheduling parameters corresponding to the task code are assigned to the task thread pool, the task thread pool is dynamically configured, which may be to initialize the configuration parameters of the task thread pool, such as setting a fixed number of core threads and a maximum number of threads, so as to prevent the API frequency control limit from being exceeded when calling the API concurrently. The embodiment of the present invention can flexibly adjust the number of threads for concurrent API calls, support the concurrent execution of data synchronization tasks under distributed conditions, and solve the problem of API requests exceeding the frequency control limit. It is worth noting that one data synchronization task corresponds to only one task thread pool.

[0090] In an optional embodiment, the task thread pool does not set a task queue cache.

[0091] It should be noted that the task queue cache is used to store tasks waiting to be executed by threads in the thread pool, but the task thread pool in the embodiment of the present invention does not set a task queue cache, which can effectively reduce memory usage.

[0092] In an optional embodiment, the method further includes:

[0093] When there is no available thread in the task thread pool, transmitting the assigned task scheduling parameters to the main thread;

[0094] Determine whether the main thread is in a blocked state;

[0095] If yes, returning the assigned task scheduling parameters to the message queue;

[0096] If not, all the target APIs are called concurrently based on the assigned task scheduling parameters and the main thread to obtain the target synchronization data.

[0097] Specifically, in combination with the above embodiments, the task thread pool does not set a task queue cache. On the one hand, the message queue can be used to buffer task requests during peak periods, so that tasks can queue in the queue and wait for processing, rather than directly letting the task thread pool process all tasks. This buffering mechanism effectively prevents the problem of thread pool resource exhaustion caused by instantaneous high concurrency; on the other hand, after obtaining the assigned task scheduling parameters, the task thread pool will dynamically configure the parameters of the task thread pool (flexibly adjust the number of concurrent API call threads) according to the current task scheduling parameters (data synchronization tasks) to avoid the thread pool exceeding the API frequency control limit and thread pool overload.

[0098] In addition, since the task thread pool does not set the task queue cache, and in order to avoid blocking the task thread pool, a custom rejection strategy is set (such as Figure 2As shown in the figure): When the task thread pool is full (no idle threads), the new task (assigned task scheduling parameters) is handed over to the main thread for execution instead of placing it in the memory queue; this has the following advantages: (1) It can reduce memory usage: avoid tasks piling up in the memory queue (task queue cache), effectively solving the problem of system memory overflow. (2) It can block the execution of the main thread and then block queue consumption: by handing over new tasks to the main thread for execution, the main thread is forced to temporarily stop receiving more data synchronization tasks while processing tasks, which effectively controls the rate of task submission and prevents overload.

[0099] When the task thread pool has available resources, the new task will be transferred to the API call executor for further processing. The embodiment of the present invention ensures efficient utilization of system resources and maintains the stability and performance of the system.

[0100] In an optional embodiment, before the concurrently calling of all the target APIs, the assigned task scheduling parameters are cached.

[0101] Specifically, in the process of API calling the executor, first add aspect processing before and after the API call, such as saving the task (task scheduling parameters) to the Redis cache before the API call; then after the API call is completed, delete the relevant task scheduling parameters in the Redis cache and obtain the target data returned by the API, such as Figure 2 The embodiment of the present invention provides an effective task interruption protection mechanism, that is, when an abnormal interruption occurs during the execution of a data synchronization task (assigned task scheduling parameters) by a task thread pool, the mechanism can provide compensation.

[0102] In an optional embodiment, during the concurrent calling of all the target APIs, if any of the target APIs is abnormally interrupted, the abnormal interruption is recorded in an error handling table; based on a preset time period, the error handling table is backtracked, and the target API where the abnormal interruption occurred is called again.

[0103] It should be noted that for each data synchronization task, the system will open a task thread pool for execution. In the thread executor, multiple APIs are called concurrently according to the pre-set number of threads, thereby improving the efficiency of API calls. In specific implementation, different APIs can be classified into business categories, such as account categories, fund categories, and permission categories. Different types of APIs correspond to different API call executors, and these executors will also add interception processors in pre- and post-processing, such as log printing and execution time recording. During the API call process, if an exception occurs, it will be handled by a uniformly defined exception handler and the exception will be recorded in the error handling table. The system will automatically backtrack the error handling table at regular intervals and retry the call to the abnormal API, thereby effectively solving the problem of data loss caused by abnormal interruption when calling the API. That is, when encountering errors, exceptions, manual pauses or other system interruptions, it can continue execution from the interrupted location (breakpoint) instead of starting from the beginning or skipping the part.

[0104] In addition, after completing all API calls, the system will delete the relevant task scheduling parameters in the Redis cache and parse the target data returned by the API, which may contain JSON and text formats. The parsed target data will be sent to the data warehouse through the message queue for cleaning. At the same time, the system will also send the amount of data returned by this API call through a unified message topic, and clean the content of these messages into an API call data verification result table for secondary data verification. The verification method is to compare the amount of aggregated data with the amount of detailed data to ensure the consistency and accuracy of the data.

[0105] See also Figure 3 , is a structural diagram of an embodiment of a data synchronization device provided by the present invention.

[0106] A second aspect of the present invention provides a data synchronization device, which is used to implement the data synchronization method described in any one of the embodiments of the first aspect, and the device includes:

[0107] A synchronization task acquisition module 11 is used to acquire data synchronization tasks;

[0108] The scheduling parameter encapsulation module 12 is used to obtain the attribute information of the target synchronization data and the calling information of N target APIs according to the data synchronization task, and encapsulate them into the task scheduling parameters; wherein N≥1;

[0109] The scheduling parameter delivery module 13 is used to deliver the task scheduling parameters to the message queue;

[0110] A scheduling parameter allocation module 14 is used to obtain task scheduling parameters allocated to the task thread pool from the message queue;

[0111] The scheduling parameter execution module 15 is used to concurrently call all the target APIs based on the assigned task scheduling parameters and the task thread pool when there are available threads in the task thread pool, and obtain the target synchronization data defined by the attribute information.

[0112] Optionally, the task scheduling parameters in the scheduling parameter encapsulation module 12 include: task message subject, task timestamp, API account information, API authorization information, data time range, data type, data synchronization quantity and MD5 encryption information.

[0113] Optionally, the scheduling parameter allocation module 14 includes:

[0114] A reading unit, used for reading the task scheduling parameters to be assigned from the message queue;

[0115] A cache uniqueness determination unit, used to determine whether the task scheduling parameters to be assigned meet cache uniqueness;

[0116] A stopping unit, used for stopping the execution of the data synchronization task if the cache uniqueness is not satisfied;

[0117] A task frequency control limit judgment unit, configured to judge whether the number of requests for processing the data synchronization task exceeds a preset task frequency control limit if cache uniqueness is satisfied;

[0118] A returning unit, used for returning the task scheduling parameters to be allocated to the message queue if the preset task frequency control limit is exceeded;

[0119] The allocating unit is used to allocate the task scheduling parameters to be allocated to the task thread pool if the preset task frequency control limit is not exceeded.

[0120] Optionally, after the scheduling parameter allocation module 14, the device further includes:

[0121] The dynamic configuration module is used to dynamically configure the task thread pool according to the assigned task scheduling parameters so that the task thread pool does not exceed the corresponding API frequency control limit when concurrently calling any one of the target APIs.

[0122] Optionally, the task thread pool does not set a task queue cache.

[0123] Optionally, the device further comprises:

[0124] A main thread transmission module, used for transmitting the assigned task scheduling parameters to the main thread when there are no available threads in the task thread pool;

[0125] A blocking judgment module, used to judge whether the main thread is in a blocked state;

[0126] A returning module, configured to return the assigned task scheduling parameters to the message queue if yes;

[0127] The synchronization data acquisition module is used to concurrently call all the target APIs based on the assigned task scheduling parameters and the main thread to obtain the target synchronization data if not.

[0128] Optionally, before the concurrently calling of all the target APIs, the allocated task scheduling parameters are cached.

[0129] Optionally, during the concurrent calling of all the target APIs, if any of the target APIs is abnormally interrupted, the abnormal interruption is recorded in an error handling table; based on a preset time period, the error handling table is backtracked, and the target API where the abnormal interruption occurs is called again.

[0130] It should be noted that the data synchronization device provided in the second aspect of the present invention can implement all the processes of the data synchronization method described in the first aspect of the present invention. The functions of each module in the device and the technical effects achieved are respectively the same as the functions and technical effects achieved by the data synchronization method described in the first aspect of the present invention, and will not be repeated here.

[0131] An embodiment of the third aspect of the present invention also provides a computer-readable storage medium, which includes a stored computer program; wherein, when the computer program is running, it controls the device where the computer-readable storage medium is located to execute the data synchronization method described in any embodiment of the first aspect above.

[0132] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the technical principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A data synchronization method, characterized in that: include: Get data synchronization tasks; According to the data synchronization task, the attribute information of the target synchronization data and the calling information of N target APIs are obtained, and encapsulated into the task scheduling parameters; wherein N≥1; Delivering the task scheduling parameters to a message queue; Obtaining task scheduling parameters assigned to the task thread pool from the message queue; When there are available threads in the task thread pool, all the target APIs are called concurrently based on the assigned task scheduling parameters and the task thread pool, and the target synchronization data defined by the attribute information is obtained.

2. The data synchronization method according to claim 1, characterized in that: The task scheduling parameters include: task message subject, task timestamp, API account information, API authorization information, data time range, data type, data synchronization quantity and MD5 encryption information.

3. The data synchronization method according to claim 1, characterized in that: The step of obtaining the task scheduling parameters assigned to the task thread pool from the message queue includes: Reading the task scheduling parameters to be assigned from the message queue; Determining whether the task scheduling parameters to be assigned meet cache uniqueness; If the cache uniqueness is not satisfied, then the data synchronization task is stopped; If the cache uniqueness is satisfied, determining whether the number of requests for processing the data synchronization task exceeds a preset task frequency control limit; If the preset task frequency control limit is exceeded, the task scheduling parameters to be assigned are returned to the message queue; If the preset task frequency control limit is not exceeded, the task scheduling parameters to be allocated are allocated to the task thread pool.

4. The data synchronization method according to claim 1, characterized in that: After obtaining the task scheduling parameters assigned to the task thread pool from the message queue, the method further includes: The task thread pool is dynamically configured according to the assigned task scheduling parameters so that the task thread pool does not exceed the corresponding API frequency control limit when concurrently calling any one of the target APIs.

5. The data synchronization method according to claim 1, characterized in that: The task thread pool does not set a task queue cache.

6. The data synchronization method according to claim 1, characterized in that: The method further comprises: When there is no available thread in the task thread pool, transmitting the assigned task scheduling parameters to the main thread; Determine whether the main thread is in a blocked state; If yes, returning the assigned task scheduling parameters to the message queue; If not, all the target APIs are called concurrently based on the assigned task scheduling parameters and the main thread to obtain the target synchronization data.

7. The data synchronization method according to claim 1 or 5, characterized in that: Before the concurrently calling all the target APIs, the assigned task scheduling parameters are cached.

8. The data synchronization method according to claim 7, characterized in that: In the process of concurrently calling all the target APIs, if any one of the target APIs is abnormally interrupted, the abnormal interruption is recorded in an error handling table; Based on a preset time period, the error handling table is traced back, and the target API where the abnormal interruption occurs is called again.

9. A data synchronization device, characterized in that: include: Synchronization task acquisition module, used to obtain data synchronization tasks; A scheduling parameter encapsulation module, used to obtain attribute information of target synchronization data and call information of N target APIs according to the data synchronization task, and encapsulate them into task scheduling parameters; wherein N≥1; A scheduling parameter delivery module, used to deliver the task scheduling parameters to the message queue; A scheduling parameter allocation module, used for obtaining task scheduling parameters allocated to the task thread pool from the message queue; The scheduling parameter execution module is used to concurrently call all the target APIs based on the assigned task scheduling parameters and the task thread pool when there are available threads in the task thread pool, and obtain the target synchronization data defined by the attribute information.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored computer program; wherein, when the computer program is run, it controls the device where the computer-readable storage medium is located to execute the data synchronization method according to any one of claims 1 to 8.

Citation Information

Patent Citations

  • Method for solving internet of things cloud service avalanche effect based on JAVA multi-thread technology

    CN110287013A

  • Data synchronization method and device, computer equipment and storage medium

    CN110704458A

  • Data synchronization system and method

    CN112615907A

  • Data updating method and device, equipment and storage medium

    CN114647656A

  • Local mirror image uploading optimization method and system

    CN117155956A