Data cache synchronization method and system based on flexible configuration of SaaS platform

Abstract

The invention discloses a data cache synchronization method and system based on flexible configuration of an SaaS platform. The invention relates to the technical field of SaaS platforms. And loading a user-defined configuration file which comprises a synchronization strategy, a period and a mode of data caching. The system comprises an initialization data caching module, a synchronous control module, a configuration management module and a log monitoring module. Establishing connection with a data source according to specified data source information in the configuration file; initializing a local cache, and setting timeout time and concurrency size parameters of the cache according to configuration; according to the method, the user is allowed to perform customized configuration according to specific requirements, so that different service scenes can be more flexibly adapted, and the flexibility and response speed of services are improved. A user can adjust a synchronization strategy, a period and a mode according to service requirements, so that the use effect of the data cache is optimized. According to the invention, by introducing the distributed lock and the synchronization control mechanism, conflicts and race conditions in the data synchronization process can be avoided.

Description

Technical Field

[0001] The present invention relates to the technical field of SaaS platforms, and particularly to a flexible configuration-based data cache synchronization method and system for SaaS platforms. Background Art

[0002] With the rapid progress of cloud computing and big data technologies, software as a service (SaaS) platforms have become an essential infrastructure in many business scenarios. In these diverse application scenarios, the data cache synchronization mechanism is directly related to the response speed of application programs and the user interaction experience;

[0003] Traditional data cache synchronization strategies, such as Figures 2 to 3 shown, usually rely on static and rigid configuration modes, which significantly weaken their flexibility and configurability, thereby limiting their ability to meet the personalized needs of different user groups and complex business scenarios:

[0004] Traditional data cache synchronization solutions often adopt a "one-size-fits-all" design concept and lack necessary customization configuration options. This limitation not only hinders the system from being optimized according to specific user requirements but also restricts the dynamic adjustment and innovative development of business logic. In today's data-driven era, there are a wide variety of data sources and cache technologies are also changing rapidly. Traditional synchronization methods fail to effectively support the access of diverse data sources and seamless integration with various cache systems, which undoubtedly becomes a major bottleneck restricting business expansion and technological innovation.

[0005] As the business scale continues to expand and the data volume surges, the concurrent access volume that the system needs to handle also climbs accordingly. When dealing with such high-load scenarios, traditional synchronization strategies often show problems such as low efficiency and slow response, and it is difficult to meet the strict requirements of modern businesses for high performance and instantaneity. Moreover, in complex and changeable network environments and harsh operating conditions, the stability of existing solutions becomes a major hidden danger. Frequent system failures not only affect the user experience but may also cause irreversible damage to the business due to data inconsistency or loss.

[0006] Therefore, the present invention proposes a flexible configuration-based data cache synchronization method and system for SaaS platforms. Summary of the Invention

[0007] In view of this, the present invention hopes to provide a flexible configuration-based data cache synchronization method and system for SaaS platforms to solve or alleviate the technical problems existing in the prior art, that is, it cannot be customized according to the specific needs of users. It does not support multiple data sources and cache systems, restricting the development of the business. At the same time, in complex network environments and operating conditions, existing solutions are prone to failures, and at least provide a beneficial option; the technical solution of the present invention is implemented as follows:

[0008] First aspect, a data cache synchronization method and system based on flexible configuration of the SaaS platform:

[0009] (1) Overview:

[0010] The present invention aims to construct an efficient, stable and flexible data cache synchronization mechanism. By loading user-defined configuration files, corresponding data cache synchronization policies can be formulated and executed according to different scenarios and requirements. It not only supports multiple synchronization methods such as real-time synchronization, scheduled synchronization and incremental synchronization to meet the requirements of data timeliness and synchronization efficiency in different business scenarios, but also effectively copes with the data synchronization challenges in high-concurrency scenarios through the introduction of anti-breakthrough mechanisms and synchronization thread pool management, ensuring the stability and performance of the system. At the same time, it also pays attention to exception handling and feedback of synchronization results. By capturing and handling exception situations during the synchronization process, and providing detailed synchronization results and status information, the reliability and usability of the system are enhanced.

[0011] (2) Technical solutions:

[0012] To achieve the above technical objectives, after receiving the input data cache synchronization scheme activation instruction, the following operation steps are started.

[0013] 2.1 Step S1, configuration loading and initialization:

[0014] Load user-defined configuration files, including data cache synchronization policies, periods and methods.

[0015] Initialize the data cache module, synchronization control module, configuration management module and log monitoring module.

[0016] 2.1.1 Step S100, load configuration file:

[0017] Read the path of the user-defined configuration file. Use the configuration loader to load the content of the configuration file.

[0018] 2.1.2 Step S101, parse configuration parameters, including:

[0019] Parse the data cache synchronization policy, including real-time, scheduled and incremental;

[0020] Parse the synchronization period, including daily or hourly;

[0021] Parse the synchronization method, including full amount or increment.

[0022] 2.1.3 Step S102, initialize module

[0023] Initialize the data cache module and set the cache policy;

[0024] Initialize the synchronization control module and set the synchronization mode and parameters according to the configuration;

[0025] Initialize the configuration management module and save the parsed configuration parameters;

[0026] Initialize the log monitoring module and set the log record level and output method.

[0027] 2.2 Step S2, Data Engineering:

[0028] Establish a connection to the data source according to the data source information specified in the configuration file; Initialize the local cache and set the cache timeout and concurrency size parameters according to the configuration.

[0029] Preload the cache data to improve the subsequent data access speed.

[0030] 2.2.1 Step S200, Establish Data Source Connection:

[0031] Select the corresponding connection method according to the data source type (including database or API) specified in the configuration file; Configure the connection parameters, including the host name, port number, user name, and password.

[0032] 2.2.2 Step S201, Initialize Local Cache:

[0033] Set the cache timeout according to the configuration; Set the cache concurrency size parameters, including the maximum connection number and thread pool size; Initialize the cache storage structure, including key-value pairs and lists.

[0034] 2.2.3 Step S202, Preload Cache Data

[0035] Retrieve the data to be cached from the data source; Load the data into the local cache; Set the initial state of the cached data to valid or invalid.

[0036] 2.3 Step S3, Data Synchronization Execution:

[0037] Execute real-time synchronization operations, scheduled synchronization operations, or incremental synchronization operations according to the synchronization policy.

[0038] 2.3.1 Step S300, Judge Synchronization Policy:

[0039] Judge which synchronization operation to execute according to the synchronization policy parameters in the configuration management module:

[0040] If configured for real-time synchronization, listen for changes in the data source;

[0041] When the data source changes, immediately synchronize and update the local cache;

[0042] 2.3.2 Step S301, perform scheduled synchronization or incremental synchronization:

[0043] If configured for scheduled synchronization: set up a scheduled task according to the synchronization period parameter; trigger the synchronization operation at the specified time point to update the local cache.

[0044] If configured for incremental synchronization: record the data changes after the last synchronization; only synchronize the changed data part to reduce the synchronization overhead.

[0045] 2.4 Step S4, load anti-breakthrough and concurrency control:

[0046] The anti-breakthrough mechanism limits the number of origin requests to avoid cache breakthrough. Through the synchronization thread pool management, control the number of threads for concurrently executing synchronized data to ensure the stability of the system.

[0047] 2.4.1 Step S400, perform the anti-breakthrough mechanism:

[0048] Set the limit number of origin requests to the maximum number of requests per second; when the origin requests exceed the limit, perform cache degradation processing or return the default value.

[0049] 2.4.2 Step S401, manage the synchronization thread pool

[0050] Configure the number of threads and the queue size of the thread pool; use the thread pool to manage the execution of the synchronization operation to avoid system instability caused by excessive concurrency.

[0051] 2.5 Step S5, exception handling and recovery:

[0052] During the data synchronization process, capture and handle the occurring exception situations.

[0053] 2.5.1 Step S500, capture the exception:

[0054] During the data synchronization process, use a try-catch block to capture the occurring exception; record the exception information, including the exception type, occurrence time, and error location.

[0055] 2.5.2 Step S501, handle the exception:

[0056] Judge the handling method according to the exception type and perform retry, degradation, or ignore; if retry is required, set the number of retries and the interval; if the exception cannot be handled, record the log and notify the user or the upper-layer application.

[0057] 2.5.3 Step S502, recovery operation:

[0058] After the exception handling is completed, check whether the system status has returned to normal; if not, re-execute the synchronization operation or start the backup mechanism.

[0059] 2.6 Step S6, Synchronization Result Feedback:

[0060] Feed back the result of data synchronization to the user or the upper - layer application. If the synchronization fails, provide error information and a retry mechanism to facilitate the user to troubleshoot and fix problems.

[0061] That is, feed back the result of data synchronization (such as success, failure, partial success, etc.) to the user or the upper - layer application; if the synchronization is successful, provide the data version or timestamp after synchronization.

[0062] If the synchronization fails, generate error information, including an error code and an error description; at the same time, provide a retry mechanism to allow the user or the upper - layer application to re - trigger the synchronization operation. Finally, record the failure log to facilitate subsequent problem troubleshooting and system optimization.

[0063] (III) Mechanisms for Solving Technical Problems:

[0064] 3.1 Customized Configuration Mechanism:

[0065] Allow users to set parameters such as the synchronization policy, period, and method of data caching through a configuration file. This enables users to flexibly adjust the synchronization policy according to specific business requirements to meet personalized needs. Adopt a modular design so that different modules can be independently configured and extended. This design method not only improves the scalability of the system but also facilitates customized development according to different business requirements.

[0066] 3.2 Support for Multiple Data Sources and Cache Systems:

[0067] Regardless of whether the data source is a relational database, a NoSQL database, or other types of data storage systems, unified access and operations can be carried out. Similarly, cache system adaptation is provided to support multiple cache systems. This allows users to select the most suitable cache system according to business requirements without worrying about compatibility issues with the synchronization scheme.

[0068] 3.3 Fault Tolerance and Recovery Mechanism:

[0069] Built - in an exception - handling mechanism that can capture and handle various abnormal situations during the synchronization process. When a fault occurs, it will automatically attempt to resume the synchronization operation to ensure data integrity and consistency. For synchronization failures caused by network fluctuations or system load, etc., the solution provides a retry mechanism. By attempting the synchronization operation multiple times, the success rate of synchronization is improved, and the risk of data inconsistency caused by faults is reduced.

[0070] In a complex network environment and operating conditions, in order to avoid conflicts and race conditions during data synchronization, a distributed lock and synchronization control mechanism is introduced. These mechanisms can ensure that multiple synchronization tasks can correctly access and operate data when executed concurrently, thereby improving the stability and reliability of the system.

[0071] Second aspect, a flexible configuration-based data cache synchronization method and system for SaaS platform:

[0072] As Figure 5 shown, this system is used to implement the flexible configuration-based data cache synchronization method and system described above, and it includes:

[0073] (1) A configuration management module responsible for loading user-defined configuration files and parsing the synchronization strategy, period, and method parameters of the data cache.

[0074] (2) A data cache module, a synchronization control module, and a log monitoring module that are initialized according to the parameters parsed by the configuration management module and set their respective working modes and parameters.

[0075] Among them, a synchronization control module that establishes a connection with the data source according to the data source information specified in the configuration file. The synchronization control module decides to perform real-time synchronization, scheduled synchronization, or incremental synchronization operations according to the synchronization strategy provided by the configuration management module

[0076] Among them, the data cache module implements an anti-breakthrough mechanism to limit the number of requests to the source, and avoid a large number of requests directly impacting the data source when the cache fails or is not hit.

[0077] Among them, the synchronization control module manages through a synchronization thread pool to control the number of threads for concurrently synchronizing data, preventing the system from crashing or experiencing performance degradation due to excessive concurrency.

[0078] Among them, the synchronization control module is responsible for capturing and handling abnormal situations that occur during data synchronization, such as network errors, data source failures, etc.

[0079] Among them, the log monitoring module records the process and results of exception handling, providing a basis for subsequent fault troubleshooting and system optimization.

[0080] Among them, the synchronization control module feeds back the result of data synchronization to the user or the upper-layer application through the user interface module.

[0081] Among them, if the synchronization fails, the synchronization control module will generate an error message and a retry mechanism, and provide it to the user through the user interface module, facilitating the user to troubleshoot and repair problems.

[0082] Compared with the prior art, the beneficial effects of the present invention are:

[0083] I. Improving business flexibility and response speed: By allowing users to perform customized configurations according to specific requirements, the present invention can more flexibly adapt to different business scenarios, improving business flexibility and response speed. Users can adjust synchronization strategies, cycles, and methods according to business needs, thereby optimizing the use effect of data caching. By introducing a distributed lock and synchronization control mechanism, the present invention can avoid conflicts and race conditions during data synchronization, ensuring data consistency and integrity. This helps improve the robustness and reliability of the system and reduces business risks caused by data synchronization problems.

[0084] II. Supporting multiple data sources and caching systems to promote business development: The present invention supports multiple data sources, including relational databases, NoSQL databases, etc., breaking the limitations of data sources and enabling users to more flexibly select and use data sources. This helps enterprises integrate data resources from different sources, improving the utilization value of data and the development potential of the business.

[0085] III. Adaptation to multiple caching systems: By providing a caching system adaptation layer, the present invention supports multiple caching systems. Users can select the most suitable caching system according to business needs, thereby improving the efficiency and performance of data caching. This helps enterprises optimize data storage and access strategies and enhance the overall performance of the business system.

[0086] IV. Enhancing system stability and reliability: The solution of the present invention incorporates fault tolerance and recovery mechanisms such as exception handling and retry mechanisms, which can capture and handle various abnormal situations during synchronization, reducing the impact of faults on the business. This helps improve system stability and reliability and ensures the smooth progress of data caching synchronization operations. BRIEF DESCRIPTION OF THE DRAWINGS

[0087] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the following drawings are only some embodiments of the present application, and those of ordinary skill in the art can obtain other drawings without creative efforts based on these drawings.

[0088] Figure 1 is a schematic flowchart of the method of the present invention;

[0089] Figure 2 is a schematic diagram of the traditional data caching synchronization strategy (active pull mode);

[0090] Figure 3 is a schematic diagram of the traditional data caching synchronization strategy (push - pull mode);

[0091] Figure 4Schematic diagram of the SDK architecture of the present invention;

[0092] Figure 5 Schematic diagram of the system composition of the present invention. Detailed implementation manners

[0093] To make the above objects, features, and advantages of the present invention more obvious and understandable, the following will describe the detailed implementation manners of the present invention in conjunction with the accompanying drawings. Many specific details are set forth in the following description in order to fully understand the present invention. However, the present invention can be implemented in many other ways different from those described herein, and those skilled in the art can make similar improvements without departing from the connotation of the present invention. Therefore, the present invention is not limited by the specific embodiments disclosed below;

[0094] It should be noted that the various embodiments in this specification are described in a progressive manner, and the key points of each embodiment are the differences from other embodiments. The same or similar parts among the various embodiments can be referred to each other. For the devices disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple, and the relevant parts can be referred to the description of the method part.

[0095] Explanation of related terms:

[0096] (1) Synchronization strategy, period, and method of data caching: It refers to how to plan, when, and by what method to synchronize the updates of the data source to the cache.

[0097] (2) Cached data: A copy of data stored on a local or remote fast-access device, used to improve the data reading speed.

[0098] (3) Synchronization strategy: Determines the transmission rules and methods of data from the data source to the cache, including when to trigger synchronization, which data to synchronize, etc.

[0099] (4) Real-time synchronization operation: When the data source changes, immediately synchronize these changes to the cache to maintain data consistency.

[0100] (5) Scheduled synchronization operation: At regular time intervals, regularly synchronize data from the data source to the cache, regardless of whether the data source has changed.

[0101] (6) Incremental synchronization operation: Only synchronize the data that has changed in the data source since the last synchronization to reduce the amount and time of synchronized data.

[0102] (7) Synchronization thread pool management: By creating and managing a thread pool to execute synchronization tasks, improving the concurrency and efficiency of synchronization operations.

[0103] (9) Control the number of threads for concurrent execution of synchronous data: Limit the number of threads that perform synchronous operations simultaneously to prevent system resource overload.

[0104] (10) Retry mechanism: When the synchronization fails, automatically attempt to re-execute the synchronization operation to increase the success rate of synchronization.

[0105] Example 1: As Figure 1 、 4 shown, this example discloses a flexible configuration-based data cache synchronization method in an online car-hailing SaaS platform; Suppose an online car-hailing company develops a SAAS platform, which needs to efficiently and flexibly process data from multiple data sources (such as passenger requests, driver locations, order information, etc.) and be able to operate stably under complex network environments and operating conditions. Users (i.e., online car-hailing companies) hope to be able to customize and configure data caches, synchronization policies, etc. according to their specific needs to improve platform performance and user experience.

[0106] In this example, regarding step S1: Configuration loading and initialization:

[0107] Specifically, step S100: Load the configuration file: The user uploads or specifies a configuration file through the platform interface, which contains synchronization policies, cycles, and methods of data caching, etc. The system reads the configuration file path and uses a configuration loader (such as JSON, YAML parser) to load the content of the configuration file.

[0108] Specifically, step S101: Parse the configuration parameters: Parse the synchronization policy of data caching (such as real-time, timed, incremental). Parse the synchronization cycle (such as early morning every day, every hour). Parse the synchronization method (such as full synchronization of all order data, incremental synchronization of newly generated orders).

[0109] Specifically, step S102: Initialize the module:

[0110] Initialize the data cache module and set the cache policy (such as LRU, TTL) according to the configuration.

[0111] Initialize the synchronization control module and set the synchronization mode and parameters (such as the interface for real-time monitoring, the scheduler for timed tasks) according to the configuration.

[0112] Initialize the configuration management module to save the parsed configuration parameters for subsequent module use.

[0113] Initialize the log monitoring module and set the log record level (such as INFO, ERROR) and output method (such as console, file).

[0114] In this example, regarding step S2: Data engineering:

[0115] Specifically, step S200: Establish a data source connection: Based on the data source type specified in the configuration file (such as MySQL database, RESTful API), select the corresponding connection method. Configure connection parameters, such as the hostname, port number, username, and password of the database, or the URL and authentication information of the API.

[0116] Specifically, step S201: Initialize the local cache: Set the cache timeout according to the configuration, such as caching order information for 30 minutes. Set the concurrency size parameter of the cache, such as a maximum connection number of 100 and a thread pool size of 20. Initialize the cache storage structure, such as using Redis to store key-value pairs or Memcached to store lists.

[0117] Specifically, step S202: Preload cache data: Retrieve the data that needs to be cached from the data source, such as the distribution of drivers in popular areas and the order prices of common routes. Load the data into the local cache and set the initial state of the cache data to valid.

[0118] In this embodiment, regarding step S3: Data synchronization execution:

[0119] Specifically, step S300: Determine the synchronization strategy: Based on the synchronization strategy parameters in the configuration management module, determine which synchronization operation to execute. If configured for real-time synchronization, listen for changes in the data source. For example, when a new order request is initiated by a passenger, immediately synchronize and update the local cache.

[0120] Specifically, step S301: Execute scheduled synchronization or incremental synchronization: If configured for scheduled synchronization, set up a scheduled task according to the synchronization period parameter, such as synchronizing driver location information every hour. If configured for incremental synchronization, record the data changes after the last synchronization and only synchronize the newly generated order data to reduce the synchronization overhead.

[0121] In this embodiment, regarding step S4: Load anti-breakthrough and concurrency control:

[0122] Specifically, step S400: Execute the anti-breakthrough mechanism: Set the limit number of back-source requests to the maximum number of requests per second, such as 1000 requests / second. When the back-source requests exceed the limit, perform cache degradation processing, such as returning the default driver location or the historical order price.

[0123] Specifically, step S401: Manage the synchronization thread pool: Configure the number of threads and the queue size of the thread pool, such as 50 threads and a queue length of 100. Use the thread pool to manage the execution of synchronization operations to ensure that the number of threads concurrently executing synchronized data is within a controllable range and avoid system instability.

[0124] In this embodiment, regarding step S5: Exception handling and recovery:

[0125] Specifically, step S500: Capture exceptions: During the data synchronization process, use a try-catch block to capture exceptions that occur, such as database connection failures and API call timeouts. Record the exception information, including the exception type, occurrence time, and error location, for subsequent analysis.

[0126] Specifically, step S501: Handle exceptions: Determine the handling method based on the exception type. For example, if the database connection fails, perform a retry; if the API call times out, perform a fallback and return the default value. If a retry is required, set the number of retries to 3 times with an interval of 5 seconds. If the exception cannot be handled, record the log and notify the user or the upper-layer application to facilitate timely troubleshooting.

[0127] Specifically, step S502: Recovery operation: After the exception handling is completed, check whether the system status has returned to normal. If not, re-execute the synchronization operation or start the backup mechanism, such as reading data from the backup database.

[0128] In this embodiment, regarding step S6: Synchronization result feedback: Feedback the result of the data synchronization to the user or the upper-layer application.

[0129] If the synchronization is successful, provide the synchronized data version or timestamp, such as "Synchronization successful, data version v2.0, timestamp 2023-04-01 12:00:00". If the synchronization fails, generate an error message, including an error code (such as E001) and an error description (such as "Database connection failed").

[0130] At the same time, provide a retry mechanism that allows the user or the upper-layer application to click the "Resynchronize" button to re-trigger the synchronization operation. Record the failure log, including the exception type, occurrence time, error location, and handling method, for subsequent system optimization.

[0131] It can be understood that by loading the user-defined configuration file and parsing the configuration parameters, the system can be flexibly customized according to the specific needs of the user, such as the synchronization strategy, period, and method of data caching. This meets the personalized needs of different online car-hailing companies and improves the flexibility and applicability of the system.

[0132] It can be understood that the system supports multiple data sources (such as databases, APIs) and cache systems (such as Redis, Memcached). By specifying the data source type and cache policy in the configuration file, the system can establish connections with different data sources and use different cache systems for data storage and access. This expands the scope of application of the system and supports the needs of multiple business scenarios of online car-hailing companies.

[0133] It can be understood that by loading the anti-breakthrough mechanism and managing the synchronization thread pool, the system can effectively limit the number of origin requests and control the number of threads for concurrently executing synchronized data, thereby avoiding system failures in high-concurrency or unstable network situations. At the same time, the exception handling and recovery mechanism can capture and handle exceptions that occur during the synchronization process, ensuring the stability and reliability of the system.

[0134] It can be understood that through customized configuration and data caching mechanisms, the system can respond to user requests faster, improve data access speed, and thus enhance the user experience. Enhance the flexibility and scalability of the system: Support multiple data sources and caching systems, enabling the system to more easily adapt to different business scenarios and changing requirements. Through the anti-breakthrough mechanism, concurrency control, and exception handling mechanism, the system can operate stably in complex network environments and operating conditions, reducing the likelihood of failures.

[0135] Embodiment 2: On the basis of Embodiment 1, this embodiment further provides a Python execution program for the solution described in Embodiment 1:

[0136]

[0137]

[0138]

[0139]

[0140]

[0141]

[0142]

[0143]

[0144] In the above program, the program first loads a user-defined configuration file, which contains parameters such as the synchronization strategy, period, and method of data caching. Initialize the data caching module and reset the local cache. Initialize the synchronization control module and save the synchronization strategy parameters in the configuration. Initialize the configuration management module and save the configuration parameters as global variables for subsequent use. According to the data source type specified in the configuration file, select the corresponding connection method. In this example, only the connection information is printed, and a real connection should be established in actual applications.

[0145] All of the above embodiments merely represent the implementation modes of the relevant practical applications of the present invention. The descriptions thereof are relatively specific and detailed, but should not be construed as limiting the scope of the invention patent. It should be noted that for those of ordinary skill in the art, without departing from the concept of the present invention, several modifications and improvements can be made, and these all fall within the protection scope of the present invention. Therefore, the protection scope of the present invention patent shall be subject to the appended claims.

[0146] For those skilled in the art, it can be further realized that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be implemented by electronic hardware, computer software, or a combination of the two. To clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described according to functions in the above description. Whether these functions are executed in a hardware or software manner depends on the specific application and design constraints of the technical solution. Professional technicians can use different methods for each specific application to implement the described functions, but such implementation should not be considered as exceeding the scope of the present invention.

[0147] At the same time, those skilled in the art can understand that all or part of the processes of implementing the methods of all the above embodiments can be completed by instructing relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above methods. Among them, any reference to a memory, storage, database, or other medium provided in the present application and used in the embodiments can include non-volatile and / or volatile memories. Non-volatile memories can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memories can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (SSRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

Claims

1. A data caching and synchronization method with flexible configuration based on the SaaS platform, characterized in that, After receiving the activation instruction for the input data cache synchronization solution, the following operation steps are started: S1. Load the user-defined configuration file; S2. Establish a connection with the data source according to the data source information specified in the configuration file; S3. Execute real-time synchronization operation, scheduled synchronization operation or incremental synchronization operation according to the synchronization policy; S4. The anti-breakthrough mechanism restricts the number of origin requests to avoid cache breakdown; S5. During the data synchronization process, capture and handle the abnormal situations that occur.

2. The data cache synchronization method according to claim 1, wherein: In the above S1, loading the user-defined configuration file includes the synchronization policy, period and method of data cache.

3. The data cache synchronization method according to claim 1, wherein: The execution process of S1 includes: S100. Read the path of the user-defined configuration file; use the configuration loader to load the content of the configuration file; S101. Parse the configuration parameters, including: Parse the synchronization policy of data cache, including real-time, scheduled and incremental; Parse the synchronization period, including daily or hourly; Parse the synchronization method, including full amount or increment.

4. The data cache synchronization method according to claim 1, wherein: In the above S2, it also includes initializing the local cache, setting the timeout time and concurrency size parameters of the cache according to the configuration; and preloading the cache data.

5. The data cache synchronization method according to claim 4, characterized in that: The implementation method of S2 includes: S200. Select the corresponding connection method according to the data source type specified in the configuration file; configure the connection parameters, including host name, port number, user name and password; S201. Set the timeout time of the cache according to the configuration; set the concurrency size parameters of the cache, including the maximum connection number and thread pool size; initialize the cache storage structure, including key-value pairs and lists; S202. Obtain the data to be cached from the data source; load the data into the local cache; set the initial state of the cache data to valid or invalid.

6. The data cache synchronization method according to claim 1, wherein: In the above S4, through the synchronization thread pool management, control the number of threads for concurrently executing synchronized data.

7. The data cache synchronization method according to claim 6, wherein: The implementation method of S4 includes: S400. Set the limit number of origin requests to the maximum number of requests per second; when the origin requests exceed the limit, perform cache degradation processing or return the default value; S401. Configure the number of threads and queue size of the thread pool; use the thread pool to manage the execution of synchronization operations to avoid system instability caused by excessive concurrency.

8. The data cache synchronization method according to claim 1, characterized in that: It also includes S6. Synchronization result feedback: Feed back the result of data synchronization to the user or the upper-layer application. If the synchronization fails, provide error information and a retry mechanism.

9. A system for implementing the data cache synchronization method according to any one of claims 1 to 8, characterized in that The system includes: A configuration management module responsible for loading the user-defined configuration file and parsing the synchronization policy, period and method parameters of data cache; A data cache module, a synchronization control module and a log monitoring module that are initialized according to the parameters parsed by the configuration management module and set their respective working modes and parameters.

10. The system according to claim 9, characterized in that: The data cache module implements the anti-breakthrough mechanism to restrict the number of origin requests; The synchronization control module controls the number of threads for concurrently executing synchronized data through the synchronization thread pool management; The synchronization control module is responsible for capturing and handling the abnormal situations that occur during the data synchronization process; The log monitoring module records the process and result of exception handling; The synchronization control module feeds back the result of data synchronization to the user or the upper-layer application through the user interface module.

Images