Distributed system dictionary data access and synchronization management system and method

Abstract

The application discloses a distributed system dictionary data access and synchronization management system and method, and relates to the technical field of system management. The distributed system dictionary data access and synchronization management system comprises a business system and a dictionary management system. A dictionary software development kit is integrated in the business system, the dictionary software development kit is coupled with the dictionary management system, the dictionary software development kit is synchronized with the dictionary management system, and the business system accesses dictionary data in the dictionary management system through the dictionary software development kit. The application solves the problems of high dictionary data access delay, cross-service data inconsistency and difficult unified management of cache.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and more specifically, to a distributed system dictionary data access and synchronization management system and method. Background Technology

[0002] With the development of information technology, distributed systems have become the mainstream architecture for enterprise applications and internet services. In distributed systems, business logic is typically broken down into multiple independent services, each of which may have independent data storage, configuration management, and access interfaces. In practical applications, dictionary data, as basic configuration data and business logic mapping data, is widely used in scenarios such as data transformation, state management, type mapping, access control, and business rule definition.

[0003] With the development of distributed system architecture, dictionary data management technology has gradually evolved in the following directions: 1. Centralized dictionary service: Provides a dictionary data query interface through a unified distributed service. The distributed service uses dictionary data to obtain dictionary data by calling the dictionary service via RPC or HTTP.

[0004] 2. Combining local caching with long polling synchronization mechanism: Distributed services that need to use dictionary data cache dictionary data locally and obtain the latest data changes through an active long polling mechanism. The advantage of this approach is high access performance, but it needs to handle cache consistency and synchronization latency issues.

[0005] Therefore, dictionary data management in traditional architectures typically suffers from the following problems: 1. Under high concurrency access, the query performance of centralized dictionary services is insufficient, which can easily become a performance bottleneck in large-scale distributed systems and poses a single point of failure risk.

[0006] 2. While the combination of local caching and long polling synchronization mechanisms offers high query performance, under unstable network conditions, dictionary data updates and synchronization mechanisms lag, making it difficult to guarantee distributed consistency.

[0007] 3. The centralized dictionary service and local caching and long polling synchronization mechanism do not provide a unified toolkit for dictionary data users, resulting in high integration costs for users.

[0008] Therefore, how to improve the problems of high latency in dictionary data access, inconsistency of cross-service data, and difficulty in unified management of cache in distributed systems has become an urgent problem to be solved. Summary of the Invention

[0009] In view of this, the present invention provides a distributed system dictionary data access and synchronization management system and method based on SDK, in order to improve the problems of high latency in dictionary data access, cross-service data inconsistency, and difficulty in unified management of cache in distributed systems.

[0010] In a first aspect, the present invention provides a distributed system dictionary data access and synchronization management system, including a business system and a dictionary management system, wherein a dictionary software development kit is integrated into the business system, the dictionary software development kit is coupled to the dictionary management system, the dictionary software development kit and the dictionary management system synchronize dictionary data, and the business system accesses the dictionary data in the dictionary management system through the dictionary software development kit.

[0011] Optionally, the dictionary software development kit includes a dictionary application programming interface, memory cache, disk cache, data statistics module, and data synchronization module, wherein, The dictionary application programming interface is used to provide the business system with a call interface for accessing dictionary data; The memory cache is used to query dictionary data before the disk cache; The disk cache is used to query dictionary data when dictionary data cannot be found in the memory cache. The data statistics module is used to record the frequency of calls and access times of statistical dictionary data, and report them to the dictionary management system. The data synchronization module is used to keep the data in the memory cache and the disk cache consistent with the data in the dictionary management system.

[0012] Optionally, the data synchronization module includes a timer synchronizer, an event listener, and a dictionary management system application programming interface, wherein, The timer synchronizer is used to periodically call the dictionary management system application interface to query the full dictionary data, put it into the disk cache, and clear the memory cache; The event listener is used to listen for dictionary data change notifications issued by the dictionary management system. After the event listener detects a dictionary data change notification, it calls the application interface of the dictionary management system to query the newly added dictionary data, writes the newly added dictionary data to the disk cache, and clears the memory cache. The dictionary management system application interface is used to provide an application interface for the dictionary management system, and to call up all the dictionary data in the dictionary management system.

[0013] Optionally, a distributed database may also be included, coupled to the dictionary management system and the dictionary software development tool respectively, for storing dictionary data. Dictionary key values ​​are used as sharding keys for storage, and data versions with the same dictionary key value are placed in the same data partition.

[0014] Secondly, the present invention provides a method for accessing and synchronizing dictionary data in a distributed system, comprising: The dictionary software development kit synchronizes the full dictionary data with the dictionary management system; The business system initiates a request to query dictionary data; The dictionary software development kit responds to the dictionary data query request by querying the full dictionary data in the dictionary software development kit and returns the queried data to the business system.

[0015] Optionally, the dictionary software development kit includes a dictionary application programming interface (API), a memory cache, a disk cache, a data statistics module, and a data synchronization module. The data synchronization module includes a timed synchronizer and a dictionary management system API. The synchronization of the entire dictionary data between the dictionary software development kit and the dictionary management system includes: The dictionary software development kit periodically triggers a task to synchronize all dictionary data. The dictionary management system application interface is used to query the full dictionary data from the dictionary management system. The dictionary management system returns the full dictionary data to the dictionary software development kit, overwrites the disk cache, and clears the memory cache.

[0016] Optionally, the business system initiates a request to query the data corresponding to the dictionary key-value pairs; In response to the request to query dictionary key-value pairs, the dictionary application interface first checks whether the memory cache contains such data. If the memory cache does contain the data, it sends the data to the business system. If the memory cache does not contain the data, it queries the disk cache, caches the data in the disk cache into the memory cache, and then sends the cached dictionary key-value pairs to the business system.

[0017] Optionally, the distributed system dictionary data access and synchronization management system includes a distributed database, the data synchronization module further includes an event listener, and the method further includes: The dictionary management system receives and saves newly added dictionary data; The dictionary management system publishes dictionary data change messages to the distributed database; The event listener listens for dictionary data change messages; The dictionary management system application interface is invoked to query the dictionary management system for details of newly added dictionary data. The dictionary management system then sends the newly added dictionary data to the dictionary software development kit, updates the disk cache, and clears the memory cache.

[0018] Optionally, if the memory cache has data corresponding to the dictionary key-value pair, then the data corresponding to the dictionary key-value pair is sent to the business system, and a cache hit is recorded and sent to the data statistics module. If the memory cache does not have data corresponding to the dictionary key-value pair, then query the data in the disk cache and simultaneously record the cache miss and send it to the data statistics module; After the data statistics module reports the statistics of cache hits and cache misses to the dictionary management system, it clears the local statistics.

[0019] Thirdly, the present invention also provides a computer-readable storage medium storing a computer-executable program that, when executed by a processor, implements the above-described distributed system dictionary data access and synchronization management method.

[0020] Compared with existing technologies, the distributed system dictionary data access and synchronization management system and method provided by this invention achieves at least the following beneficial effects: This invention integrates a dictionary software development kit into the business system, allowing all business services to access dictionary data uniformly through the dictionary software development kit. This solves the problems of high latency in dictionary data access, inconsistency of data across services, and difficulty in unified management of cache, and also reduces the costs of redundant development and maintenance.

[0021] In this invention, the dictionary software development kit and dictionary data storage are separated, allowing the system to be flexibly expanded and supporting multi-node deployment and high-concurrency access.

[0022] Of course, any product implementing this invention does not necessarily need to achieve all of the technical effects described above at the same time.

[0023] Other features and advantages of the invention will become clear from the following detailed description of exemplary embodiments of the invention with reference to the accompanying drawings. Attached Figure Description

[0024] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments of the invention and, together with their description, serve to explain the principles of the invention.

[0025] Figure 1 This invention provides a structural diagram of a distributed system dictionary data access and synchronization management system. Figure 2 This is another distributed system dictionary data access and synchronization management system structure diagram provided by the present invention; Figure 3 This is a flowchart of a distributed system dictionary data access and synchronization management method provided by the present invention; Figure 4 This is a flowchart of another distributed system dictionary data access and synchronization management method provided by the present invention. Detailed Implementation

[0026] Various exemplary embodiments of the present invention will now be described in detail with reference to the accompanying drawings. It should be noted that, unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps set forth in these embodiments do not limit the scope of the invention.

[0027] The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit the invention or its application or use.

[0028] Techniques, methods, and equipment known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods, and equipment should be considered part of the specification.

[0029] In all the examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not as limitations. Therefore, other examples of exemplary embodiments may have different values.

[0030] It should be noted that similar labels and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be discussed further in subsequent figures.

[0031] Hereinafter, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of embodiments of this disclosure, unless otherwise stated, "a plurality of" means two or more. Furthermore, the use of "based on" or "according to" implies openness and inclusiveness, because processes, steps, calculations, or other actions "based on" or "according to" one or more of the stated conditions or values ​​may in practice be based on additional conditions or beyond the stated values.

[0032] For ease of understanding, the terms or nouns involved in the embodiments of this invention will be introduced first.

[0033] SDK stands for Software Development Kit, a collection of tools, libraries, APIs, documentation, and sample code designed to help developers use a platform or service more easily.

[0034] Dictionary data refers to a collection of data in an information system used to express fixed business semantics, with a limited range of values ​​and stable meanings. This type of data usually exists in the form of "code-name-meaning" and is used to uniformly identify and interpret information such as business status, type, and classification.

[0035] Business system: A collection of information systems built around business processes, business rules, and business data to achieve specific business objectives. This system automates business logic processing through modeling, execution, and management of business activities, and provides stable and reusable business capabilities to external parties.

[0036] Dictionary management system: Through centralized management and unified release of dictionary data, it provides consistent and configurable basic data support for various business systems and is an important infrastructure to ensure the semantic consistency of data between systems.

[0037] RPC stands for Remote Procedure Call, a communication protocol that allows functions / methods on remote servers to be called as if they were local functions, hiding the complexity of the underlying network communication.

[0038] HTTP, or Hyper Text Transfer Protocol, is an application layer protocol used for communication between clients and servers. It uses a request-response mechanism and is the foundational protocol for modern web applications and many service interfaces.

[0039] Distributed Systems: A distributed system is a system composed of a group of interconnected computing nodes (Nodes) with independent computing and storage capabilities. These nodes are physically distributed, but appear to the user as a unified and coordinated system. Its core objectives are to achieve resource sharing, task collaboration, performance scalability, and high availability in a multi-machine environment.

[0040] Pub / Sub (Publish / Subscribe) is a messaging pattern where the publisher does not send messages directly to specific recipients, but instead publishes them to a "topic" or "channel." The consumer (Subscriber) receives messages published under that topic by subscribing to that topic.

[0041] Redis (Remote Dictionary Server) is an open-source, high-performance key-value database that supports in-memory storage and persistence. It can be used as a caching system, a message broker, or a lightweight database. It is a non-relational database that stores unstructured data, i.e., key-value pairs, and the relationships between the data can only be processed through application logic.

[0042] Combination Figure 1 The present invention provides a distributed system dictionary data access and synchronization management system 100, including a business system 101 and a dictionary management system 102. Dictionary software development kit 1 is integrated into the business system 101. Dictionary software development kit 1 is coupled to dictionary management system 102. Dictionary software development kit 1 and dictionary management system 102 synchronize dictionary data. Business system 101 accesses dictionary data in dictionary management system 102 through dictionary software development kit 1.

[0043] Specifically, the business system 101 in this invention is a collection of information systems built around business processes, business rules and business data to achieve specific business objectives. It includes ERP (Enterprise Resource Planning), CRM (Customer Relationship Management), SCM (Supply Chain Management), financial system, HR system, etc., covering the entire process of procurement, production, sales, finance, human resources, etc. For example, it is an order management system for e-commerce platforms, a manufacturing execution system for manufacturing industries, etc.

[0044] Dictionary data serves as the "foundation dictionary" for Business System 101. Through standardized and centralized management, it ensures consistent understanding of the same concepts across all modules of the system. For example, the product category dictionary allows users to browse products along the path of "electronic products → mobile phones → smartphones." Another example is the risk level dictionary, which categorizes customer risk into three levels: "low," "medium," and "high," driving different approval processes.

[0045] The dictionary management system 102 is responsible for managing and maintaining the dictionary data, while the business system 101 is responsible for using the dictionary data.

[0046] Existing distributed systems suffer from problems such as high access latency, cross-service data inconsistency, and difficulty in unified cache management. This invention addresses these issues by integrating a dictionary software development kit (SDK) into the business system 101. The Dictionary SDK is synchronized with the dictionary management system 102. When querying dictionary data, the business system 101 uses the Dictionary SDK to retrieve the data. Thus, all business services access dictionary data uniformly through the Dictionary SDK, eliminating the need for RPC or HTTP calls to dictionary services and the need for a distributed service to cache dictionary data locally.

[0047] In some embodiments, reference is made to Figure 2 The dictionary software development kit 1 includes a dictionary application programming interface 11, a memory cache 12, a disk cache 13, a data statistics module 15, and a data synchronization module 14, among which... Dictionary application programming interface 11 is used to provide a call interface for business system 101 to access dictionary data; Memory cache 12 is used to query dictionary data over disk cache 13; Disk cache 13 is used to query dictionary data when it cannot be found in memory cache 12; The data statistics module 15 is used to record the frequency of calls and access time of statistical dictionary data, and report them to the dictionary management system 102. The data synchronization module 14 is used to keep the data in the memory cache 12 and disk cache 13 consistent with the data in the dictionary management system 102.

[0048] Specifically, the dictionary software development kit 1 (dictionary SDK) integrated in business system 101 includes a dictionary application programming interface 11, a memory cache 12, a disk cache 13, a data statistics module 15, and a data synchronization module 14. Dictionary Application Programming Interface 11 (i.e., Dictionary API) is the calling interface provided by business system 101 for accessing dictionary data. The Dictionary API realizes the coupling between business system 101 and dictionary SDK. This Dictionary API provides business system 101 with an entry point for accessing dictionary data and supports queries by single key (key value) and batch keys.

[0049] Memory Cache 12: When business system 101 queries dictionary data via the dictionary API, it first checks memory cache 12. If the data is found in memory cache 12, it is returned directly; otherwise, it queries disk cache 13 and places the retrieved data into memory cache 12. The purpose of memory cache 12 is to speed up the query speed of frequently accessed keys by trading space for time.

[0050] Disk cache 13: When business system 101 starts, it will query the full data of the dictionary and cache it on the local disk. It provides a data query entry point for memory cache 12. If the data cannot be found in memory cache 12, it will query from disk cache 13.

[0051] Data statistics module 15: mainly records the frequency of calls and access time of statistical dictionary data, and reports the statistical data to the dictionary management system 102, so as to realize the perception of dictionary usage on the management end.

[0052] Data synchronization module 14: The function of data synchronization module 14 is to ensure that the data in the memory cache 12 and disk cache 13 inside the dictionary SDK and the data maintained by the dictionary management system 102 are eventually consistent.

[0053] In some alternative embodiments, reference continues to be made to... Figure 2 The data synchronization module 14 includes a timed synchronizer 141, an event listener 142, and a dictionary management system application programming interface 143, wherein... The timed synchronizer 141 is used to periodically call the dictionary management system application interface 143 to query the full dictionary data, put it into the disk cache 13, and clear the memory cache 12. Event listener 142 is used to listen for dictionary data change notifications issued by dictionary management system 102. After the event listener 142 hears the dictionary data change notification, it calls the dictionary management system application interface 143 to query the newly added dictionary data, write the newly added dictionary data to disk cache 13, and clear memory cache 12. The dictionary management system application interface 143 is used to provide an application interface for the dictionary management system 102, and to call the full dictionary data in the dictionary management system 102.

[0054] Specifically, the data synchronization module 14 is responsible for ensuring eventual consistency between the data in the dictionary SDK's internal memory cache 12 and disk cache 13 and the data maintained by the dictionary management system 102. It comprises the following three parts: Timed synchronizer 141: When the service starts, it calculates a random number for 10-20 minutes. Using this random number as the time interval, it periodically calls the dictionary management system application interface 143 to query the full dictionary data, puts it into the disk cache 13, and clears the memory cache 12. This provides a fallback for data synchronization, ensuring that even if the dictionary management system 102 notification event listener 142 fails, the dictionary data will still achieve eventual consistency.

[0055] Event Listener 142: Listens for dictionary data change notifications issued by the dictionary management system 102. When the administrator makes changes to the dictionary data in the dictionary management system 102, the dictionary management system 102 will send a change event. After being notified by the event listener 142, it will call the dictionary management system application interface 143 to query the details of the changed fields, write the data to the disk cache 13, and finally clear the memory cache 12 to ensure that the latest data can be queried next time.

[0056] Dictionary Management System Application Programming Interface 143 (Dictionary Management System 102 API): The API interface provided by Dictionary Management System 102 to Dictionary SDK, allowing Dictionary SDK to query dictionary data within the remote Dictionary Management System 102.

[0057] In some optional embodiments, the distributed system dictionary data access and synchronization management system 100 includes a distributed database coupled to the dictionary management system 102 and the dictionary software development tool, respectively, for storing dictionary data. Dictionary key values ​​are used as sharding keys for storage, and data versions with the same dictionary key value are placed in the same data partition.

[0058] The dictionary data of this invention is stored in the distributed database Redis. It uses a high-performance distributed storage scheme and uses the dictionary key as the sharding key to ensure that all versions of the same key are placed in the same data partition, so as to realize fast query of different data versions of the key without having to scan all partitions of the database globally.

[0059] This invention supports version management of dictionary data, with each record having a version number or update time for update and synchronization determination.

[0060] The dictionary data of this invention is stored in Redis, which has the following advantages: Compared to MySQL, which primarily organizes data in tables and stores it on disk, Redis primarily organizes data in key-value pairs and stores the data in a cache, where dictionary keys are of type String. Redis is programmable, allowing interactive commands to execute operations or batch operations via scripts. Redis can be extended using C / C++ / Rust. Redis features persistence, primarily using memory with a backup on disk; data stored in memory is backed up on disk, and in case of server failure or power outage, Redis replicates the data back to memory. Redis supports distributed clustering, allowing deployment on different servers. While a single Redis instance can store a limited amount of data, multiple Redis nodes can be deployed on each server, each storing a portion of the data. Redis offers high availability, with master and slave nodes storing the same data, and the master node serving as a backup.

[0061] Redis's most significant feature is its speed. It stores data in memory, making in-memory access thousands or even tens of thousands of times faster than accessing data on disk. Redis utilizes I / O multiplexing, using a single thread to manage multiple sockets. Redis primarily employs a single-threaded model, avoiding unnecessary overhead from thread contention between multiple threads and the overhead of context switching between threads.

[0062] Based on the same inventive concept, this invention also provides a method for accessing and synchronizing dictionary data in a distributed system, combined with... Figure 3 Simultaneously combined Figure 1 and Figure 2 Methods for accessing and synchronizing dictionary data in distributed systems include: S1, the dictionary software development kit and dictionary management system synchronize the full dictionary data; S2, the business system initiates a request to query dictionary data; S3, the dictionary software development kit responds to the dictionary data query request, queries the full dictionary data in the dictionary software development kit, and returns the queried data to the business system.

[0063] This invention integrates a dictionary software development kit (SDK) into a business system 101, synchronizing dictionary data between the Dictionary Software Development Kit 1 and the Dictionary Management System 102. When querying dictionary data, the business system 101 uses the Dictionary Software Development Kit 1 to query the dictionary data. As a result, all business services access dictionary data uniformly through the Dictionary Software Development Kit 1, eliminating the need to obtain dictionary data by calling dictionary services via RPC or HTTP, and eliminating the need to use a distributed service to cache dictionary data locally.

[0064] In some alternative embodiments, combined with Figure 3 and Figure 4 At the same time, combined Figure 2 The dictionary software development kit 1 includes a dictionary application programming interface 11, a memory cache 12, a disk cache 13, a data statistics module 15, and a data synchronization module 14. The data synchronization module 14 includes a timed synchronizer 141 and a dictionary management system application programming interface 143. Step S1, which synchronizes the dictionary software development kit 1 with the dictionary management system 102, includes the following steps: 1. The dictionary software development kit triggers a task to synchronize all dictionary data on a scheduled basis; 2. Query the full dictionary data from the dictionary management system via the dictionary management system application programming interface; 3. The dictionary management system returns all dictionary data to the dictionary software development kit; 4. Write the entire dictionary data to the disk cache; 5. Clear memory cache.

[0065] Specifically, for the triggering mechanism in step 1, Cron expressions or distributed scheduling frameworks (such as Quartz or Elastic-Job) can be used to trigger tasks on a schedule to avoid task loss due to single-machine failure. Here, the timed synchronizer 141 of the data synchronization module 14 can be used to trigger tasks on a schedule.

[0066] For step 2, the dictionary management system 102 and the dictionary SDK are decoupled through the dictionary management system 102 API. Business services do not need to care about the underlying storage and synchronization mechanism, and only need to use the API in the dictionary SDK to query dictionary data.

[0067] For step 3, if the data volume is extremely large, pagination query can be used (page=1&size=1000). In addition, a timestamp should be included when returning the data. The dictionary SDK needs to verify whether the local cache timestamp is consistent with the returned data to avoid overwriting old data.

[0068] For step 4, the overwrite is written to disk cache 13. The file format can be JSON or Protobuf / MessagePack. JSON is highly readable and suitable for debugging and manual inspection, while Protobuf / MessagePack uses binary format, which has a fast parsing speed and saves disk space.

[0069] Clearing the memory cache 12. Implementing a cache eviction policy can be done by full clearing: directly clearing all dictionary data in memory (suitable for small business systems 101). Alternatively, clearing by type can be done: precisely clearing the corresponding cache based on the returned dictionary type list (dictionaries.type) (suitable for large business systems 101, reducing cache rebuilding overhead).

[0070] If certain dictionaries are accessed frequently, popular data can be asynchronously loaded into memory immediately after being cleared, avoiding delays in the first query.

[0071] This process enables highly reliable, low-latency dictionary data synchronization, while also facilitating operation, maintenance, and troubleshooting.

[0072] The data synchronization module 14 based on the dictionary SDK supports dynamic expansion, version iteration, and canary updates of dictionary data, facilitating the smooth deployment of new versions in distributed systems.

[0073] In some alternative embodiments, combined with Figure 3 and Figure 4 Steps S2 and S3 are as follows: 6. The business system initiates a request to query the corresponding data for dictionary key-value pairs; 7. When responding to a query request for dictionary key-value pairs, the dictionary application interface first checks if the memory cache contains the corresponding dictionary key-value pairs. 9. If the memory cache contains dictionary key-value pairs, then send the dictionary key-value pairs to the business system. 11. If the memory cache does not have corresponding dictionary key-value data, then query the data in the disk cache; 12. Cache data from the disk cache into the memory cache; 13. Send the cached dictionary key-value pairs to the business system.

[0074] In this invention, high-frequency access data is preferentially stored in a high-speed cache (memory), while low-frequency or cold data is stored in low-speed storage (disk).

[0075] Version control: For example, ClickHouse uses a Revision mechanism to update the cache. Each time the dictionary data changes, a new version is generated, and the system can seamlessly switch to the latest version, avoiding service interruptions caused by cache invalidation.

[0076] Data loading and write-back in progress: Disk to Memory: When the first query or cache miss occurs, data is loaded from the disk into the memory cache (such as Redis, Ehcache), and subsequent queries are read directly from memory.

[0077] Memory to Disk: During full synchronization, a "temporary file + renaming" strategy is adopted (e.g., dict_cache.tmp → dict_cache.json) to ensure the atomicity of the write process and avoid data corruption.

[0078] In this embodiment of the invention, the dictionary query process adopts a multi-level caching architecture. Through the collaborative mechanism of memory cache 12 and disk cache 13, the query efficiency is significantly improved while ensuring data consistency.

[0079] In some alternative embodiments, reference continues to be made to... Figure 4 and combination Figure 1 and Figure 2 The distributed system dictionary data access and synchronization management system 100 includes a distributed database, a data synchronization module 14, and an event listener 142. The method also includes an S4 incremental dictionary data synchronization process, specifically including the following steps: 14. The dictionary management system receives and saves newly added dictionary data; 15. The dictionary management system publishes dictionary data change messages to the distributed database; 16. Event listeners monitor dictionary data change messages (keys and events); 17. Call the dictionary management system application programming interface to query the dictionary management system for details of newly added dictionary data; 18. The dictionary management system sends newly added dictionary data to the dictionary software development kit 1; 19. Update disk cache; 20. Clear memory cache.

[0080] This invention not only enables full dictionary data synchronization but also supports incremental dictionary data synchronization. When dictionary data is updated, the dictionary SDK can achieve incremental synchronization through publish / subscribe, and a scheduled task in the dictionary SDK can provide full data synchronization as a fallback, ensuring data consistency and low latency across all service nodes.

[0081] In some alternative embodiments, combined with Figure 4 The distributed system dictionary data access and synchronization management method also includes the S5 data statistics reporting process, which specifically includes the following steps: 8. If the memory cache contains dictionary key-value pairs, send the corresponding dictionary key-value pairs to the business system, and simultaneously record the cache hit and send it to the data statistics module. 10. If the memory cache does not have the corresponding dictionary key-value data, then query the data in the disk cache and record the cache miss to send to the data statistics module. 21. The data statistics module reports the statistics of cache hits and cache misses to the dictionary management system; 22. Clear local statistical data.

[0082] The data statistics module 15 in the dictionary SDK of this invention can record the number of times each service calls the dictionary data and the access time, and report the data to the dictionary management system 102 through the data statistics module 15 for analysis of dictionary data usage. The system can optimize caching strategies and update frequency based on the access statistics results, thereby improving system performance and resource utilization.

[0083] Based on the above embodiments, the present invention has the following beneficial effects: Unified access: All business services can access dictionary data uniformly through the SDK, reducing the cost of redundant development and maintenance.

[0084] High-efficiency synchronization: Version control and incremental update mechanisms are used to ensure data consistency among nodes in the distributed system and reduce latency.

[0085] Low storage footprint: Long strings are mapped to IDs for storage, reducing database storage space and improving query performance.

[0086] Excellent scalability: The SDK layer is separated from the dictionary storage, allowing the system to be flexibly expanded and supporting multi-node deployment and high-concurrency access.

[0087] Traceable statistics: Records call volume and access time, facilitating operation and maintenance optimization and business analysis.

[0088] Thirdly, the present invention also provides a computer-readable storage medium storing a computer-executable program that, when executed by a processor, implements the steps of the above-described SDK-based distributed system dictionary data access and synchronization management method.

[0089] The present invention also provides a computer-readable storage medium storing a computer-executable program that, when the computer program runs on an SDK-based distributed system dictionary data access and synchronization management system, causes the SDK-based distributed system dictionary data access and synchronization management system to execute the various steps of the method flow shown in the above method embodiments.

[0090] While specific embodiments of the invention have been described in detail by way of examples, those skilled in the art should understand that the examples are for illustrative purposes only and not intended to limit the scope of the invention. Those skilled in the art should understand that modifications can be made to the above embodiments without departing from the scope and spirit of the invention. The scope of the invention is defined by the appended claims.

Claims

1. A distributed system dictionary data access and synchronization management system, characterized in that, The system includes a business system and a dictionary management system. A dictionary software development kit is integrated into the business system. The dictionary software development kit is coupled to the dictionary management system and synchronizes dictionary data with the dictionary management system. The business system accesses the dictionary data in the dictionary management system through the dictionary software development kit.

2. The distributed system dictionary data access and synchronization management system according to claim 1, characterized in that, The dictionary software development kit includes a dictionary application programming interface, memory cache, disk cache, data statistics module, and data synchronization module. The dictionary application programming interface is used to provide the business system with a call interface for accessing dictionary data; The memory cache is used to query dictionary data before the disk cache; The disk cache is used to query dictionary data when dictionary data cannot be found in the memory cache. The data statistics module is used to record the frequency of calls and access times of statistical dictionary data, and report them to the dictionary management system. The data synchronization module is used to keep the data in the memory cache and the disk cache consistent with the data in the dictionary management system.

3. The distributed system dictionary data access and synchronization management system according to claim 2, characterized in that, The data synchronization module includes a timer synchronizer, an event listener, and a dictionary management system application interface, wherein... The timer synchronizer is used to periodically call the dictionary management system application interface to query the full dictionary data, put it into the disk cache, and clear the memory cache; The event listener is used to listen for dictionary data change notifications issued by the dictionary management system. After the event listener detects a dictionary data change notification, it calls the application interface of the dictionary management system to query the newly added dictionary data, writes the newly added dictionary data to the disk cache, and clears the memory cache. The dictionary management system application interface is used to provide an application interface for the dictionary management system, and to call up all the dictionary data in the dictionary management system.

4. The distributed system dictionary data access and synchronization management system according to claim 1, characterized in that, It also includes a distributed database, which is coupled to the dictionary management system and the dictionary software development tool, respectively, for storing dictionary data. The dictionary key value is used as the sharding key for storage, and data versions with the same dictionary key value are placed in the same data partition.

5. A method for dictionary data access and synchronization management in a distributed system, characterized in that, include: The dictionary software development kit synchronizes the full dictionary data with the dictionary management system; The business system initiates a request to query dictionary data; The dictionary software development kit responds to the dictionary data query request by querying the full dictionary data in the dictionary software development kit and returns the queried data to the business system.

6. The distributed system dictionary data access and synchronization management method according to claim 5, characterized in that, The dictionary software development kit includes a dictionary application programming interface (API), memory cache, disk cache, data statistics module, and data synchronization module. The data synchronization module includes a timed synchronizer and a dictionary management system API. The synchronization of the dictionary software development kit with the dictionary management system for all dictionary data includes: The dictionary software development kit periodically triggers a task to synchronize all dictionary data. The dictionary management system application interface is used to query the full dictionary data from the dictionary management system. The dictionary management system returns the full dictionary data to the dictionary software development kit, overwrites the disk cache, and clears the memory cache.

7. The distributed system dictionary data access and synchronization management method according to claim 6, characterized in that, The business system initiates a request to query the data corresponding to dictionary key-value pairs; In response to the request to query dictionary key-value pairs, the dictionary application interface first checks whether the memory cache contains such data. If the memory cache does contain the data, it sends the data to the business system. If the memory cache does not contain the data, it queries the disk cache, caches the data in the disk cache into the memory cache, and then sends the cached dictionary key-value pairs to the business system.

8. The distributed system dictionary data access and synchronization management method according to claim 7, characterized in that, The distributed system dictionary data access and synchronization management system includes a distributed database, the data synchronization module further includes an event listener, and the method further includes: The dictionary management system receives and saves newly added dictionary data; The dictionary management system publishes dictionary data change messages to the distributed database; The event listener listens for dictionary data change messages; The dictionary management system application interface is invoked to query the dictionary management system for details of newly added dictionary data. The dictionary management system then sends the newly added dictionary data to the dictionary software development kit, updates the disk cache, and clears the memory cache.

9. The distributed system dictionary data access and synchronization management method according to claim 7, characterized in that, If the memory cache contains data corresponding to the dictionary key-value pair, then the data corresponding to the dictionary key-value pair is sent to the business system, and a cache hit is recorded and sent to the data statistics module. If the memory cache does not have data corresponding to the dictionary key-value pair, then query the data in the disk cache and simultaneously record the cache miss and send it to the data statistics module; After the data statistics module reports the statistics of cache hits and cache misses to the dictionary management system, it clears the local statistics.

10. A computer-readable storage medium, characterized in that, It stores a computer-executable program that, when executed by a processor, implements the distributed system dictionary data access and synchronization management method as described in any one of claims 5 to 9.

Images