Data distributed management system and method based on multi-layer database

Abstract

This invention provides a distributed data management system and method based on a multi-layer database, relating to the field of computer technology. The system includes a central service node and at least one client node communicatively connected to the central service node. The central service node deploys a shared parameter cache database and at least one server-side application, while the client nodes deploy at least one client-side application. The shared parameter cache database is configured to: upon receiving a write request for a first parameter key from the server-side application and / or the client application, update the parameter value corresponding to the first parameter key stored in the shared parameter cache database; after updating the parameter value, push the updated parameter value to all subscribers of the first parameter key. This invention achieves distributed data management across machines, improves data consistency and real-time performance throughout the distributed system, and reduces system load.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to a distributed data management system and method based on a multi-layer database. Background Technology

[0002] In display and control software systems, the system typically needs to receive, process, and display a large number of parameters in real time. These parameters may originate from multiple data streams, such as sensor data, control commands, external communication interfaces, or even remote data sharing across processes or machines. Therefore, an efficient, stable, and scalable in-memory database is crucial for display and control software.

[0003] Currently, data sharing typically relies on shared memory or traditional databases. While shared memory is an efficient inter-process communication method, its application is limited to cross-process communication within the same computer. When the display and control system is distributed across different computer nodes, shared memory cannot achieve cross-machine data sharing, severely restricting its application scenarios. Traditional databases (such as relational databases), although supporting cross-machine access over a network, usually rely on active polling by the client, which suffers from poor real-time performance and high system overhead. Summary of the Invention

[0004] This invention provides a data distributed management system and method based on a multi-layer database, which solves the technical problems in the prior art where shared memory only supports cross-process but not cross-machine, and has poor data synchronization real-time performance and high system overhead. It realizes a data distributed management system that can support cross-machine, has high real-time performance and low system overhead.

[0005] This invention provides a distributed data management system based on a multi-layer database, comprising: A central service node, and at least one client node that is communicatively connected to the central service node; The central service node is equipped with a shared parameter cache database and at least one server application, while the client node is equipped with at least one client application. The shared parameter cache database is configured as follows: When a write request for the first parameter key is received from the server application and / or the client application, the parameter value corresponding to the first parameter key stored in the shared parameter cache database is updated. After updating the parameter value, push the updated parameter value to all subscribers of the first parameter key.

[0006] According to the data distributed management system based on a multi-layer database of the present invention, the server application and / or the client application integrate a parameter cache database and a historical parameter cache database. The parameter cache database is used to cache the latest parameter value corresponding to each parameter key; and, The historical parameter cache database is used to cache the historical time series parameter values ​​corresponding to each parameter key.

[0007] According to the data distributed management system based on a multi-layer database of the present invention, the parameter cache database is configured as follows: The local storage space is divided into multiple shards, and parameter keys are used to index and access the corresponding parameter values; wherein, the parameter key includes at least one shard mapping dimension, which is used to map the parameter key to the corresponding shard.

[0008] According to the data distributed management system based on a multi-layer database of the present invention, the server application and / or the client application are configured to perform a single locking operation, the single locking operation including the following steps: The first target slice is determined based on the slice mapping dimension of the single parameter key to be operated on; In the case of a write operation, an exclusive lock is applied to the first target fragment, and the corresponding parameter values ​​are written to the first target fragment; Alternatively, in the case of a read operation, a shared lock is applied to the first target fragment, and the corresponding parameter value is read from the first target fragment.

[0009] According to the data distributed management system based on a multi-layer database of the present invention, the server application and / or the client application are configured to perform batch locking operations, the batch locking operations including the following steps: Based on the partitioning mapping dimension of the multiple second parameter keys to be operated on, the multiple second parameter keys are grouped into corresponding second target partitions; Generate a locking order corresponding to multiple second target fragments; Locks are applied sequentially to all the second target segments according to the locking order; While all the second target fragments are locked, perform batch data processing on multiple second parameter keys.

[0010] According to the data distributed management system based on a multi-layer database of the present invention, the server application and / or the client application are configured to perform batch lock-free operations, the batch lock-free operations including the following steps: When the external caller applies a lock to the third target slice corresponding to multiple third parameter keys, the multiple third parameter keys are traversed, and the parameter values ​​of the corresponding third parameter keys are stored and retrieved in the multiple third target slices.

[0011] According to the data distributed management system based on a multi-layer database of the present invention, the historical parameter cache database includes a first circular buffer, a second circular buffer, and a write head pointer corresponding to each parameter key: The first circular buffer is used to store parameter values ​​written in chronological order; The second circular buffer is used to store the Boolean value corresponding to each parameter value; The write head pointer is used to indicate the storage location of the next parameter value to be written in the first and second circular buffers.

[0012] According to the data distributed management system based on a multi-layer database of the present invention, the server application and / or the client application are configured to perform a first read operation, the first read operation including the following steps: Determine the parameter value corresponding to the first read operation; If the parameter value is the latest parameter value, read the current value of the write head pointer, determine the latest parameter value sequence number based on the current value, and determine the first storage location of the current value in the first circular buffer and the second circular buffer; read the parameter value and Boolean value from the first storage location; or, If the parameter value is a historical parameter value, determine the target historical parameter value sequence number specified by the first read operation, and determine the second storage location of the historical parameter value corresponding to the target historical parameter value sequence number in the first and second circular buffers according to the write head pointer and the capacity of the first circular buffer; read the parameter value and Boolean value from the second storage location.

[0013] According to the data distributed management system based on a multi-layer database of the present invention, the server application and / or the client application are configured to perform a second read operation, the second read operation including the following steps: Determine the starting sequence number and the number of reads corresponding to the second read operation; Based on the write head pointer and the capacity of the first circular buffer, determine whether the starting sequence number is within the valid sequence number window; If the starting sequence number is within the valid sequence number window, the parameter value of the number of reads and the corresponding Boolean value are read continuously starting from the storage location corresponding to the starting sequence number.

[0014] According to the data distributed management system based on a multi-layer database of the present invention, the server application and / or the client application are configured as follows: Register the conditional function corresponding to the fourth parameter key; When writing the latest parameter value corresponding to the fourth parameter key into the first circular buffer, the judgment function is called, and the boolean value of the judgment function is written into the corresponding storage location in the second circular buffer.

[0015] This invention also provides a data distributed management method based on a multi-layer database, applied to a data distributed management system based on a multi-layer database. The system includes a central service node and at least one client node communicatively connected to the central service node. The central service node deploys a shared parameter cache database and at least one server-side application, and the client node deploys at least one client application. The method includes: When a write request for the first parameter key is received from the server application and / or the client application, the parameter value corresponding to the first parameter key stored in the shared parameter cache database is updated. After updating the parameter value, push the updated parameter value to all subscribers of the first parameter key.

[0016] The data distributed management method based on a multi-layer database according to the present invention further includes: Based on the parameter caching database integrated within the server application and / or the client application, the latest parameter value corresponding to each parameter key is cached; and, Based on the historical parameter cache database integrated within the server application and / or the client application, the historical time series parameter value corresponding to each parameter key is cached.

[0017] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the data distributed management method based on a multi-layer database as described above.

[0018] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the data distributed management method based on a multi-layer database as described above.

[0019] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the data distributed management method based on a multi-layer database as described above.

[0020] The data distributed management system based on a multi-layered database provided by this invention achieves cross-machine distributed data management by constructing a distributed system architecture including a central service node and multiple client nodes. Furthermore, the central service node deploys a shared parameter cache database and at least one server-side application, while the client nodes deploy at least one client-side application. Based on the shared parameter cache database, updates are proactively pushed to all subscribers after parameter updates in the server-side application and / or client-side application, improving data consistency and real-time performance throughout the distributed system, avoiding resource waste caused by invalid polling, and reducing system load. Attached Figure Description

[0021] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0022] Figure 1 This is one of the structural diagrams of a multi-layer database-based distributed data management system provided in the embodiments of the present invention.

[0023] Figure 2 This is the second schematic diagram of the structure of the distributed data management system based on a multi-layer database provided in the embodiments of the present invention.

[0024] Figure 3 This is one of the flowcharts of a distributed data management method based on a multi-layer database provided in an embodiment of the present invention.

[0025] Figure 4 This is the second flowchart of the data distributed management method based on a multi-layer database provided in this embodiment of the invention.

[0026] Figure 5 This is a schematic diagram of the structure of the electronic device provided in an embodiment of the present invention. Detailed Implementation

[0027] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0028] Currently, in display and control software systems (such as industrial control, simulation, or monitoring systems), there are typically multiple independent application or service modules that may be deployed on different nodes but need to share and access a common set of status parameters or configuration data. Ensuring that these applications distributed across different nodes can efficiently and promptly obtain consistent parameter information is a key technical challenge.

[0029] Currently, the following methods are commonly used to achieve data sharing in a distributed environment: Data sharing is based on traditional databases: for example, using a relational database such as PostgreSQL as the central data storage, with each node accessing the database via a network to achieve data sharing. However, because its access operations usually involve disk I / O, the latency is high (usually in the millisecond range), making it difficult to meet the fast data response requirements of many display and control systems; Synchronization based on a polling mechanism: Nodes obtain the latest parameter values ​​by periodically querying the central server (i.e., "polling"). However, this method is not timely; after a parameter is updated, it needs to wait for the next polling cycle to detect the change, resulting in data delays and temporary inconsistencies. Secondly, it has high system overhead; even if the data has not changed, a large number of polling requests will continuously consume the resources of the central server and the network, and the problem becomes particularly prominent when the number of nodes increases. Based on local caching: Using shared memory or local hash tables, extremely fast data sharing can be achieved within a single node. However, the applicability of this method is very limited. It usually only supports cross-process communication within the same physical machine and cannot directly support distributed data sharing across machines. It is not suitable for distributed system architectures composed of multiple nodes.

[0030] Based on this, embodiments of the present invention propose a data distributed management system based on a multi-layer database, which meets the requirements of data sharing, low-latency access, and real-time data synchronization among distributed nodes.

[0031] Figure 1 This is a schematic diagram of the structure of the distributed data management system based on a multi-layer database provided by the present invention, as shown below. Figure 1 As shown, it includes: a central service node 100, and at least one client node 200 that is communicatively connected to the central service node 100.

[0032] In this embodiment, the central service node 100 can be a cluster consisting of one or more physical servers, virtual machines, or cloud server instances, and there are no restrictions on this. The central service node 100 is responsible for centrally managing shared parameters, maintaining data consistency, and coordinating communication between the nodes.

[0033] Here, client node 200 can be a server in another microservice cluster, or an application terminal, etc. Any device that needs to access and respond to changes in shared parameters can act as a client node.

[0034] In practical applications, the communication connection between the central service node 100 and the client node 200 can adopt various application layer protocols based on the TCP / IP protocol stack, or establish a long connection, or a short connection polling, etc.

[0035] In terms of system deployment, the central service node 100 is equipped with a shared parameter cache database 110 and at least one server application 120, while the client node 200 is equipped with at least one client application 210.

[0036] The shared parameter cache database 110 is the core data storage component running on the central service node 100. It is used to store all parameters in the system that need to be shared and synchronized. It is responsible for processing parameters from the client application 210 deployed on the client node 200 and the server application 120 deployed on the central service node 100.

[0037] Here, the parameters involved in this embodiment are typically represented as key-value pairs. The parameter key in the key-value pair is a unique identifier for the parameter. It can be a simple string or a complex structured string containing hierarchical information; there is no restriction on this. The parameter value in the key-value pair is the data corresponding to the parameter key, and its type can be arbitrary, such as integer, floating-point, boolean, string, etc.

[0038] The server application 120 and the client application 210 are business logic programs running on the central service node 100 and the client node 200, respectively. They can both rely on the shared parameters managed by the central service node 100 to complete their own functions.

[0039] In this embodiment, the shared parameter cache database 110 implements a data synchronization mechanism that follows a publish-subscribe pattern.

[0040] Specifically, parameter subscription is a prerequisite for proactive push notifications. When the server application 120 and / or client application 210 initialize or need to monitor a certain parameter, they send a subscription request to the central service node 100. This request explicitly includes one or more first parameter keys of interest. Upon receiving the subscription request, the central service node 100 associates the requester's identity information (such as ID, IP address, etc.) with the corresponding parameter key and records it in a subscription table. Thus, each parameter key maintains a list of all its subscribers.

[0041] Accordingly, when a subscriber no longer needs to pay attention to a certain parameter, they can send an unsubscribe request, and the central service node 100 will remove the subscriber from the subscription list.

[0042] Based on this, the shared parameter cache database is further configured as follows: When a write request for the first parameter key is received from the server application 120 and / or the client application 210, the parameter value corresponding to the first parameter key stored in the shared parameter cache database 110 is updated; after updating the parameter value, the updated parameter value is pushed to all subscribers of the first parameter key.

[0043] When either the server application 120 or the client application 210 needs to update a shared parameter, the application typically first constructs a write request containing a first parameter key and a new parameter value, and then sends the write request. Upon receiving the write request, the shared parameter cache database 110 is instructed to update the new parameter value stored corresponding to the first parameter key.

[0044] Furthermore, before writing the new parameter value corresponding to the first parameter key, an access lock can be applied to the relevant data, and then the lock can be released after the write is completed. If the parameter key does not yet exist in the shared parameter cache database, the corresponding storage space can also be created automatically.

[0045] After confirming the successful update of the parameter value, the push operation is executed immediately. The central service node 100 will look up all subscribers who have subscribed to the first parameter key in the previously established subscription table based on the first parameter key. Then, it will proactively and one by one push the update message containing the new parameter value to all subscribers. After receiving the push, all subscribers can update the parameter value they are using locally and execute the corresponding business logic.

[0046] The data distributed management system based on a multi-layer database provided in this invention achieves cross-machine data distributed management by constructing a distributed system architecture including a central service node 100 and multiple client nodes 200. Furthermore, the central service node 100 deploys a shared parameter cache database 110 and at least one server-side application 120, while the client nodes 200 deploy at least one client-side application 210. Based on the shared parameter cache database 110, updates are proactively pushed to all subscribers after parameter updates in the server-side application 120 and / or client-side application 210, improving data consistency and real-time performance throughout the distributed system, avoiding resource waste caused by invalid polling, and reducing system load.

[0047] In some embodiments, reference Figure 2 , Figure 2This is the second schematic diagram of the structure of the distributed data management system based on a multi-layer database provided by the present invention, as shown below. Figure 2 As shown, the server application 120 and / or the client application 110 integrate a parameter cache database 300 and a historical parameter cache database 400. The parameter cache database 300 is used to cache the latest parameter value corresponding to each parameter key; and the historical parameter cache database 400 is used to cache the historical time series parameter value corresponding to each parameter key.

[0048] Specifically, in this embodiment, each application that needs to interact with shared parameters (whether it is the central service node 100 or the client node 200) has a local data cache structure deployed inside, forming a second-level data management hierarchy and a third-level data management hierarchy.

[0049] In one example, when an application receives a push notification of a parameter update in the shared parameter cache database 110, it stores the latest parameter value in its local parameter cache database 300. When the application needs to read the value of that parameter, it can retrieve it directly and quickly from the local parameter cache database 300 without having to make a request to the central service node 100 over the network each time.

[0050] Here, the parameter cache database 300 can be a hash table, dictionary, or other efficient key-value storage structure in the application's memory, without restriction.

[0051] In another example, whenever the value of a parameter changes, in addition to updating the parameter cache database 300 mentioned above, a record of this change (e.g., the latest parameter value, change timestamp, etc.) is appended to the historical parameter cache database 400. This database allows for the review of historical data and provides batch query capabilities to support time series analysis.

[0052] The data distributed management system based on a multi-layer database provided in this embodiment of the invention, by setting up a multi-layer database structure, on the one hand, greatly improves the parameter reading performance through the local parameter caching database 300; on the other hand, it enhances the traceability and maintainability of the system through the historical parameter caching database 400.

[0053] In some embodiments, the parameter cache database 300 is configured as follows: The local storage space is divided into multiple shards, and parameter keys are used to index and access the corresponding parameter values; wherein, the parameter key includes at least one shard mapping dimension, which is used to map the parameter key to the corresponding shard.

[0054] Here, sharding refers to dividing local storage (such as hard drives and memory) into N non-overlapping, logically complete subsets according to preset sharding rules (such as range, hash value, and business attributes). Each subset is called a shard, and each shard can be stored independently on different physical nodes (servers, disk partitions, etc.) or logical regions. The shards do not overlap and, when combined, can completely cover the original data. For example, if the local storage space of the parameter cache database is 16GB, it can be divided into eight 2GB shards, with each shard independently managing a portion of the parameter key-value pairs.

[0055] Furthermore, in order to ensure that the parameter key can be accurately routed to its corresponding shard, in this embodiment, the parameter key is set to include at least one shard mapping dimension for mapping the parameter key to the corresponding shard.

[0056] Here, the sharding mapping dimension refers to the specific dimension in the parameter key used to locate the data shards, and is a key component of the high-dimensional key. By using the value of this sharding mapping dimension, the storage location of the shard corresponding to the parameter key value can be quickly calculated, avoiding traversing all shards for querying and directly improving access efficiency.

[0057] For example, if the parameter key is a three-dimensional key (device ID, x coordinate, y coordinate), then the device ID can be set as the sharding mapping dimension. By calculating the hash value of the device ID, the shard where the device parameter is stored can be located directly.

[0058] In practical applications, we can define a high-dimensional structure for the parameter keys (such as two-dimensional keys, three-dimensional keys, four-dimensional keys, etc.) based on the business scenario, and specify one or more of these dimensions as the sharding mapping dimensions.

[0059] The data distributed management system based on a multi-level database provided in this invention, through a sharding mechanism, can achieve more efficient data classification and querying compared to the traditional single-level index based on string keys.

[0060] In some embodiments, the server application 120 and / or the client application 210 are configured to perform a single locking operation, which includes the following steps: The first target slice is determined based on the slice mapping dimension of the single parameter key to be operated on; In the case of a write operation, an exclusive lock is applied to the first target fragment, and the corresponding parameter values ​​are written to the first target fragment; Alternatively, in the case of a read operation, a shared lock is applied to the first target fragment, and the corresponding parameter value is read from the first target fragment.

[0061] Specifically, when an application needs to read or write a single parameter, it first calculates or resolves the first target shard to which the parameter key belongs, based on the sharding mapping dimension of the single parameter key to be operated on. For example, if a hash mapping method is used, the first target shard to which the parameter key belongs is calculated using a preset mapping algorithm (such as hash modulo).

[0062] Next, the appropriate lock is applied and the operation is executed based on the operation type (read or write). Specifically, in the case of a write operation, the application needs to acquire an exclusive lock on the first target shard. An exclusive lock is exclusive; once successfully acquired, no other thread (whether reading or writing) can access the shard, thus ensuring the atomicity and data consistency of the write operation. After acquiring the lock, the application writes the corresponding parameter values ​​to the first target shard and then releases the lock. In the case of a read operation, the application acquires a shared lock on the first target shard. A shared lock allows multiple threads to perform read operations on the same shard simultaneously, but blocks any write operations. This mechanism allows multiple read operations to run concurrently, improving read efficiency. After acquiring the lock, the application reads the corresponding parameter values ​​from the first target shard and then releases the lock.

[0063] The data distributed management system based on a multi-layer database provided in this invention locates the target shard based on the sharding mapping dimension and applies exclusive locks or shared locks accordingly. While ensuring data consistency in high-concurrency scenarios, it minimizes the lock range to improve parameter access efficiency, thus achieving a balance between security and efficiency.

[0064] In some embodiments, the server application 120 and / or the client application 210 are configured to perform batch locking operations, which include the following steps: Based on the partitioning mapping dimension of the multiple second parameter keys to be operated on, the multiple second parameter keys are grouped into corresponding second target partitions; Generate a locking order corresponding to multiple second target fragments; Locks are applied sequentially to all the second target segments according to the locking order; While all the second target fragments are locked, perform batch data processing on multiple second parameter keys.

[0065] It should be understood that batch locking operation refers to a centralized read and write operation on multiple second parameter keys. The atomicity of batch processing is guaranteed throughout the process through a locking mechanism, which is suitable for scenarios that require simultaneous operation on multiple parameters.

[0066] In this embodiment, firstly, after receiving multiple second parameter keys, the application extracts the sharding mapping dimension of each key, calculates the second target shard to which each key belongs using a mapping algorithm, and groups all parameter keys according to their corresponding shards. For example, key_A and key_C may be mapped to shard_1, while key_B may be mapped to shard_5.

[0067] The application needs to generate a deterministic locking order for all involved secondary target shards (such as shard_1 and shard_5). For example, the shards can be sorted by their IDs in ascending order (e.g., locking order shard_1 > shard_5).

[0068] Next, the application applies locks to all the second target shards in sequence according to the locking order. For example, if the locking order is shard_1>shard_5, it can first try to lock shard_1, and then lock shard_5 if successful. If a shard is already locked by another operation, it waits for that shard to be released before continuing, until all shards are locked.

[0069] Finally, with all the second target fragments locked, the application begins performing bulk data processing (e.g., bulk reads or bulk writes) on multiple second parameter keys. After the operations are complete, the application releases all locks in the reverse order of locking (or any order).

[0070] It should be understood that in this embodiment, different types of locks are applied for different types of batch locked operations. Specifically, a shared lock is applied when performing a batch locked read operation, and an exclusive lock is applied when performing a batch locked write operation.

[0071] The data distributed management system based on a multi-layer database provided in this invention avoids the risk of deadlock while ensuring the atomicity and consistency of batch data processing by grouping batch parameter keys by shards, unifying the locking order, and locking all data before executing the operation.

[0072] In some embodiments, the server application 120 and / or the client application 210 are configured to perform batch lock-free operations, which include the following steps: When the external caller applies a lock to the third target slice corresponding to multiple third parameter keys, the multiple third parameter keys are traversed, and the parameter values ​​of the corresponding third parameter keys are stored and retrieved in the multiple third target slices.

[0073] Here, "batch lock-free operation" refers to batch data access operations for multiple third parameter keys. In this embodiment, "lock-free" does not mean completely eliminating the use of locks, but rather that the application executing the batch operation does not actively apply locks. Lock acquisition and release are handled by the external caller, who must ensure that all involved fragments are locked before entry.

[0074] Specifically, if the external caller has already applied locks to the third target shards corresponding to multiple third parameter keys, the application executing the batch operation iterates through the multiple third parameter keys and locates the corresponding third target shard (which has been externally locked) based on the complete high-dimensional structure of each third parameter key (including the shard mapping dimension). Within the corresponding third target shard, the application directly performs the parameter value access operation (write or read) without needing to check or apply locks again (because the external lock validity is guaranteed). Since the shards are locked, the batch operation will not be interfered with by other concurrent operations, and data consistency is guaranteed by the external locking mechanism.

[0075] Furthermore, after the application performing the batch operation has completed accessing all the third parameter keys, it can notify the external caller that the operation is complete, and the external caller can release the lock on the third target shard according to its policy, allowing other operations to access these shards.

[0076] The data distributed management system based on a multi-layer database provided in this embodiment of the invention transfers lock control to the external caller, enabling local applications to efficiently perform batch parameter access under the protection of existing locks. This improves the flexibility and execution efficiency of batch operations while ensuring data consistency.

[0077] In some embodiments, the historical parameter cache database 400 includes a first circular buffer, a second circular buffer, and a write head pointer corresponding to each parameter key: The first circular buffer is used to store parameter values ​​written in chronological order; The second circular buffer is used to store the Boolean value corresponding to each parameter value; The write head pointer is used to indicate the storage location of the next parameter value to be written in the first and second circular buffers.

[0078] It should be understood that a circular buffer refers to a storage space that is connected end to end. Its storage is written in chronological order. When the storage space is full, new data will overwrite the earliest written data.

[0079] In this embodiment, the second circular buffer is a storage space corresponding to the first circular buffer. It also adopts a circular structure and corresponds one-to-one with the storage location of the first circular buffer.

[0080] The write head pointer, an integer or counter, indicates the storage location of the next parameter value to be written in the first and second circular buffers. This pointer increments each time a new parameter value is written.

[0081] When new parameter values ​​are generated in chronological order, the historical parameter cache database 400 writes the new parameter value to position N of the first circular buffer according to the position indicated by the write head pointer (position N). At the same time, according to the state of the new parameter value (such as whether it has been validated), the corresponding boolean value (such as True) is written to position N of the second circular buffer.

[0082] After writing is complete, the write head pointer automatically updates its indicated position (indicated position N+1). If the maximum capacity of the buffer has been reached, the pointer jumps to position 0 (forming a circular loop). If the buffer is full, new data will overwrite the oldest data (e.g., when the pointer points to position 0, it overwrites the data at position 0), always preserving the latest timing data.

[0083] The data distributed management system based on a multi-layer database provided in this invention achieves efficient rolling storage, fast reading and writing, and status filtering of historical parameters by storing timing parameter values ​​in a first circular buffer, marking status in a second circular buffer, and synchronously managing the write position using a write head pointer.

[0084] In some embodiments, the server application 120 and / or the client application 210 are configured to perform a first read operation, the first read operation including the following steps: Determine the parameter value corresponding to the first read operation; If the parameter value is the latest parameter value, read the current value of the write head pointer, determine the latest parameter value sequence number based on the current value, and determine the first storage location of the current value in the first circular buffer and the second circular buffer; read the parameter value and Boolean value from the storage location; or, If the parameter value is a historical parameter value, determine the target historical parameter value sequence number specified by the first read operation, and determine the second storage location of the historical parameter value corresponding to the target historical parameter value sequence number in the first and second circular buffers according to the write head pointer and the capacity of the first circular buffer; read the parameter value and Boolean value from the storage location.

[0085] In this embodiment, different reading strategies are adopted for different types of parameter values ​​read in the first reading operation.

[0086] Specifically, if the parameter value to be read in the first read operation is the latest parameter value, the corresponding first circular buffer (storing parameter values) and second circular buffer (storing state Boolean values) are located according to the parameter key. The current write head pointer is read to determine the latest parameter value index = head - 1; the first storage location is determined to be the latest parameter value index. The parameter value is read from the first storage location in the first circular buffer, and the Boolean value is read from the same location in the second circular buffer. The parameter value and the Boolean value are then returned.

[0087] If the parameter value to be read in the first read operation is a historical parameter value, then the specific historical cursor value specified by the user (e.g., seq=1023) is determined, which is the target historical parameter value sequence number. Based on the current write head pointer head and the capacity N of the first circular buffer, the valid sequence number range is determined as [head-N, head-1] (N is the buffer capacity; for example, if head=1500 and N=1000, the valid range is 500~1499). If the target sequence number (e.g., 1023) is within the valid range, then the second storage location is determined based on the target historical parameter value sequence number and the capacity N of the first circular buffer. Finally, the parameter value of the first circular buffer and the Boolean value of the second circular buffer are read from the calculated second storage location, and the parameter value and Boolean value are returned.

[0088] The data distributed management system based on a multi-layer database provided in this invention distinguishes parameter value types and reads corresponding parameter values ​​and Boolean values ​​from a dual-ring buffer, thus fulfilling the need to read the latest values ​​and specific historical values, while balancing accuracy and access efficiency.

[0089] In some embodiments, the server application and / or the client application are configured to perform a second read operation, the second read operation including the following steps: Determine the starting sequence number and the number of reads corresponding to the second read operation; Based on the write head pointer and the capacity of the first circular buffer, determine whether the starting sequence number is within the valid sequence number window; If the starting sequence number is within the valid sequence number window, the parameter value of the number of reads and the corresponding Boolean value are read continuously starting from the storage location corresponding to the starting sequence number.

[0090] Here, the starting sequence number refers to the starting point of continuous reading, identifying the beginning position of a segment of historical data. The number of reads refers to the total number of parameter values ​​that need to be read continuously starting from the starting sequence number.

[0091] In this embodiment, when the application receives a batch read request containing a start sequence number and the number of reads, it first searches for the corresponding historical buffer (i.e., the first circular buffer and the second circular buffer) using the parameter keys. If it does not exist, it returns null or an error; if it exists, it proceeds to the next step of verification.

[0092] Specifically, the write head pointer `head` of the current buffer is read, and it is determined whether the input starting sequence number is within the valid sequence number window. The valid sequence number window is from `head-N` to `head-1`, where N is the capacity of the first circular buffer. If the starting sequence number is less than `head-N`, it means that the data has been overwritten, and an error or message indicating that part is invalid should be returned. If the starting sequence number is greater than `head-1`, it means that the sequence number has not yet been written, and an empty string or error should also be returned.

[0093] If the starting sequence number is valid, calculate and read the storage location one by one, starting from the starting sequence number. Finally, pack all the required parameter values ​​and their corresponding Boolean values ​​into an array in sequence number order, and return the actual starting and ending sequence numbers.

[0094] The data distributed management system based on a multi-layer database provided in this embodiment of the invention achieves efficient and accurate acquisition of batch historical data by determining the starting sequence number and the number of reads, verifying the validity of the sequence number, and continuously reading a specified number of parameter values ​​and Boolean values ​​from the corresponding position.

[0095] In some embodiments, the server application 120 and / or the client application 210 are configured as follows: Register the conditional function corresponding to the fourth parameter key; When writing the latest parameter value corresponding to the fourth parameter key into the first circular buffer, the judgment function is called, and the boolean value of the judgment function is written into the corresponding storage location in the second circular buffer.

[0096] Here, the fourth parameter key refers to the specific parameter key whose historical parameter value needs to be marked with a status. Each fourth parameter key corresponds to an independent first circular buffer and a second circular buffer.

[0097] In this embodiment, during the system initialization or parameter key creation phase, the user can register a judgment function for a specific fourth parameter key according to business needs, and establish a mapping relationship between the fourth parameter key and the judgment function (such as storing it through a hash table) to ensure that when the parameter value of the parameter key is written later, the corresponding judgment function can be found.

[0098] When the latest parameter value for the fourth parameter key is generated and needs to be written to the historical parameter cache database 400, the write position (e.g., position N) of the parameter value in the first circular buffer of the historical parameter cache database 400 is determined based on the write head pointer. A judgment function is called, passing the latest parameter value to the judgment function corresponding to the parameter key, and a boolean result (e.g., True) is obtained. Then, the latest parameter value is written to position N in the first circular buffer, and the boolean value (True) returned by the judgment function is written to position N in the second circular buffer. Finally, the pointer is moved from N to N+1 (circular loop), completing one write operation.

[0099] The data distributed management system based on a multi-layer database provided in this embodiment of the invention registers a custom judgment function for the fourth parameter key and automatically calls the function to generate a Boolean state and store it synchronously when the latest parameter value is written. While ensuring storage efficiency, it flexibly adapts to the state judgment requirements of different business scenarios.

[0100] This invention also provides a data distributed management method based on a multi-layer database, applied to a data distributed management system based on a multi-layer database. The system includes a central service node and at least one client node communicatively connected to the central service node. The central service node deploys a shared parameter cache database and at least one server-side application, while the client node deploys at least one client application. (Reference) Figure 3 As shown, Figure 3 This is one of the flowcharts of a data distributed management method based on a multi-layer database provided in an embodiment of the present invention. The method includes steps 310 and 320.

[0101] Step 310: When a write request for the first parameter key is received from the server application and / or the client application, the parameter value corresponding to the first parameter key stored in the shared parameter cache database is updated.

[0102] Step 320: After updating the parameter value, push the updated parameter value to all subscribers of the first parameter key.

[0103] It should be noted that the data distributed management system based on a multi-layer database involved in the embodiments of the present invention is the same as the data distributed management system based on a multi-layer database involved in the above embodiments, and will not be described in detail here.

[0104] In this embodiment, parameter subscription is a prerequisite for proactive push notifications. When the server-side application and / or client-side application initializes or needs to monitor a specific parameter, it sends a subscription request to the central service node. This request explicitly includes one or more first parameter keys of interest. Upon receiving the subscription request, the central service node associates the requester's identity information (such as ID, IP address, etc.) with the corresponding parameter key and records this information in a subscription table. Thus, each parameter key maintains a list of all its subscribers.

[0105] Correspondingly, when a subscriber no longer needs to pay attention to a certain parameter, they can send an unsubscribe request, and the central service node will remove the subscriber from the subscription list.

[0106] Specifically, before writing the new parameter value corresponding to the first parameter key, an access lock can be applied to the relevant data, and then the lock can be released after the write is completed. If the parameter key does not yet exist in the shared parameter cache database, the corresponding storage space can also be created automatically.

[0107] After confirming the successful update of the parameter value, the push operation is executed immediately. The central service node will use the first parameter key to find all subscribers subscribed to that first parameter key in the previously established subscription table. Subsequently, it will proactively and systematically push the update message containing the new parameter value to all subscribers. Upon receiving the push, all subscribers can update the parameter value they are currently using locally and execute the corresponding business logic.

[0108] The data distributed management method based on a multi-layer database provided in this invention can push the updated value to all subscribers of the parameter key after receiving a first parameter key write request from the server and / or client application and updating the corresponding parameter value in the shared parameter cache database. This achieves real-time updates of shared parameters and synchronous awareness among subscribers, ensuring the consistency and timeliness of parameter synchronization in the distributed system.

[0109] In some embodiments, reference Figure 4 As shown, Figure 4 This is a second flowchart illustrating a distributed data management method based on a multi-layered database provided in an embodiment of the present invention. The method further includes: Based on the parameter cache database integrated within the server application and / or the client application, the latest parameter value corresponding to each parameter key is cached; Based on the historical parameter cache database integrated within the server application and / or the client application, the historical time series parameter value corresponding to each parameter key is cached.

[0110] It should be understood that a circular buffer refers to a storage space that is connected end to end. Its storage is written in chronological order. When the storage space is full, new data will overwrite the earliest written data.

[0111] In this embodiment, the second circular buffer is a storage space corresponding to the first circular buffer. It also adopts a circular structure and corresponds one-to-one with the storage location of the first circular buffer.

[0112] The write head pointer, an integer or counter, indicates the storage location of the next parameter value to be written in the first and second circular buffers. This pointer increments each time a new parameter value is written.

[0113] When new parameter values ​​are generated in chronological order, the historical parameter cache database 400 writes the new parameter value to position N of the first circular buffer according to the position indicated by the write head pointer (position N). At the same time, according to the state of the new parameter value (such as whether it has been validated), the corresponding boolean value (such as True) is written to position N of the second circular buffer.

[0114] After writing is complete, the write head pointer automatically updates its indicated position (indicated position N+1). If the maximum capacity of the buffer has been reached, the pointer jumps to position 0 (forming a circular loop). If the buffer is full, new data will overwrite the oldest data (e.g., when the pointer points to position 0, it overwrites the data at position 0), always preserving the latest timing data.

[0115] The data distributed management method based on a multi-layer database provided in this invention achieves efficient rolling storage, fast reading and writing, and status filtering of historical parameters by storing timing parameter values ​​in a first circular buffer, marking status in a second circular buffer, and synchronously managing the write position using a write head pointer.

[0116] Figure 5 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 5 As shown, the scanning device may include: a processor 510, a communications interface 520, a memory 530, and a communication bus 540, wherein the processor 510, the communications interface 520, and the memory 530 communicate with each other via the communication bus 540. The processor 510 can call logical instructions in the memory 530 to execute a data distributed management method based on a multi-layer database, the method including: When a write request for the first parameter key is received from the server application and / or the client application, the parameter value corresponding to the first parameter key stored in the shared parameter cache database is updated. After updating the parameter value, push the updated parameter value to all subscribers of the first parameter key.

[0117] Furthermore, the logical instructions in the aforementioned memory 530 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0118] On the other hand, the present invention also provides a computer program product, the computer program product comprising a computer program that can be stored on a non-transitory computer-readable storage medium, wherein when the computer program is executed by a processor, the computer is able to execute the data distributed management method based on a multi-layer database provided by the above methods, the method comprising: When a write request for the first parameter key is received from the server application and / or the client application, the parameter value corresponding to the first parameter key stored in the shared parameter cache database is updated. After updating the parameter value, push the updated parameter value to all subscribers of the first parameter key.

[0119] In another aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the data distributed management method based on a multi-tiered database provided by the methods described above, the method comprising: When a write request for the first parameter key is received from the server application and / or the client application, the parameter value corresponding to the first parameter key stored in the shared parameter cache database is updated. After updating the parameter value, push the updated parameter value to all subscribers of the first parameter key.

[0120] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0121] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0122] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A distributed data management system based on a multi-layered database, characterized in that, include: A central service node, and at least one client node that is communicatively connected to the central service node; The central service node is equipped with a shared parameter cache database and at least one server application, while the client node is equipped with at least one client application. The shared parameter cache database is configured as follows: When a write request for the first parameter key is received from the server application and / or the client application, the parameter value corresponding to the first parameter key stored in the shared parameter cache database is updated. After updating the parameter value, push the updated parameter value to all subscribers of the first parameter key.

2. The data distributed management system based on a multi-layer database according to claim 1, characterized in that, The server application and / or the client application integrate a parameter cache database and a historical parameter cache database. The parameter cache database is used to cache the latest parameter value corresponding to each parameter key; as well as, The historical parameter cache database is used to cache the historical time series parameter values ​​corresponding to each parameter key.

3. The data distributed management system based on a multi-layer database according to claim 2, characterized in that, The parameter cache database is configured as follows: The local storage space is divided into multiple shards, and parameter keys are used to index and access the corresponding parameter values; wherein, the parameter key includes at least one shard mapping dimension, which is used to map the parameter key to the corresponding shard.

4. The data distributed management system based on a multi-layer database according to claim 3, characterized in that, The server application and / or the client application are configured to perform a single lock operation, which includes the following steps: The first target slice is determined based on the slice mapping dimension of the single parameter key to be operated on; In the case of a write operation, an exclusive lock is applied to the first target fragment, and the corresponding parameter values ​​are written to the first target fragment; Alternatively, in the case of a read operation, a shared lock is applied to the first target fragment, and the corresponding parameter value is read from the first target fragment.

5. The data distributed management system based on a multi-layer database according to claim 3, characterized in that, The server application and / or the client application are configured to perform batch locking operations, which include the following steps: Based on the partitioning mapping dimension of the multiple second parameter keys to be operated on, the multiple second parameter keys are grouped into corresponding second target partitions; Generate a locking order corresponding to multiple second target fragments; Locks are applied sequentially to all the second target segments according to the locking order; While all the second target fragments are locked, perform batch data processing on multiple second parameter keys.

6. The data distributed management system based on a multi-layer database according to claim 3, characterized in that, The server application and / or the client application are configured to perform batch lock-free operations, which include the following steps: When the external caller applies a lock to the third target slice corresponding to multiple third parameter keys, the multiple third parameter keys are traversed, and the parameter values ​​of the corresponding third parameter keys are stored and retrieved in the multiple third target slices.

7. The data distributed management system based on a multi-layer database according to claim 2, characterized in that, The historical parameter cache database includes a first circular buffer, a second circular buffer, and a write head pointer for each parameter key: The first circular buffer is used to store parameter values ​​written in chronological order; The second circular buffer is used to store the Boolean value corresponding to each parameter value; The write head pointer is used to indicate the storage location of the next parameter value to be written in the first and second circular buffers.

8. The data distributed management system based on a multi-layer database according to claim 7, characterized in that, The server application and / or the client application are configured to perform a first read operation, which includes the following steps: Determine the parameter value corresponding to the first read operation; If the parameter value is the latest parameter value, read the current value of the write head pointer, determine the latest parameter value sequence number based on the current value, and determine the first storage location of the current value in the first circular buffer and the second circular buffer; read the parameter value and Boolean value from the first storage location; or, If the parameter value is a historical parameter value, determine the target historical parameter value sequence number specified by the first read operation, and determine the second storage location of the historical parameter value corresponding to the target historical parameter value sequence number in the first and second circular buffers according to the write head pointer and the capacity of the first circular buffer; read the parameter value and Boolean value from the second storage location.

9. The data distributed management system based on a multi-layer database according to claim 7, characterized in that, The server application and / or the client application are configured to perform a second read operation, which includes the following steps: Determine the starting sequence number and the number of reads corresponding to the second read operation; Based on the write head pointer and the capacity of the first circular buffer, determine whether the starting sequence number is within the valid sequence number window; If the starting sequence number is within the valid sequence number window, the parameter value of the number of reads and the corresponding Boolean value are read continuously starting from the storage location corresponding to the starting sequence number.

10. The data distributed management system based on a multi-layer database according to claim 7, characterized in that, The server application and / or the client application are configured as follows: Register the conditional function corresponding to the fourth parameter key; When writing the latest parameter value corresponding to the fourth parameter key into the first circular buffer, the judgment function is called, and the boolean value of the judgment function is written into the corresponding storage location in the second circular buffer.

11. A data distributed management method based on a multi-layer database, applied to a data distributed management system based on a multi-layer database, the system comprising a central service node and at least one client node communicatively connected to the central service node; wherein, The central service node is equipped with a shared parameter cache database and at least one server-side application, and the client node is equipped with at least one client-side application. The method comprises: When a write request for the first parameter key is received from the server application and / or the client application, the parameter value corresponding to the first parameter key stored in the shared parameter cache database is updated. After updating the parameter value, push the updated parameter value to all subscribers of the first parameter key.

12. The data distributed management method based on a multi-layer database according to claim 11, characterized in that, The method further includes: Based on the parameter cache database integrated within the server application and / or the client application, the latest parameter value corresponding to each parameter key is cached; Based on the historical parameter cache database integrated within the server application and / or the client application, the historical time series parameter value corresponding to each parameter key is cached.

Images