Dynamic storage configuration method and system based on KV database

Abstract

The invention discloses a KV database-based dynamic storage configuration method. The method comprises the following steps of deploying a KV database and a configuration manager on each node of a distributed system cluster; setting configuration data in a key-value pair form required by the service in a KV database; a configuration change threshold Cth is defined, the change condition of the configuration data is continuously monitored, the configuration data is substituted into a historical change rate formula to calculate the current change rate, and when it is detected that the change rate CR exceeds the preset threshold Cth, a configuration updating process is triggered; adjusting a cache strategy, and executing a configuration updating process; an instruction is sent to a connected related service or application program for reloading configuration, and the execution configuration updating process specifically comprises the following steps that latest configuration data in a KV database is read in a configuration manager according to a preset configuration template; generating a configuration file in combination with the configuration template Tpl and the latest configuration data CD; and distributing the generated configuration file to each node of the system cluster.

Description

Technical Field

[0001] The present invention relates to the field of dynamic storage configuration of a KV database, and in particular to a dynamic storage configuration method and system based on a KV database. Background Art

[0002] With the rapid development of Internet technology, the information age has arrived. In this era, data processing, transmission and storage have become key technical links. All walks of life are actively seeking efficient and stable data processing solutions to improve business efficiency and user experience. Especially driven by emerging technologies such as distributed systems, cloud computing and big data, the demand for data management and configuration updates is becoming increasingly urgent, prompting continuous innovation and development in related technology fields.

[0003] In the current technical environment, configuration management and automated deployment have become important means to improve system flexibility and maintainability. Configuration management tools such as Spring Cloud Config are widely used in distributed systems to achieve centralized management and dynamic updates of configurations. At the same time, event bus technologies such as Spring Cloud Bus also provide strong support for configuration propagation and refresh, making it possible to update configurations without affecting service operation.

[0004] The current existing technology usually includes the following steps when performing configuration updates: first, setting and storing configuration information on the configuration server; second, the client integrates the configuration management tool and the event bus to receive configuration update notifications; then, when the configuration changes, the refresh command is sent to all connected clients through the event bus; finally, after receiving the command, the client obtains the latest configuration from the configuration server and reloads it to complete the configuration update process.

[0005] However, traditional systems often lack efficient caching mechanisms and are unable to effectively reduce direct access to configuration data read requests to the configuration server, resulting in high system latency. In particular, in high-concurrency scenarios, the configuration server is overloaded, affecting the stable operation of the system. Secondly, existing technologies usually fail to intelligently allocate cache resources and lack strategies for caching based on the frequency of configuration data use, resulting in unnecessary cache occupation, inefficient memory use, and low resource utilization. In addition, existing technologies often lack mechanisms for regularly evaluating cache efficiency and are unable to dynamically adjust cache strategies based on evaluation results, making it impossible for the cache state to always remain optimal, further affecting system performance and resource utilization. Summary of the invention

[0006] Based on this, the purpose of the present invention is to provide a dynamic storage configuration method and system based on a KV database to solve the technical problem that the existing methods have many unnecessary cache occupancy.

[0007] In view of the above problems, the present application provides a dynamic storage configuration method and system based on a KV database.

[0008] The present application provides a dynamic storage configuration method based on a KV database, comprising the following steps: deploying a KV database and a configuration manager on each node of a distributed system cluster; setting configuration data in the form of key-value pairs required for the business in the KV database; defining a configuration change threshold C_th, and continuously monitoring the changes in the configuration data, and bringing the configuration data into a historical change rate formula to calculate the current change rate, wherein the historical change rate formula is as follows:

[0009]

[0010] Where CR represents the current change rate, ΔVal represents the difference between the current configuration value and the previous configuration value, and ΔT represents the time interval between two configuration value changes, so as to evaluate the change of configuration data; when it is detected that the change rate CR exceeds the preset threshold C_th, the configuration update process is triggered; the cache strategy is adjusted and the configuration update process is executed; instructions are sent to the connected related services or applications to reload the configuration;

[0011] The execution configuration update process specifically includes the following steps: reading the latest configuration data in the KV database according to the preset configuration template in the configuration manager; generating a configuration file by combining the configuration template Tpl and the latest configuration data CD; and distributing the generated configuration file to each node of the system cluster.

[0012] In summary, the present invention mainly has the following beneficial effects:

[0013] 1. By introducing a cache submodule and adopting an LRU cache elimination strategy, the system can process configuration data read requests more efficiently. The cache mechanism reduces the number of direct accesses to the configuration server, thereby reducing system latency and improving overall performance. Especially in high-concurrency scenarios, the adjustment of the cache strategy can significantly reduce the load on the configuration server and ensure stable operation of the system.

[0014] 2. The present invention manually sets a frequency value and caches the configuration template parsing results and the latest configuration data that are higher than this value, so that the system can allocate cache resources more intelligently. This strategy avoids unnecessary cache occupation, optimizes memory usage, and improves resource utilization. At the same time, regularly evaluates cache efficiency and adjusts the strategy based on the evaluation results to ensure that the cache is always kept in the best state. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 This is the main logic diagram of the present invention;

[0016] Figure 2 This is the logic diagram of step S500 of the present invention. DETAILED DESCRIPTION

[0017] The following will be combined with the accompanying drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and cannot be understood as limiting the present invention.

[0018] The following describes an embodiment of the present invention based on its overall structure.

[0019] A dynamic storage configuration method based on a KV database comprises the following steps:

[0020] S100: deploy the KV database and configuration manager on each node of the distributed system cluster;

[0021] KV database is a Key-Value database. In a distributed system, services are usually deployed on multiple nodes to improve the scalability, reliability and performance of the system. KV database and configuration manager are common components in distributed systems. KV database is used to store and manage key-value data, which is efficient and concise; configuration manager is used to centrally manage and distribute configuration information to ensure the consistency of each node and dynamic configuration update.

[0022] For example, in a distributed system cluster consisting of 5 nodes, in order to deploy the KV database and configuration manager on each node, the corresponding software is installed on each node, the network is configured, and communication between nodes is ensured.

[0023] Among them, deploying the KV database and configuration manager on each node of the distributed system cluster also includes the following steps:

[0024] S110: Select ETCD as the KV database and configure it in cluster mode;

[0025] S120: The consistent hashing algorithm is used to ensure balanced distribution of data in the etcd cluster and improve data access efficiency.

[0026] ETCD is a highly available distributed key-value storage system, commonly used for configuration management, service discovery, synchronization, etc. ETCD supports cluster mode, that is, multiple ETCD instances can work together to provide services, thereby improving the reliability and availability of the system. In cluster mode, ETCD selects a leader through a mechanism to handle write operations and replicate these operations to other nodes to ensure data consistency.

[0027] The consistent hashing algorithm is a commonly used data distribution strategy in distributed systems. It can ensure the balanced distribution of data in the cluster and reduce the problems of data migration and uneven node load. In the ETCD cluster, the consistent hashing algorithm can be used to evenly distribute key-value pairs on each node, thereby improving data access efficiency and preventing certain nodes from becoming hot spots and causing performance bottlenecks.

[0028] S200: Setting configuration data in the form of key-value pairs required for the business in the KV database;

[0029] Configuration data in the form of key-value pairs allows us to flexibly store and manage various configuration information. Whether it is a simple parameter setting or a complex configuration structure, it can be stored in the form of key-value pairs.

[0030] When business requirements change, update the corresponding key-value pairs in the KV database. The KV database stores data in the form of key-value pairs, and the corresponding value can be directly located through the key without traversing the entire data set.

[0031] S300: Define a configuration change threshold C_th, and continuously monitor the change of configuration data. Substitute the configuration data into the historical change rate formula to calculate the current change rate. The historical change rate formula is as follows:

[0032]

[0033] Among them, CR represents the current change rate, ΔVal represents the difference between the current configuration value and the previous configuration value, and ΔT represents the time interval between two configuration value changes, which is used to evaluate the change of configuration data;

[0034] Among them, it also includes:

[0035] S310: Obtaining configuration data change information by periodic polling or event triggering;

[0036] S320: Calculate the change rate CR in real time and compare it with a preset threshold C_th to determine whether the configuration update process needs to be triggered.

[0037] First, define a configuration change threshold C_th, which is a standard for measuring the degree of change in configuration data. Then continuously monitor the changes in configuration data to detect and handle abnormal changes in a timely manner. For example, the maximum number of connections of the service component configuration parameter has a normal change range of less than 5%. Set C_th to 5%. Then, continuously monitor the changes in this configuration parameter.

[0038] In order to obtain configuration data change information, we can use periodic polling or event triggering. Periodic polling means actively querying the current value of configuration data at regular intervals and comparing it with the previous value. Event triggering means that when the configuration data changes, an event is triggered by the system or application, and the change information is notified to the monitoring program.

[0039] For example, for the configuration parameter of the maximum number of connections, its current value is polled every 1 minute and recorded in the monitoring system. For another example, an event listener is configured. When the "maximum number of connections" changes, the listener will receive a notification and send the change information to the monitoring program.

[0040] After obtaining the configuration data change information, the change rate CR is calculated in real time. The change rate CR can be calculated by comparing the difference between the current value and the previous value. Then, we compare CR with the preset threshold C_th. If CR exceeds C_th, it means that the configuration data has changed abnormally and the configuration update process needs to be triggered.

[0041] For example, the value of the maximum number of connections is 100, and the value in the current polling is 110. We can calculate the change rate CR as (110-100) / 100=10%. Since 10% is greater than the preset threshold of 5%, the configuration update process needs to be triggered.

[0042] S400: When it is detected that the change rate CR exceeds a preset threshold value C_th, a configuration update process is triggered;

[0043] Among them, it also includes:

[0044] S410: During the process of entering and modifying configuration data, real-time verification of user input is performed using regular expressions and logical expressions;

[0045] S420: Evaluate input quality using an error detection rate formula, where the error detection rate formula is as follows:

[0046]

[0047] Among them, ER represents the error rate, NE represents the number of incorrect inputs, and TI represents the total number of inputs;

[0048] S430: Retrieve corresponding error prompts and correction suggestions based on the error rate.

[0049] To prevent users from entering incorrect or unreasonable data, regular expressions and logical expressions are used for real-time verification during the input and modification of configuration data. This can effectively block illegal characters, format errors or logical contradictions.

[0050] Among them, regular expressions and logical expressions are two commonly used expression types in programming and data processing.

[0051] Regular expressions, or regexes for short, are a pattern used to match specific character combinations in a string. They can help us find, replace, or verify the content in a string quickly and accurately. For example, we can use regular expressions to find all the numbers in an email address and verify whether an input is a valid mobile phone number.

[0052] Logical expressions are expressions used to express logical relationships, and include logical operators (such as AND, OR, NOT) and logical variables. The result of a logical expression is usually a Boolean value (true or false), which is used to determine whether a certain condition is true. For example, in programming, logical expressions are used to determine whether a user has permission to access a certain page, or the size relationship between two numbers.

[0053] For example, when a user tries to set the price of a product, the system uses regular expressions to ensure that the input is a legal number, and uses logical expressions to check whether the price is within a reasonable range, such as not being lower than the cost price.

[0054] The error detection rate formula is used to quantify the degree of error in user input. When the system detects that the price entered by the user is negative or exceeds the preset maximum value, it will calculate the degree of error based on the error detection rate formula and decide whether to allow saving or prompt the user to correct it. Optionally, the system intelligently provides the user with error prompts and correction suggestions based on the error detection rate results. This helps users quickly locate the problem and make corrections based on the suggestions.

[0055] Among them, the above-mentioned intelligence retrieves the corresponding text content through cloud data or a preset corresponding text database combined with the recognized input text content, and outputs the retrieved text.

[0056] S500: Adjust the cache strategy and execute the configuration update process;

[0057] The step of adjusting the cache strategy includes the following steps:

[0058] S511: Introduce a cache submodule in the configuration manager and use an LRU cache elimination strategy, where LRU stands for Least Recently Used;

[0059] S512: Manually set the frequency value;

[0060] S513: Cache the parsing results and the latest configuration data of the configuration templates that are higher than the frequency value;

[0061] S514: Use a cache hit rate formula to periodically evaluate cache efficiency, and adjust the cache strategy according to the evaluation result. The cache hit rate formula is as follows:

[0062]

[0063] Among them, HR represents the cache hit rate, NH represents the number of cache hits, and TR represents the total number of requests.

[0064] The cache submodule is used to store frequently accessed configuration data. When the cache reaches the preset size, the least recently used cache item will be automatically eliminated to make room for new data.

[0065] A frequency value is manually set to determine which configuration template parsing results and the latest configuration data need to be cached. This frequency value can be adjusted according to actual business needs and system performance.

[0066] The system will cache the parsing results and the latest configuration data of configuration templates that are accessed more frequently than the set frequency value. This ensures that frequently accessed configuration data can be quickly obtained from the cache, improving system performance.

[0067] Use the cache hit rate formula regularly to evaluate the efficiency of the cache. If the cache hit rate is low, it means that the cache strategy may need to be adjusted, such as increasing the cache size, adjusting the frequency value, etc. By continuously optimizing the cache strategy, system performance and resource utilization can be improved.

[0068] The execution configuration update process specifically includes the following steps:

[0069] S521: Reading the latest configuration data in the KV database according to a preset configuration template in the configuration manager;

[0070] S522: Generate a configuration file by combining the configuration template Tpl and the latest configuration data CD;

[0071] S523: Distribute the generated configuration file to each node of the system cluster.

[0072] When the configuration data is updated, the configuration manager reads the latest configuration data from the KV database according to the preset configuration template to ensure that the configuration manager always obtains the latest configuration information.

[0073] The configuration manager will combine the latest configuration data and configuration template to generate a new configuration file. This configuration file will contain all the configuration information required by the system. Use Confd as the configuration manager and configure its parameters to read configuration data from ETCD. This ensures that Confd can correctly obtain configuration data from ETCD. Among them, ETCD is a distributed key-value storage system.

[0074] S522 also includes:

[0075] S522-1: The configuration manager selects Confd and configures its parameters for reading configuration data from ETCD;

[0076] S522-2: Use the concurrency control mechanism of Confd to manage the generation and distribution process of configuration files.

[0077] In the process of generating and distributing configuration files, Confd's concurrency control mechanism is used to ensure that multiple nodes do not try to modify the same configuration file at the same time, thereby avoiding conflicts and data inconsistencies. Finally, the configuration manager distributes the generated configuration file to each node in the system cluster. In this way, each node can obtain the latest configuration information and make corresponding adjustments and optimizations based on this information.

[0078] S600: Sending instructions to the connected related services or applications to reload configurations;

[0079] A dynamic storage configuration system based on a KV database, the system is used to carry and run all the above steps, the system includes:

[0080] KV database module, which is used to store system configuration data in the form of key-value pairs, supports cluster mode, and ensures balanced data distribution through consistent hashing algorithm;

[0081] The cache submodule, as part of the configuration manager module, uses the LRU algorithm to cache the parsing results and the latest configuration data of commonly used configuration templates to reduce the number of direct accesses to the KV database and improve the efficiency of configuration generation and distribution.

[0082] Preferably, it also includes:

[0083] The configuration manager module is connected to the KV database module and is used to execute the configuration update process, including reading the latest configuration data in the KV database according to the preset configuration template, generating configuration files and distributing them to each node of the system cluster, and supporting concurrency control mechanisms to improve efficiency;

[0084] The configuration verification module is used to perform real-time verification during the entry and modification of configuration data, provide error prompts and correction suggestions, and ensure the correctness of the configuration data.

[0085] Preferably, it also includes:

[0086] The monitoring and alarm module is used to continuously monitor the operating status of the KV database and configuration manager, discover potential problems through predefined performance indicators and alarm rules, and perform corresponding operations or send alarm notifications when failures or anomalies are detected;

[0087] The template management submodule, as part of the configuration manager module, is used to manage and maintain configuration templates to ensure that templates can be correctly parsed and applied when generating configuration files.

[0088] Preferably, it also includes:

[0089] The front-end page or API interface module is used to provide operation and maintenance personnel with an interface for entering and modifying configuration data, and supports data verification logic to ensure the correct entry and modification of configuration data;

[0090] The logging submodule, as part of the monitoring and alarm module, is used to record key events and abnormal information during system operation for troubleshooting and performance analysis.

[0091] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements a method for assembling tree-structured data in reverse order; an electronic device includes a processor and a memory, in which a computer program is stored, and when executed by the processor, the computer program implements a method for assembling tree-structured data in reverse order.

[0092] Those skilled in the art will understand that the various digital numbers such as the first and second involved in this application are only for the convenience of description, and are not used to limit the scope of this application, nor do they indicate a sequence of precedence. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the objects associated with each other are in an "or" relationship. "At least one" refers to one or more. At least two refers to two or more. "At least one", "any one" or similar expressions refer to any combination of these items, including any combination of single items (individuals) or plural items (individuals). For example, at least one item (individual, kind) of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, where a, b, c can be single or multiple.

[0093] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in this application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices.

[0094] The steps of the method or algorithm described in this application can be directly embedded in hardware, software units executed by a processor, or a combination of the two. The software unit can be stored in RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, register, hard disk, removable disk, CD-ROM or any other form of storage medium in the art. Exemplarily, the storage medium can be connected to the processor so that the processor can read information from the storage medium and write information to the storage medium. Optionally, the storage medium can also be integrated into the processor. The processor and the storage medium can be arranged in an ASIC, and the ASIC can be arranged in a terminal. Optionally, the processor and the storage medium can also be arranged in different components in the terminal. These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are performed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide for implementing the process in the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0095] Although the present application has been described in conjunction with specific features and embodiments thereof, it is obvious that various modifications and combinations may be made thereto without departing from the spirit and scope of the present application. Accordingly, this specification and the accompanying drawings are merely exemplary illustrations of the present application and are deemed to have covered any and all modifications, variations, combinations or equivalents within the scope of the present application. Obviously, a person skilled in the art may make various modifications and variations to the present application without departing from the scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the present application and its equivalents, the present application is intended to include these modifications and variations.

Claims

1. A dynamic storage configuration method based on KV database, characterized in that: The following steps are involved: Deploy the KV database and configuration manager on each node of the distributed system cluster; Set the configuration data in the form of key-value pairs required by the business in the KV database; Define the configuration change threshold C_th, and continuously monitor the changes in configuration data. Substitute the configuration data into the historical change rate formula to calculate the current change rate. The historical change rate formula is as follows: Among them, CR represents the current change rate, ΔVal represents the difference between the current configuration value and the previous configuration value, and ΔT represents the time interval between two configuration value changes, which is used to evaluate the change of configuration data; When it is detected that the change rate CR exceeds the preset threshold C_th, the configuration update process is triggered; Adjust cache strategy and execute configuration update process; Send instructions to connected services or applications to reload configuration; The execution configuration update process specifically includes the following steps: In the configuration manager, read the latest configuration data in the KV database according to the preset configuration template; Generate a configuration file by combining the configuration template Tpl and the latest configuration data CD; Distribute the generated configuration file to each node in the system cluster.

2. The method for dynamic storage configuration based on KV database according to claim 1 is characterized in that: The deployment of the KV database and configuration manager on each node of the distributed system cluster includes: Select ETCD as the KV database and configure it in cluster mode; The consistent hashing algorithm ensures balanced distribution of data in the etcd cluster and improves data access efficiency.

3. The method for dynamic storage configuration based on KV database according to claim 2 is characterized in that: The configuration change threshold C_th is defined, and the change of configuration data is continuously monitored, and the configuration data is brought into the historical change rate formula to calculate the current change rate, including: Use periodic polling or event triggering to obtain configuration data change information; The change rate CR is calculated in real time and compared with the preset threshold C_th to determine whether the configuration update process needs to be triggered.

4. The method for dynamic storage configuration based on KV database according to claim 1 is characterized in that: When the change rate CR is detected to exceed a preset threshold value C_th, the configuration update process is triggered, including: During the process of entering and modifying configuration data, user input is verified in real time through regular expressions and logical expressions; The error detection rate formula is used to evaluate the input quality, where the error detection rate formula is as follows: Among them, ER represents the error rate, NE represents the number of incorrect inputs, and TI represents the total number of inputs; According to the error rate, the corresponding error prompts and correction suggestions are retrieved.

5. The method for dynamic storage configuration based on KV database according to claim 1 is characterized in that: The step of combining the configuration template Tpl and the latest configuration data CD to generate a configuration file includes: The configuration manager selects Confd and configures its parameters for reading configuration data from etcd; Use Confd's concurrency control mechanism to manage the generation and distribution of configuration files.

6. The method for dynamic storage configuration based on KV database according to claim 1 is characterized in that: The adjusting the cache strategy also includes: Introduced a cache submodule in the configuration manager and used the LRU cache eviction strategy, where LRU stands for Least Recently Used; Manually set the frequency value; Cache the parsing results and latest configuration data of configuration templates that are higher than the frequency value; Use the cache hit rate formula to regularly evaluate cache efficiency and adjust the cache strategy based on the evaluation results. The cache hit rate formula is as follows: Among them, HR represents the cache hit rate, NH represents the number of cache hits, and TR represents the total number of requests.

7. According to the contents of claims 1-6 above, the present application also provides a dynamic storage configuration system based on a KV database, characterized in that: The system is used to carry out and execute the steps of claims 1 to 6 above, and the system comprises: The KV database module is used to store system configuration data in the form of key-value pairs, supports high availability, high concurrency and strong consistency cluster mode, and ensures balanced data distribution through consistent hashing algorithm; The cache submodule, as part of the configuration manager module, uses the LRU algorithm to cache the parsing results and the latest configuration data of commonly used configuration templates to reduce the number of direct accesses to the KV database and improve the efficiency of configuration generation and distribution.

8. A dynamic storage configuration system based on KV database according to claim 7, characterized in that: Also includes: The configuration manager module is connected to the KV database module and is used to execute the configuration update process, including reading the latest configuration data in the KV database according to the preset configuration template, generating configuration files and distributing them to each node of the system cluster, and supporting concurrency control mechanisms to improve efficiency; The configuration verification module is used to perform real-time verification during the entry and modification of configuration data, provide error prompts and correction suggestions, and ensure the correctness of the configuration data.

9. A dynamic storage configuration system based on KV database according to claim 8, characterized in that: Also includes: The monitoring and alarm module is used to continuously monitor the operating status of the KV database and configuration manager, discover potential problems through predefined performance indicators and alarm rules, and perform corresponding operations or send alarm notifications when failures or anomalies are detected; The template management submodule, as part of the configuration manager module, is used to manage and maintain configuration templates to ensure that templates can be correctly parsed and applied when generating configuration files.

10. A dynamic storage configuration system based on a KV database according to claim 7, characterized in that: include: The front-end page or API interface module is used to provide operation and maintenance personnel with an interface for entering and modifying configuration data, and supports data verification logic to ensure the correct entry and modification of configuration data; The logging submodule, as part of the monitoring and alarm module, is used to record key events and abnormal information during system operation for troubleshooting and performance analysis.

Images