Application configuration cache self-adaptive updating method and system based on configuration subscription and incremental analysis

Abstract

The invention relates to the technical field of computer data processing, in particular to an application configuration cache self-adaptive updating method and system based on configuration subscription and incremental analysis, and the method comprises the steps that a target service reads target application configuration for the first time and subscribes registration information to a subscription registration center; acquiring database change record information, and extracting update information configured by the target application according to the change record information; extracting a configuration mark and an operation type related to the target application configuration according to the update information of the target application configuration; and matching the application service of the target business according to the subscription registration information configured by the target application and the configuration mark, and synchronizing the update information to the application service cache according to the operation type. The method aims at solving the technical problem that a large number of invalid query requests are generated when a database is actively polled.

Description

Technical Field

[0001] The present invention relates to the technical field of computer data processing, and in particular to an application configuration cache adaptive updating method and system based on configuration subscription and incremental parsing. Background Art

[0002] In modern application development, configuration management is a key factor in ensuring system flexibility and maintainability. Generally, configurations can be divided into two categories: system configuration and business configuration. System configurations are generally stored in configuration centers or static configuration files. They change less frequently and often require a service restart to take effect. Therefore, such configurations are loaded into memory when the service starts and remain valid until the service is terminated. In contrast, business configurations have a higher frequency of change and require real-time effectiveness without stopping the service. These configurations are usually saved in a database for dynamic adjustment.

[0003] However, there is a significant performance bottleneck in directly querying the database to obtain the latest configuration when processing each business request. Frequent database access not only consumes a lot of resources, but may also cause tremendous pressure on the database in high-concurrency scenarios. To alleviate this problem, the existing technology uses a caching mechanism to optimize the access efficiency of configuration data. Specifically, the configuration information read from the database will be temporarily saved in the application's memory, and a certain cache validity period (such as 1 minute) will be set. During this period, all access to the configuration item will directly use the copy in the cache, thereby avoiding unnecessary database interactions; once the set validity period is exceeded, the latest configuration data will be pulled from the database again and the cache will be updated.

[0004] Although the above method can improve the system response speed and reduce the database burden to a certain extent, even if the actual configuration has not changed, a database query operation still needs to be performed every cache cycle. When multiple instances of the same application are deployed, the database access volume will increase exponentially. At the same time, ensuring the consistency between the cache and the database is a complex task that requires the introduction of complex synchronization processing logic. On the other hand, due to the use of a timed refresh method, the actual effective time of the configuration may be delayed, up to the validity period of the cache setting, which limits application scenarios that require immediate response. Summary of the invention

[0005] In order to solve the technical problem of actively polling a database to generate a large number of invalid query requests, the present invention provides an application configuration cache adaptive update method and system based on configuration subscription and incremental parsing. The technical solution adopted is as follows:

[0006] The technical solution of the first aspect of the present invention provides an application configuration cache adaptive update method based on configuration subscription and incremental parsing, the method comprising:

[0007] The target business reads the target application configuration for the first time and subscribes to the registration information from the subscription registration center;

[0008] Obtain database change record information, and extract update information of target application configuration based on the change record information;

[0009] Extracting configuration flags and operation types related to the target application configuration according to the update information of the target application configuration;

[0010] The application service of the target business is matched according to the subscription registration information and configuration flags configured by the target application, and the update information is synchronized to the application service cache according to the operation type.

[0011] Furthermore, the target service reads the target application configuration for the first time and subscribes to the registration information from the subscription registration center, including:

[0012] When the target service reads the target application configuration for the first time, it checks whether the target application configuration exists in the cache. If not, it queries the target application configuration from the database and caches it, and starts an asynchronous thread to subscribe to the registration information from the configuration subscription registration center.

[0013] Furthermore, the subscription registration information at least includes:

[0014] Service name, used by the subscription registration center to identify different services;

[0015] Configuration update callback address, used to send change notifications to the corresponding application service when changes occur;

[0016] The database name, table name, and field name are used to match the configuration scope that the application service is concerned about.

[0017] Furthermore, the cache of the application service is a memory cache for storing application configuration data, and the configuration data in the cache remains unchanged before receiving an update notification from a subscription registration center.

[0018] Furthermore, the method further comprises:

[0019] When the target application configuration is read again after the first read, check whether the subscription of the target application configuration to the subscription registration center is successful. If successful, the configuration in the memory cache is directly used; if the subscription fails, the application service records the subscription failure information and re-executes the query configuration and subscription operations during the first read.

[0020] Furthermore, the subscription registration center is connected to the database in communication, and the subscription registration center obtains the change record information from the log file of the database, and extracts the update information related to the target application configuration.

[0021] Further, extracting configuration flags and operation types related to the target application configuration according to the update information of the target application configuration includes:

[0022] The subscription registration center extracts the database name, table name, and field name related to the change as the configuration mark according to the update information of the target application configuration, and determines the operation type.

[0023] Further, matching the application service of the target business according to the subscription registration information and the configuration flag configured by the target application, and synchronizing the update information to the application service cache according to the operation type includes:

[0024] Based on the configuration scope that the application service is interested in in the subscription registration information, the database name, table name, and field name related to the extracted changes are matched to extract the target application service to which the changes need to be pushed;

[0025] After receiving the change notification, the target application service synchronizes the change to the cache according to the change operation type.

[0026] Furthermore, synchronizing the changes to the cache according to the change operation type includes:

[0027] When the change operation type is update, the value of the corresponding configuration item in the cache is updated to the new value;

[0028] When the change operation type is new, add new configuration content to the cache;

[0029] When the change operation type is delete, the corresponding configuration item is deleted from the cache.

[0030] The technical solution of the second aspect of the present invention provides an application configuration cache adaptive update system based on configuration subscription and incremental parsing, the system adopts the application configuration cache adaptive update method based on configuration subscription and incremental parsing described in the technical solution of the first aspect of the present invention, the system includes:

[0031] The application service module includes a cache unit and a service processing unit, wherein the cache unit is used to store application configurations, and the service processing unit is used to process services and read and update the configurations in the cache when necessary;

[0032] The subscription registration center module is used to receive subscription information of application services, receive log files from the database and extract change information, match application services that need to be pushed changes according to subscription information, and push configuration change notifications to application services;

[0033] The database module is used to store application configuration data and generate log files to record changes when data is updated.

[0034] The present invention has the following beneficial effects:

[0035] The present invention provides an adaptive update method for application configuration cache based on configuration subscription and incremental parsing. The application service obtains the target application configuration from the database only when reading the configuration for the first time, and completes the subscription registration at the same time. After that, except for the case of subscription failure, the application service does not need to actively query the database; by obtaining database change record information, extracting the update information of the target application configuration according to the change record information, extracting the configuration flag and operation type related to the target application configuration, the application service relies on the update information pushed by the subscription registration center to update the cache, avoiding unnecessary database polling, thereby greatly reducing invalid checks; on the other hand, compared with the problem of time interval in the prior art of actively polling the database, this method can timely obtain the change record information of the database, and synchronize the update information to the cache of the target application service. The cache of the application service can be quickly updated, which greatly shortens the effective time after the configuration change, so that the application service can timely process the business based on the new configuration, and improves the response speed of the entire system to configuration changes. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the technical solutions and advantages in the embodiments of the present invention or the prior art, the drawings required for use in the embodiments or the prior art descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0037] Figure 1 A method flow chart of a method for adaptively updating an application configuration cache based on configuration subscription and incremental parsing provided by an embodiment of the present invention;

[0038] Figure 2 A schematic diagram of the structure of an application configuration cache adaptive update system based on configuration subscription and incremental parsing provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0039] In order to further explain the technical means and effects adopted by the present invention to achieve the predetermined purpose of the invention, the following, in combination with the accompanying drawings and preferred embodiments, describes in detail the specific implementation method, structure, features and effects of an application configuration cache adaptive update method and system based on configuration subscription and incremental parsing proposed by the present invention. In the following description, different "one embodiment" or "another embodiment" does not necessarily refer to the same embodiment. In addition, specific features, structures or characteristics in one or more embodiments may be combined in any suitable form.

[0040] Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs.

[0041] The following is a detailed description of a specific solution of an application configuration cache adaptive update method and system based on configuration subscription and incremental parsing provided by the present invention in conjunction with the accompanying drawings.

[0042] See also Figure 1 , which shows a method flow chart of an application configuration cache adaptive update method based on configuration subscription and incremental parsing provided by an embodiment of the present invention, the method comprising:

[0043] Step S100: the target service reads the target application configuration for the first time and subscribes to the registration information from the subscription registration center; specifically, when the target service reads the target application configuration for the first time, it checks whether the target application configuration exists in the cache, and the cache system can be a memory-based data structure, such as a hash table or a similar fast search storage mechanism; if not, the target application configuration is queried from the database and cached, and an asynchronous thread is started to subscribe to the registration information from the configuration subscription registration center; in the asynchronous thread, a subscription registration request is initiated to the configuration subscription registration center; first, a subscription registration message containing necessary information is constructed, and then the subscription registration message is sent to the configuration subscription registration center through a network communication protocol (such as HTTP, TCP, etc.); after receiving the message, the subscription registration center stores the subscription information of the application service in its internal subscription management data structure for subsequent configuration update matching.

[0044] The subscription registration information at least includes:

[0045] Service name, used by the subscription registration center to identify different services;

[0046] Configuration update callback address, used to send change notifications to the corresponding application service when changes occur; this address can be a URL based on HTTP or an address format supported by other network communication protocols; specifically, in the subscription registration center, a mapping relationship between the service name of each application service and the corresponding configuration update callback address is stored; when the configuration in the database changes and is determined to be related to a certain application service through matching, the subscription registration center sends the change notification to the configuration update callback address of the application service through a network request based on the stored mapping relationship; for example, if the HTTP protocol is used, the subscription registration center will initiate a POST request to the above callback address and include the change information in the request body;

[0047] The database name, table name, and field name are used to match the configuration scope that the application service is concerned about. Specifically, after the subscription registration center receives the change record information of the database, it needs to parse the change information to find out the database name, table name, and field name involved in the change. Then, this information is matched with the database name, table name, and field name in the subscription registration information of each application service. For example, if the change information shows that the "order_status" field of the "orders" table in the "ecommerceDB" has been updated, the subscription registration center will find all application services that have subscribed to the combination of database name, table name, and field name, and send the change notification to them. In this way, it is ensured that only application services that are really concerned about specific configuration changes will receive notifications, avoiding unnecessary information push and processing.

[0048] Preferably, the cache of the application service is a memory cache for storing application configuration data, and the configuration data in the cache remains unchanged before receiving an update notification from a subscription registration center; specifically, when the application service is started and cache initialization and configuration data storage are completed, the cache has no expiration time and will continue to monitor the network connection from the subscription registration center. Only when a valid update notification sent by the subscription registration center is received will the cache data update operation be triggered, thereby ensuring that the configuration data in the cache remains unchanged before this; since the configuration data in the cache remains unchanged before receiving an update notification, the application service does not need to frequently query the database to verify whether the configuration has been updated, thereby avoiding the performance overhead caused by a large number of repeated database query operations, and the configuration data in the cache can be kept synchronized with the configuration source in the database.

[0049] Step S100 also includes: when reading the target application configuration again after the first reading, checking whether the subscription of the target application configuration to the subscription registration center is successful, if successful, directly using the configuration in the memory cache; if the subscription fails, the application service records the subscription failure information, and re-executes the query configuration and subscription operation during the first reading; specifically, this can be achieved by setting a subscription status mark inside the application service, which can be set to "in progress" at the time of initial subscription, and updated to "successful" after receiving a successful confirmation message from the subscription registration center. If no successful confirmation is received within a certain period of time or feedback of subscription failure is received, it is marked as "failed". At the same time, the subscription status record of each application service can also be maintained in the subscription registration center, and the application service can obtain the latest subscription status information by sending a query request to the subscription registration center; if the subscription is successful, the configuration data is directly obtained from the memory cache and used.

[0050] In summary, in step S100, the application service only obtains the target application configuration from the database when reading the configuration for the first time, and completes the subscription registration at the same time. Thereafter, unless a subscription failure occurs, the application service does not need to actively query the database, ensuring that the application service can efficiently use the configuration in the cache when the subscription is successful, making full use of the performance advantages of the memory cache, reducing database queries, and improving business processing efficiency. At the same time, in the event of a subscription failure, by re-executing the query and subscription operations, it is ensured that the application service will not lose the ability to obtain configuration updates due to subscription problems, thereby enhancing the reliability and stability of the system, ensuring that the application service can always run under the correct configuration, and can respond to configuration changes in a timely manner.

[0051] Step S200: Obtain database change record information, and extract update information of target application configuration according to the change record information; specifically, the subscription registration center is connected to the database in communication, and the subscription registration center obtains change record information from the database log file, and extracts update information related to the target application configuration; specifically, Canal is used to capture database change data, which is an open source database change data capture (CDC) tool of Alibaba, mainly used for incremental log parsing based on MySQL database, providing incremental data subscription and consumption; the working principle of Canal is to capture data changes by parsing MySQL binary log; the log file of the database is Binlog, that is, binary log, which is a log file in MySQL database used to record all modifications to database data. Binlog is stored in binary form and records all SQL statements that modify the database, including data definition language (DDL) and data manipulation language (DML) statements; step S200 obtains and extracts update information related to the target application configuration from the database log file, thereby realizing effective monitoring and screening of database changes; this method avoids the need for application services to actively poll the database to check whether the configuration is updated, reduces unnecessary database query overhead, and improves the overall performance of the system. At the same time, only information related to the target application configuration is extracted, which can accurately focus subsequent update operations on application services that really need to be updated, reduce the transmission and processing of irrelevant information, improve the pertinence and efficiency of configuration updates, and ensure that application services can obtain configuration changes that affect them in a timely manner.

[0052] Step S300: extract the configuration flag and operation type related to the target application configuration according to the update information of the target application configuration; specifically, the subscription registration center extracts the database name, table name and field name related to the change as the configuration flag according to the update information of the target application configuration, and determines the operation type; after the subscription registration center obtains the update information related to the target application configuration extracted in step S200, it traverses the list of these update information; for each update information item, prepare to extract the configuration flag and operation type; by using Canal to obtain and filter the update information related to the target application configuration from the database Binlog, the application service avoids active polling of the database, greatly reduces the database query overhead, improves the overall system performance, and realizes effective monitoring and screening of change information, ensuring that only changes related to the target application configuration will be further processed. On this basis, step S300 accurately extracts the configuration flag and operation type, so that the system can accurately determine which application services need to receive update notifications and what type of cache update operations to perform, thereby improving the pertinence and accuracy of configuration updates, avoiding the spread of irrelevant information in the system, reducing resource waste, and ensuring that application services can update the configuration in the cache in a timely and accurate manner according to changes in the database, thereby ensuring the consistency of configuration in the entire system and the stable operation of application services.

[0053] Step S400: Match the application service of the target business according to the subscription registration information and configuration flag configured by the target application, and synchronize the update information to the application service cache according to the operation type.

[0054] Step S400 specifically includes:

[0055] Step S410: Based on the configuration scope of interest to the application service in the subscription registration information, combined with the extracted change-related database name, table name and field name, matching is performed to extract the target application service whose change needs to be pushed; specifically, after obtaining the change-related database name, table name and field name from step S300, the stored application service subscription information is traversed, and for each application service subscription information, the configuration scope of interest is compared with the changed database name, table name and field name one by one. If the two are completely matched or the inclusion relationship is established, the application service is marked as a target application service that needs to receive the change notification; finally, all successfully matched application services are extracted to form a target application service list, which will be used for subsequent change notification push.

[0056] Step S420: After receiving the change notification, the target application service synchronizes the change to the cache according to the change operation type; specifically, for the target application service list determined in step S410, the configuration update callback address of each application service is used to send the change notification to them; the notification content includes the change operation type (update, add, delete) and specific update information (such as updated value, added configuration content or deletion conditions, etc.). On the target application service side, a corresponding receiving mechanism is set to process the change notification from the subscription registration center. When receiving the notification, the change operation type in the notification is first parsed. If it is an update operation, the corresponding configuration item is found in the cache of the application service according to the new value information in the notification, and its value is updated to the new value; for a new operation, the new configuration content in the notification is added to the cache; when a deletion operation notification is received, the corresponding configuration item is found and deleted in the cache according to the deletion conditions in the notification (such as the key or other identification information of the configuration item);

[0057] Step S400 ensures that only application services that really need configuration updates will receive notifications by accurately matching the target application services, avoiding unnecessary interference with irrelevant application services, reducing the waste of network communication and processing resources, and improving the overall efficiency of the system; by accurately updating the cache in the target application service according to the change operation type, the consistency of the cache of the application service and the database configuration is guaranteed, so that the application service can be based on the latest configuration information when processing business, thereby ensuring the correct execution of business logic; this adaptive update mechanism can efficiently handle configuration updates, additions or deletions, maintains the stability and reliability of the system during configuration changes, and effectively supports the continuous and stable operation of application services in complex business environments.

[0058] In summary, the present invention provides an adaptive update method for application configuration cache based on configuration subscription and incremental parsing. The application service only obtains the target application configuration from the database when reading the configuration for the first time, and completes the subscription registration at the same time. After that, unless the subscription fails, the application service does not need to actively query the database; by obtaining the database change record information, extracting the update information of the target application configuration according to the change record information, extracting the configuration flag and operation type related to the target application configuration, the application service relies on the update information pushed by the subscription registration center to update the cache, avoiding unnecessary database polling, thereby greatly reducing invalid checks; on the other hand, compared with the existing technology of actively polling the database, there is a time interval problem. This method can obtain the change record information of the database in a timely manner, and synchronize the update information to the cache of the target application service. The cache of the application service can be quickly updated, which greatly shortens the effective time after the configuration change, so that the application service can process the business based on the new configuration in a timely manner, and improves the response speed of the entire system to configuration changes.

[0059] See also Figure 2 , which shows a schematic diagram of the structure of an application configuration cache adaptive update system based on configuration subscription and incremental resolution provided by an embodiment of the present invention. The technical solution of the second aspect of the present invention provides an application configuration cache adaptive update system based on configuration subscription and incremental resolution. The system adopts the application configuration cache adaptive update method based on configuration subscription and incremental resolution described in the technical solution of the first aspect of the present invention. The system includes:

[0060] The application service module includes a cache unit and a service processing unit, wherein the cache unit is used to store application configurations, and the service processing unit is used to process services and read and update the configurations in the cache when necessary;

[0061] The subscription registration center module is used to receive subscription information of application services, receive log files from the database and extract change information, match application services that need to be pushed changes according to subscription information, and push configuration change notifications to application services;

[0062] The database module is used to store application configuration data and generate log files to record changes when data is updated.

[0063] It should be noted that the sequence of the above embodiments of the present invention is only for description and does not represent the advantages and disadvantages of the embodiments. The processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0064] The various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referenced to each other, and each embodiment focuses on the differences from other embodiments.

Claims

1. An application configuration cache adaptive update method based on configuration subscription and incremental parsing, characterized in that: The method comprises: The target business reads the target application configuration for the first time and subscribes to the registration information from the subscription registration center; Obtain database change record information, and extract update information of target application configuration based on the change record information; Extracting configuration flags and operation types related to the target application configuration according to the update information of the target application configuration; The application service of the target business is matched according to the subscription registration information and configuration flags configured by the target application, and the update information is synchronized to the application service cache according to the operation type.

2. The method for adaptively updating application configuration cache based on configuration subscription and incremental parsing according to claim 1, characterized in that: The target business first reads the target application configuration and subscribes to the subscription registration center. The registration information includes: When the target service reads the target application configuration for the first time, it checks whether the target application configuration exists in the cache. If not, it queries the target application configuration from the database and caches it, and starts an asynchronous thread to subscribe to the registration information from the configuration subscription registration center.

3. The method for adaptively updating application configuration cache based on configuration subscription and incremental parsing as claimed in claim 2, characterized in that: The subscription registration information at least includes: Service name, used by the subscription registration center to identify different services; Configuration update callback address, used to send change notifications to the corresponding application service when changes occur; The database name, table name, and field name are used to match the configuration scope that the application service is concerned about.

4. The method for adaptively updating application configuration cache based on configuration subscription and incremental parsing as claimed in claim 2, characterized in that: The cache of the application service is a memory cache, which is used to store application configuration data, and the configuration data in the cache remains unchanged before receiving an update notification from the subscription registration center.

5. The method for adaptively updating application configuration cache based on configuration subscription and incremental parsing according to claim 1, characterized in that: The method further includes: When the target application configuration is read again after the first read, check whether the subscription of the target application configuration to the subscription registration center is successful. If successful, the configuration in the memory cache is directly used; if the subscription fails, the application service records the subscription failure information and re-executes the query configuration and subscription operations during the first read.

6. The method for adaptively updating application configuration cache based on configuration subscription and incremental parsing as claimed in claim 1, characterized in that: The subscription registration center is connected to the database in communication, and the subscription registration center obtains change record information from the log file of the database and extracts update information related to the target application configuration.

7. The method for adaptively updating application configuration cache based on configuration subscription and incremental parsing according to any one of claims 1 to 6, characterized in that: According to the update information of the target application configuration, the configuration flags and operation types related to the target application configuration are extracted, including: The subscription registration center extracts the database name, table name, and field name related to the change as the configuration mark according to the update information of the target application configuration, and determines the operation type.

8. The method for adaptively updating application configuration cache based on configuration subscription and incremental parsing according to claim 7, characterized in that: Match the target business's application service according to the subscription registration information and configuration flags configured by the target application, and synchronize the update information to the application service cache according to the operation type, including: Based on the configuration scope that the application service is interested in in the subscription registration information, the database name, table name, and field name related to the extracted changes are matched to extract the target application service to which the changes need to be pushed; After receiving the change notification, the target application service synchronizes the change to the cache according to the change operation type.

9. The method for adaptively updating application configuration cache based on configuration subscription and incremental parsing as claimed in claim 8, characterized in that: Synchronizing changes to the cache based on the type of change operation includes: When the change operation type is update, the value of the corresponding configuration item in the cache is updated to the new value; When the change operation type is new, add new configuration content to the cache; When the change operation type is delete, the corresponding configuration item is deleted from the cache.

10. An application configuration cache adaptive update system based on configuration subscription and incremental parsing, characterized in that: The system adopts the method for adaptively updating application configuration cache based on configuration subscription and incremental parsing according to any one of claims 1 to 9, and the system comprises: The application service module includes a cache unit and a service processing unit, wherein the cache unit is used to store application configurations, and the service processing unit is used to process services and read and update the configurations in the cache when necessary; The subscription registration center module is used to receive subscription information of application services, receive log files from the database and extract change information, match application services that need to be pushed changes according to subscription information, and push configuration change notifications to application services; The database module is used to store application configuration data and generate log files to record changes when data is updated.

Images