Method, device and equipment for optimizing data service by using cache mechanism

Abstract

The invention discloses a method, a device and equipment for optimizing data service by using a cache mechanism, which are applied to external nodes in a cluster system and comprise the following steps: acquiring cluster data from a first API (Application Program Interface) of a monitoring control center by using a Web application framework program, the cluster data being data of a plurality of service nodes collected by the monitoring control center; caching the acquired cluster data into a cache storage system, and persistently storing at least part of cached cluster data into a persistent storage system when a persistent storage condition is met; and providing a second API interface for data access service by utilizing a Web application framework program, interacting with a front end through the second API interface, and obtaining cluster data from the cache storage system or the persistent storage system to provide HTTP service for external access. Therefore, the problem that due to the existence of constraint conditions such as existing data dimensions and data time spans, organization of data of service nodes in a cluster system becomes lower in efficiency is solved.

Description

Background Art

[0002] In most cluster business scenarios, it is necessary to monitor the nodes of the cluster in real time. The mainstream implementation solution is to adopt a node service reporting mechanism, that is, the node runs an externally accessible service that collects information about the system's operating status and exposes the data to the service interface. There is a special node in the cluster responsible for data aggregation. This special node collects and organizes the reporting information of all nodes, and exposes the standardized and normalized data to the service interface again.

[0003] In the view part that provides monitoring data content, such as the administrator's console and the customer's resource monitoring console, the data often needs to be very real-time, so it is necessary to increase the frequency of access to the Prometheus service. In addition, due to the existence of constraints such as data dimensions and data time spans, the organization of monitoring data becomes more inefficient. Summary of the invention

[0004] The purpose of this application is to provide a method, device and apparatus for optimizing data services using a cache mechanism, which is used to solve the problem that the organization of data of service nodes in a cluster system becomes more inefficient due to the existence of constraints such as existing data dimensions and data time spans.

[0005] In a first aspect, an embodiment of the present application provides a method for optimizing data services using a cache mechanism, which is applied to an external node in a cluster system, wherein the cluster system includes multiple service nodes, a central node, and an external node, and the method includes:

[0006] Using a Web application framework program to obtain cluster data from a first API interface of the monitoring and control center, the cluster data being data of the multiple service nodes collected by the monitoring and control center;

[0007] Utilize the Web application framework program to cache the acquired cluster data in the cache storage system, and when the persistent storage condition is met, persistently store at least part of the cached cluster data in the persistent storage system;

[0008] A second API interface for data access service is provided by using a Web application framework program, and the second API interface interacts with the front end to obtain cluster data from the cache storage system or the persistent storage system to provide an HTTP service for external access.

[0009] The method for optimizing data services using a cache mechanism provided in the embodiment of the present application can obtain data from a service node through a Web application framework program on an external node and an API connection with a monitoring control center. By introducing a cache layer, i.e., a cache storage system, the data is immediately cached in the cache storage system, thereby reducing direct access to the service Prometheus-exporter of the service node, thereby reducing its load, and also improving the response speed of the client to obtain data by providing a fast data access interface. Data can be stored in a corresponding persistent storage system through a dynamic persistence strategy without affecting real-time data flow tasks.

[0010] In some possible embodiments, the Web application framework program belongs to a backend service program written in Python code in the cluster system;

[0011] The backend service program also includes an object document mapper, which is used to map the monitoring data in the document form in the cache storage system into database objects and perform persistent storage.

[0012] The Web application framework program of this embodiment serves as the backend service of the cluster system. It can interact with the frontend on the one hand and with the monitoring and control center on the other hand. The backend service program also includes an object document mapper for persistently storing cached data, which can realize persistent storage of cached data and reduce storage pressure.

[0013] In some possible embodiments, using a Web application framework program to cache the acquired cluster data to a cache storage system includes:

[0014] The acquired cluster data is formatted using a Web application framework program, and the formatted cluster data is cached in a cache storage system according to a predetermined data arrangement logic.

[0015] When the embodiment of the present application utilizes the Web application framework program to cache cluster data, format processing is performed and the cluster data is arranged according to corresponding logic to better render and display the cluster data on the front end.

[0016] In some possible embodiments, when the persistent storage condition is met, persistently storing at least part of the cached cluster data in a persistent storage system includes:

[0017] Statistics on the frequency of HTTP services accessing cached cluster data from the cache storage system for external access in the recent period;

[0018] Target cluster data whose access frequency is lower than a set frequency threshold is determined, and the target cluster data is persistently stored in a persistent storage system.

[0019] The embodiment of the present application can persistently store cluster data with low access frequency within a period of time, on the one hand to reduce the cache storage pressure of the cache storage system, on the other hand, the frequently accessed cluster data is still in the cache storage system, which can efficiently achieve rapid response to external requests for cluster data access.

[0020] In some possible embodiments, interacting with the front end through the second API interface to obtain cluster data from the cache storage system or the persistent storage system to provide an HTTP service for external access includes:

[0021] Receiving a real-time data access request sent by the front end through a Web page through the second API interface, and obtaining requested cluster data from the cache storage system according to the real-time data access request;

[0022] Receiving a non-real-time data access request sent by the front end through a Web page through the second API interface, and obtaining the requested cluster data from the persistent storage system according to the non-real-time data access request;

[0023] The acquired cluster data is sent to a front end through a socket, and is displayed through a Web page of the front end.

[0024] The embodiment of the present application stores cluster data in two ways, so as to quickly respond to real-time data access requests and ensure normal access to non-real-time data.

[0025] In some possible embodiments, when the persistent storage condition is met, persistently storing at least part of the cached cluster data in a persistent storage system includes:

[0026] Periodically determine whether the persistent storage condition is met through a scheduled task, and when the persistent storage condition is met, persistently store at least part of the cached cluster data in the persistent storage system; or

[0027] Through the distributed publish-subscribe messaging system Apache Kafka, when the subscription meets the persistent storage conditions, it notifies the execution to persistently store at least part of the cached cluster data in the persistent storage system in the form of a published message.

[0028] The embodiments of the present application can persistently store cached cluster data by means of scheduled tasks or by publishing messages to satisfy persistent storage, and can efficiently and regularly synchronize cluster data, thereby achieving persistent storage of cluster data and ensuring consistency and integrity of cluster data.

[0029] In some possible embodiments, the persistent storage system includes any one of the following: a Cassandra storage system, a PostgreSQL storage system, or a MongoDB storage system;

[0030] The cache storage system includes any one of the following: a Memcached cache storage system, a Redis cache storage system.

[0031] The embodiments of the present application can implement caching of cluster data through the above-mentioned possible cache storage system, and can implement persistent storage of cluster data through the above-mentioned possible persistent storage system.

[0032] In a second aspect, an embodiment of the present application provides a device for optimizing data services using a cache mechanism, the device serving as an external node in a cluster system, the cluster system comprising a plurality of service nodes, a central node, and an external node, the device comprising:

[0033] A cluster data acquisition module, used to acquire cluster data from the first API interface of the monitoring and control center using a Web application framework program, wherein the cluster data is data of the multiple service nodes collected by the monitoring and control center;

[0034] The cluster data storage module is used to cache the acquired cluster data into the cache storage system by using the Web application framework program, and when the persistent storage condition is met, persistently store at least part of the cached cluster data into the persistent storage system;

[0035] The external access service module is used to use the Web application framework program to provide a second API interface for data access services, interact with the front end through the second API interface, obtain cluster data from the cache storage system or the persistent storage system to provide HTTP services for external access.

[0036] In the third aspect, another embodiment of the present application also provides a device for optimizing data services using a caching mechanism, comprising at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute any method for optimizing data services using a caching mechanism provided in an embodiment of the present application.

[0037] In a fourth aspect, another embodiment of the present application further provides a computer storage medium, wherein the computer storage medium stores a computer program, and the computer program is used to enable a computer to execute any method for optimizing data services using a caching mechanism provided in an embodiment of the present application.

[0038] Other features and advantages of the present application will be described in the following description, and partly become apparent from the description, or be understood by practicing the present application. The purpose and other advantages of the present application can be realized and obtained by the structures specifically pointed out in the written description, claims, and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments of the present application will be briefly introduced below. Obviously, the drawings introduced below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0040] Figure 1 A schematic diagram of a system for optimizing data services using a cache mechanism provided in an embodiment of the present application;

[0041] Figure 2 A flow chart of a method for optimizing data services using a cache mechanism provided in an embodiment of the present application;

[0042] Figure 3 A system framework diagram of a cluster system provided in an embodiment of the present application;

[0043] Figure 4 A detailed flowchart of a method for optimizing data services using a cache mechanism provided in an embodiment of the present application;

[0044] Figure 5 A structural diagram of a device for optimizing data services using a cache mechanism provided in an embodiment of the present application;

[0045] Figure 6 A diagram of the device structure for optimizing data services using a cache mechanism provided in an embodiment of the present application. DETAILED DESCRIPTION

[0046] To further illustrate the technical solution provided by the embodiment of the present application, this is described in detail below in conjunction with the accompanying drawings and specific implementation methods. Although the embodiment of the present application provides the method operation steps as shown in the following embodiments or drawings, more or fewer operation steps may be included in the method based on routine or no creative labor. In the steps where there is no necessary causal relationship logically, the execution order of these steps is not limited to the execution order provided by the embodiment of the present application. The method can be executed in the order of the method shown in the embodiment or drawings or in parallel during the actual processing process or when the control device is executed.

[0047] In the related technology, the cluster system includes a monitoring and control center and a service node. The service node runs an externally accessible service that collects cluster data of the system operation status and exposes the cluster data to the service interface. The monitoring and control center collects and organizes the cluster data reported by all service nodes and exposes it to the service interface of the monitoring and control center. The main implementation case of this type of solution is Prometheus. The service in the service node in this solution is Prometheus-exporter, and the service in the monitoring and control center in this solution is Prometheus-operator.

[0048] In the front-end that provides a view of the monitoring data content (such as the administrator's console, the customer's resource monitoring console, etc.), the data often needs to be very real-time, so it is necessary to increase the frequency of access to Prometheus. In the solution of providing data services to the outside world in related technologies, the front-end used for data access directly obtains monitoring data through the API of the service node Prometheus-exporter. This method is simple and direct, but as the amount of data increases, the server where the service node is located needs to respond frequently to data access requests, and the server load will increase significantly, affecting its performance. In addition, the solution for providing cluster data services to the outside world in related technologies lacks automated expansion and management mechanisms, and it is difficult to adapt to dynamically changing loads. The reasons for these problems may include:

[0049] Technical limitations: Prometheus was not designed with extreme scalability requirements in mind;

[0050] Resource limitation: Redis is an in-memory database and is limited by the size of physical memory;

[0051] Design deficiencies: Data persistence and synchronization efficiency were not fully considered during system design;

[0052] Managing complexity: Lack of effective tools and processes to manage complex system architectures.

[0053] The following is an explanation and description of the technical terms related to this application:

[0054] 1) Redis cache storage system

[0055] Redis is an open source in-memory data structure storage system that can be used as a database, cache, and message broker. It supports multiple types of data structures, such as strings, hashes, lists, sets, sorted sets, etc. The Redis cache storage system is known for its excellent performance and low latency, and is suitable for scenarios that require fast access.

[0056] 2) MongoDB storage system

[0057] MongoDB is a database based on distributed file storage. It uses document storage and stores data in JSON-style BSON. It has a flexible data model, high performance, and rich query language. MongoDB uses collections to organize data, and each collection stores multiple documents. It supports replication and fault recovery, and can also process large-scale data sets through sharding. MongoDB's query language is rich and flexible, and supports indexes to improve query performance. Due to these features, MongoDB is widely used in scenarios such as big data processing, real-time analysis, and Web applications.

[0058] 3) Prometheus-operator

[0059] Prometheus is an open source system monitoring and alerting toolkit, commonly used for the collection and processing of time series data. It helps users monitor and analyze the performance and status of the system by capturing and storing metrics. Prometheus Operator is a collection of K8s applications that provides Kubernetes native deployment and management of Prometheus and related monitoring components. It mainly includes the following functions: K8s custom resources; simplified deployment configuration: such as version, persistence, retention policy and replicas; automatic generation of monitoring target configuration.

[0060] 4) Flask

[0061] Flask is a lightweight web application framework written in Python. It is easy to use, flexible, and extensible, suitable for rapid development of small to medium-sized web applications. Flask provides basic functions such as routing, error handling, template rendering, database integration, and more functions can be added through plug-ins and extensions.

[0062] 5) K8s

[0063] Kubernetes is an open source container orchestration platform for automatically deploying, scaling, and managing containerized applications. Its abbreviation "K8s" comes from the letter "K" in the word "Kubernetes" and the eight letters in "ubernete". Kubernetes allows you to run and manage containers in a cluster, supports multiple container runtimes, and provides features such as self-healing, service discovery, and load balancing.

[0064] Other features and advantages of the present application will be described in the following description, and partly become apparent from the description, or be understood by practicing the present application. The purpose and other advantages of the present application can be realized and obtained by the structures specifically pointed out in the written description, claims, and drawings.

[0065] The existence of constraints such as data dimension and data time span in the related technology makes the organization of cluster data more inefficient. In order to solve this problem, the existing related solutions propose that the front end directly obtains cluster data through the API of the service node Prometheus-exporter. However, as the amount of data increases, the server where the service node is located needs to frequently respond to data acquisition requests, and the server load will increase significantly, affecting its performance. In view of the above problems in the related technology, the embodiments of the present application provide a method, device and equipment for optimizing data services using a caching mechanism, such as Figure 1 The application scenario of the embodiment of the present application is shown, which is mainly applied to cluster systems. The cluster system of the embodiment of the present application that uses a cache mechanism to optimize data services includes multiple service nodes, a central node and an external node, wherein the service node runs a service that can be accessed externally, and the service collects cluster data of the system operation status and exposes the cluster data to the service interface; the monitoring and control center collects and organizes the cluster data reported by all service nodes, and exposes it to the service interface of the monitoring and control center again, and the external node obtains the cluster data of the multiple service nodes collected by it from the monitoring and control center. The external node can be an independent server as the back-end service of the system, or it can be integrated with the monitoring and control center on a server. The embodiment of the present application realizes the acquisition and storage control of cluster data on the external node by utilizing a Web application framework program, and provides an API interface for data access services, interacts with the front end through the API interface, obtains cluster data to provide HTTP services for external access.

[0066] The following is a detailed description of the method for optimizing data services using a cache mechanism in an embodiment of the present application in conjunction with the accompanying drawings.

[0067] like Figure 2 As shown, an embodiment of the present application provides a method for optimizing data services using a cache mechanism, which is applied to an external node in a cluster system, wherein the cluster system includes multiple service nodes, a central node, and an external node, and the method includes:

[0068] Step 201, using a Web application framework program to obtain cluster data from a first API interface of the monitoring and control center, wherein the cluster data is data of the multiple service nodes collected by the monitoring and control center;

[0069] As mentioned above, the service node runs a service Prometheus-exporter that can be accessed externally. The service Prometheus-exporter collects cluster data of the system operation status and exposes the cluster data to the service interface API; the service Prometheus-operator of the monitoring and control center collects and organizes the cluster data reported by all service nodes, and exposes it to the service interface API of the monitoring and control center again. The external node uses the Web application framework program to obtain cluster data from the first API interface of the monitoring and control center. The above-mentioned Web application framework program can be but not limited to Flask. Flask is a lightweight Web application framework written in Python. It is easy to use, flexible, and scalable, suitable for rapid development of small to medium-sized Web applications. Flask provides basic functions such as routing, error handling, template rendering, database integration, and more functions can be added through plug-ins and extensions. Therefore, in the embodiment of the present application, the external node, as a Flask server, obtains the cluster data of all service nodes collected by the monitoring and control center from the API interface of the service Prometheus-operator of the monitoring and control center.

[0070] Step 202: Cache the acquired cluster data into a cache storage system by using the Web application framework program, and when the persistent storage condition is met, persistently store at least part of the cached cluster data into the persistent storage system;

[0071] In order to improve the data access speed, the Web application framework program is used in the embodiment of the present application to store the acquired data in the cache storage system. This method can significantly reduce the number of direct accesses to the service Prometheus-exporter, and when the persistent storage conditions are met, part of the cluster data is persistently stored in the persistent storage system, solving the persistent storage problem of cached data.

[0072] The cache storage system in the embodiment of the present application includes any of the following: a Memcached cache storage system, a Redis cache storage system.

[0073] Memcached is a free, open-source, high-performance distributed memory object caching system. Memcached is simple and powerful. Its simple design facilitates rapid development, reduces development difficulty, and solves many problems of large data cache. Its API is compatible with most popular development languages. In essence, it is a simple key-value storage system. The general purpose of use is to cache database query results and reduce the number of database accesses to increase the speed and scalability of dynamic Web applications.

[0074] Redis is an open source in-memory data structure storage system that can be used as a database, cache, and message broker. It supports multiple types of data structures, such as strings, hashes, lists, sets, sorted sets, etc. The Redis cache storage system is known for its excellent performance and low latency, and is suitable for scenarios that require fast access.

[0075] In view of the characteristics of the above two cache storage systems, the external node uses Flask to obtain the data from the API interface of the Prometheus-operator service in the monitoring control center, and caches it in real time into the above cache storage system, reducing the direct access to the service Prometheus-exporter of the service node, thereby reducing its load and enabling quick response to access requests.

[0076] In order to reduce the pressure on the cache storage system, the embodiment of the present application stores at least part of the cached cluster data persistently in the persistent storage system when the persistent storage conditions are met. The specific principle can be to retain the cluster data that is dynamically accessed in real time in the cache storage system and store the cluster data that is not accessed for a long time in the persistent storage system.

[0077] Step 203: Use the Web application framework program to provide a second API interface for data access service, interact with the front end through the second API interface, obtain cluster data from the cache storage system or the persistent storage system to provide HTTP service for external access.

[0078] The method for optimizing data services using a cache mechanism provided in the embodiment of the present application can obtain data of a service node through an API connection between a Web application framework program on an external node and a monitoring and control center, and timely cache the data in the cache storage system by introducing a cache layer, i.e., a cache storage system, thereby reducing direct access to the service Prometheus-exporter of the service node, thereby reducing its load, and also improving the response speed of the client to obtain cluster data by providing a fast data access interface. Cluster data can be stored in a corresponding persistent storage system through a dynamic persistence strategy without affecting real-time data flow tasks. The embodiment of the present application has been innovated on the basis of the prior art, providing a more efficient, reliable and scalable data caching and persistence solution.

[0079] In some possible embodiments, the persistent storage system may be, but is not limited to, a relational database or a NoSQL database, and may specifically include any of the following: a Cassandra storage system, a PostgreSQL storage system, or a MongoDB storage system;

[0080] Cassandra is a hybrid non-relational database. The main feature of Cassandra is that it is not a database, but a distributed network service composed of a bunch of database nodes. A write operation to Cassandra will be replicated to other nodes, and a read operation to Cassandra will also be routed to a certain node for reading. For a Cassandra cluster, it is relatively simple to expand performance. Just add nodes to the cluster.

[0081] PostgreSQL is a free object-relational database server (database management system). The core of the PostgreSQL installation is the database server process. It allows running on a standalone server. Applications that need to access data stored in the database must go through the database process. These client programs cannot access the data directly, even if they are on the same machine as the server program.

[0082] The MongoDB storage system is a database based on distributed file storage. It uses document storage and stores data in JSON-style BSON. It has a flexible data model, high performance, and rich query language. MongoDB uses collections to organize data, and each collection stores multiple documents. It supports replication and fault recovery, and can also process large-scale data sets through sharding. MongoDB's query language is rich and flexible, and supports indexes to improve query performance. Due to these features, MongoDB is widely used in scenarios such as big data processing, real-time analysis, and Web applications.

[0083] In some possible embodiments, the Web application framework program belongs to a backend service program written in Python code in the cluster system; the backend service program also includes an object document mapper, which is used to map monitoring data in document form in the cache storage system into database objects and perform persistent storage.

[0084] The following takes the cluster system running on the Linux distribution system of the x86 architecture as an example to illustrate the system framework of the cluster system in the embodiment of the present application. Figure 3 The figure shows the system framework diagram of the cluster system, which mainly includes: presentation layer, communication layer, middleware, storage layer and system layer.

[0085] The system layer mainly includes K8s and Linux operating system (Linux OS). K8s provides a container environment. Each service runs in a container, exposing service interfaces and file systems involved in data flow.

[0086] The storage layer includes the persistent storage system MongoDB and Redis cache. For the cluster data of the service nodes, one possible method is to store the monitoring data and other data required for system operation in MongoDB, and the real-time data and other dynamically changing data in the running Redis cache.

[0087] For middleware, it includes backend services and K8s management programs. The backend services are mainly composed of multiple functions and subroutines written in Python code. In order to achieve interaction with other components and services, corresponding middleware needs to be introduced. For example, when Python programs connect to MongoDB, they need to use the object document mapper MongoEngine middleware; when Python programs need to provide HTTP services, they need to use the plug-in Flask-restx in Flask; for K8s management programs, K8s systems need to provide external resource access API services, and they need to use K8s API Server, that is, the API service of external nodes; K8s nodes need to provide external cluster data reporting API services, and they need to use Prometheus, that is, the service Prometheus-operator for monitoring nodes.

[0088] For the communication layer, it includes HTTP / HTTPS, Nginx, and Web Socket, which are used to implement the Rsetful API interface communication with the presentation layer using different communication methods.

[0089] For the presentation layer, it includes Web (HTML) and Rsetful API interfaces. It communicates with the communication layer through the Rsetful API interface and implements Web services through Web (HTML).

[0090] The process implemented in the embodiment of the present application based on the above system framework is that the monitoring control center uses the service Prometheus-operator to obtain the cluster data of the service node; Flask-restx obtains the cluster data collected by Prometheus and caches it in the Redis cache; when it is determined that the persistent storage conditions are met, MongoEngine is used to persistently store part of the cluster data in the Redis cache in MongoDB; the front end sends a data access request through the Web (HTML) to the K8s API Server via the communication layer, and transmits it to Flask-restx; Flask-restx determines that the requested cluster data is in the Redis cache or in MongoDB, obtains the requested data from the Redis cache or from MongoDB through MongoEngine, transmits it to the Rsetful API interface through Web Socket, and finally displays it on the front-end Web page.

[0091] In some possible embodiments, using a Web application framework program to cache the acquired cluster data to a cache storage system includes:

[0092] The acquired cluster data is formatted using a Web application framework program, and the formatted cluster data is cached in a cache storage system according to a predetermined data arrangement logic.

[0093] In order to better interact with the front end and enable the front end to better display data, in an embodiment of the present application, the cluster data can be formatted, for example, organized into a fixed format, and the formatted cluster data is cached in the Redis cache according to a predetermined data arrangement logic. In this way, when the Web application framework program interacts with the front end, the arranged cluster data can be directly obtained from the Redis cache and sent to the Web page for rendering and display. One possible data arrangement logic is in the form of a table or a data graph.

[0094] In some possible embodiments, when the persistent storage condition is met, persistently storing at least part of the cached cluster data in a persistent storage system includes:

[0095] Statistics on the frequency of HTTP services accessing cached cluster data from the cache storage system for external access in the recent period;

[0096] Target cluster data whose access frequency is lower than a set frequency threshold is determined, and the target cluster data is persistently stored in a persistent storage system.

[0097] This embodiment uses the Web application framework program to provide a second API interface for data access services, interacts with the front end through the second API interface, obtains cluster data from the cache storage system or persistent storage system to provide HTTP services for external access. Therefore, the frequency of data in the Redis cache being accessed in the recent period can be counted. The duration of the recent period can be preset, and a longer time can be set specifically. Of course, the duration of the data in the Redis cache not being accessed can also be used. If the duration exceeds the preset duration, it is considered that the persistent storage conditions are met, and the cache data can be persistently stored to reduce pressure. Under normal circumstances, monitoring data and other data required for system operation are accessed less frequently, so they are stored in MongoDB; real-time data and other dynamically changing data are accessed more frequently, so they are all stored in the running Redis cache.

[0098] In some possible embodiments, interacting with the front end through the second API interface to obtain cluster data from the cache storage system or the persistent storage system to provide an HTTP service for external access includes:

[0099] Receiving a real-time data access request sent by the front end through a Web page through the second API interface, and obtaining requested cluster data from the cache storage system according to the real-time data access request;

[0100] Receiving a non-real-time data access request sent by the front end through a Web page through the second API interface, and obtaining the requested cluster data from the persistent storage system according to the non-real-time data access request;

[0101] The acquired cluster data is sent to a front end through a socket, and is displayed through a Web page of the front end.

[0102] Since the real-time data and other dynamically changing data in this application are accessed frequently, they are all stored in the running Redis cache. Therefore, when a real-time data access request is received, the requested cluster data is obtained from the Redis cache according to the real-time data access request. Since the data with a lower access frequency is stored in MongoDB, when a non-real-time data access request is received, the requested cluster data is obtained from MongoDB.

[0103] In order to store data for a long time, the data in the cache is periodically synchronized to persistent storage, such as a relational database or a NoSQL database. However, related solutions usually lack an automated and efficient data synchronization mechanism, and may encounter performance and consistency issues during the data synchronization process.

[0104] In some possible embodiments, when the persistent storage condition is met, persistently storing at least part of the cached cluster data in a persistent storage system includes:

[0105] Periodically determine whether the persistent storage condition is met through a scheduled task, and when the persistent storage condition is met, persistently store at least part of the cached cluster data in the persistent storage system; or

[0106] Through the distributed publish-subscribe messaging system Apache Kafka, when the subscription meets the persistent storage conditions, it notifies the execution to persistently store at least part of the cached cluster data in the persistent storage system in the form of a published message.

[0107] The embodiment of the present application adopts the method of scheduled tasks to efficiently and regularly synchronize the cluster data in the Redis cache to MongoDB, thereby realizing the persistent storage of data and ensuring the consistency and integrity of the data. In addition, the data in the Redis cache that meets the persistent storage conditions can be efficiently and regularly synchronized to MongoDB by publishing messages when the persistent storage conditions are met through the distributed publish-subscribe messaging system Apache Kafka.

[0108] By using the method for optimizing data services using a cache mechanism provided in the embodiments of the present application, the following mechanisms can be implemented:

[0109] 1) Data flow mechanism: The data of Prometheus-operator in the monitoring control center is obtained through the Flask API and cached in Redis. After being orchestrated and organized by Redis, it can be better distributed to the backend system and database as persistent data.

[0110] 2) Data synchronization strategy: The embodiment of the present application ensures that the cluster data cached by Redis and the cluster data between MongoDB can be synchronized without affecting real-time transactions through the design of scheduled task triggering logic and task execution process.

[0111] 3) Performance optimization mechanism: The cache layer increases the real-time and reliability of data flow, thereby improving the overall performance of the data collection and presentation system.

[0112] 4) Scalable design: The entire process of data collection, data caching, data presentation, and data persistence can control the window period and polling cycle through multiple parameters, providing better customized settings.

[0113] like Figure 4 The figure shows an overall flow chart of a method for optimizing data services using a cache mechanism provided by an embodiment of the present application, which mainly includes:

[0114] After the service node of the K8s system collects data, it reports the cluster data to the Prometheus-operator service of the monitoring and control center through the data exposure interface;

[0115] Prometheus-operator provides API services to the outside world, exposing cluster data through the API interface to provide cluster data of multiple service nodes to the Flask server. When the Flask server initiates a data acquisition request, the cluster data is sent to the Flask server. It can also actively send cluster data to the Flask server.

[0116] The Flask server formats the acquired cluster data and caches the cluster data into the Redis database according to certain data arrangement logic;

[0117] The Flask service provides an API interface to respond to external requests, that is, to provide an API interface to organize cluster data;

[0118] The Flask service writes the cluster data with low access frequency in the MongoDB database to the MongoDB database at regular intervals;

[0119] When the data is successfully stored in the MongoDB database, a data write success response is returned;

[0120] The Flask server provides HTTP services to external requests by responding to the API interface of external requests and displays the corresponding services on the web page;

[0121] The front end operates the service displayed on the web page and initiates a data access request to the Flask server, thereby calling the API interface to obtain cluster data. The Flask server obtains the orchestration cluster data from the Redis database and sends it to the front end in real time through the Socket; the front end receives the orchestration data and renders the chart.

[0122] The method for optimizing data services by using a cache mechanism provided by the embodiment of the present application has the following beneficial effects:

[0123] 1) Optimize the performance of cluster data flow: Use cache as middleware to provide a high-speed and stable carrier for data flow, providing higher reliability and real-time performance for the system; collect data from Prometheus through the Flask server and cache it in the Redis database in real time, reducing direct access to Prometheus and thus reducing its load.

[0124] 2) Efficient data persistence: Data can be stored in the corresponding database system through dynamic persistence strategies without affecting the real-time data flow tasks; by adopting the method of scheduled tasks, the data in Redis can be efficiently and regularly synchronized to MongoDB, realizing the persistent storage of data while ensuring the consistency and integrity of the data.

[0125] 3) A faster way to obtain monitoring data sources: Compared with the existing solutions that obtain and parse monitoring data from each monitoring instance in real time, the embodiment of the present application obtains data through the Flask server accessing the API of the Prometheus-operator service of the monitoring control center. The data is persisted and formatted to ensure validity and reliability. Moreover, as a separate service, it can provide a higher network throughput than the monitoring instance, greatly improving the speed of data acquisition; it allows the resources of the server, cache layer, and persistence layer to be easily adjusted according to business needs to adapt to different workloads.

[0126] 4) Improved data reliability and system stability: Using mature caching technology such as Redis has higher reliability than previous HTTP / HTTPS transmission. By introducing a cache layer, it not only reduces the load on Prometheus, but also improves the response speed of the client to obtain data by providing a fast data access interface.

[0127] Based on the same inventive concept, the present application also provides a device for optimizing data services using a cache mechanism, the device being used as an external node in a cluster system, the cluster system comprising a plurality of service nodes, a central node and external nodes, such as Figure 5 As shown, the device comprises:

[0128] The cluster data acquisition module 501 is used to acquire cluster data from the first API interface of the monitoring and control center by using a Web application framework program, wherein the cluster data is data of the multiple service nodes collected by the monitoring and control center;

[0129] The cluster data storage module 502 is used to cache the acquired cluster data into the cache storage system by using the Web application framework program, and when the persistent storage condition is met, persistently store at least part of the cached cluster data into the persistent storage system;

[0130] The external access service module 503 is used to use the Web application framework program to provide a second API interface for data access service, interact with the front end through the second API interface, obtain cluster data from the cache storage system or the persistent storage system to provide HTTP service for external access.

[0131] In some possible embodiments, the Web application framework program belongs to a backend service program written in Python code in the cluster system;

[0132] The backend service program also includes an object document mapper, which is used to map the monitoring data in the document form in the cache storage system into database objects and perform persistent storage.

[0133] In some possible embodiments, the cluster data storage module uses the Web application framework program to cache the acquired cluster data to the cache storage system, including:

[0134] The acquired cluster data is formatted using a Web application framework program, and the formatted cluster data is cached in a cache storage system according to a predetermined data arrangement logic.

[0135] In some possible embodiments, when the cluster data storage module meets the persistent storage condition, persistently storing at least part of the cached cluster data in the persistent storage system includes:

[0136] Statistics on the frequency of HTTP services accessing cached cluster data from the cache storage system for external access in the recent period;

[0137] Target cluster data whose access frequency is lower than a set frequency threshold is determined, and the target cluster data is persistently stored in a persistent storage system.

[0138] In some possible embodiments, the external access service module interacts with the front end through the second API interface, obtains cluster data from the cache storage system or the persistent storage system to provide an HTTP service for external access, including:

[0139] Receiving a real-time data access request sent by the front end through a Web page through the second API interface, and obtaining requested cluster data from the cache storage system according to the real-time data access request;

[0140] Receiving a non-real-time data access request sent by the front end through a Web page through the second API interface, and obtaining the requested cluster data from the persistent storage system according to the non-real-time data access request;

[0141] The acquired cluster data is sent to a front end through a socket, and is displayed through a Web page of the front end.

[0142] In some possible embodiments, when the cluster data storage module meets the persistent storage condition, persistently storing at least part of the cached cluster data in the persistent storage system includes:

[0143] Periodically determine whether the persistent storage condition is met through a scheduled task, and when the persistent storage condition is met, persistently store at least part of the cached cluster data in the persistent storage system; or

[0144] Through the distributed publish-subscribe messaging system Apache Kafka, when the subscription meets the persistent storage conditions, it notifies the execution to persistently store at least part of the cached cluster data in the persistent storage system in the form of a published message.

[0145] In some possible embodiments, the persistent storage system includes any one of the following: a Cassandra storage system, a PostgreSQL storage system, or a MongoDB storage system;

[0146] The cache storage system includes any one of the following: a Memcached cache storage system, a Redis cache storage system.

[0147] After introducing the method and apparatus for optimizing data services by using a cache mechanism according to an exemplary embodiment of the present application, next, an apparatus for optimizing data services by using a cache mechanism according to another exemplary embodiment of the present application is introduced.

[0148] Those skilled in the art will appreciate that various aspects of the present application may be implemented as a system, method or program product. Therefore, various aspects of the present application may be specifically implemented in the following forms, namely: a complete hardware implementation, a complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software, which may be collectively referred to as "circuit", "module" or "system" herein.

[0149] In some possible implementations, the device for optimizing data services using a cache mechanism according to the present application may include at least one processor and at least one memory. The memory stores program code, and when the program code is executed by the processor, the processor executes the steps of the method for optimizing data services using a cache mechanism according to various exemplary implementations of the present application described above in this specification.

[0150] Refer to the following Figure 6 The optimization of the data service device 160 using the cache mechanism according to this embodiment of the present application is described. Figure 6 The device 160 for optimizing data services using a cache mechanism is shown as merely an example and should not limit the functions and scope of use of the embodiments of the present application.

[0151] like Figure 6As shown, the device 160 for optimizing data services using a cache mechanism is presented in the form of a general electronic device. The components of the device 160 for optimizing data services using a cache mechanism may include, but are not limited to: at least one processor 161, at least one memory 162, and a bus 163 connecting different system components (including the memory 162 and the processor 161).

[0152] Bus 163 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, a processor, or a local bus using any of a variety of bus architectures.

[0153] The memory 162 may include a readable medium in the form of a volatile memory, such as a random access memory (RAM) 1621 and / or a cache memory 1622 , and may further include a read-only memory (ROM) 1623 .

[0154] Memory 162 may also include a program / utility 1625 having a set (at least one) of program modules 1624, such program modules 1624 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0155] The data service device 160 using the cache mechanism optimization can also communicate with one or more external devices 164 (e.g., keyboards, pointing devices, etc.), and can also communicate with one or more devices that enable users to interact with the data service device 160 using the cache mechanism optimization, and / or communicate with any device that enables the data service device 160 using the cache mechanism optimization to communicate with one or more other electronic devices (e.g., routers, modems, etc.). Such communication can be performed through an input / output (I / O) interface 165. In addition, the data service device 160 using the cache mechanism optimization can also communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs) and / or public networks, such as the Internet) through a network adapter 166. As shown in the figure, the network adapter 166 communicates with other modules of the data service device 160 using the cache mechanism optimization through a bus 163. It should be understood that, although not shown in the figure, other hardware and / or software modules can be used in conjunction with the data service device 160 using the cache mechanism optimization, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0156] In some possible implementations, various aspects of a method for optimizing data services using a caching mechanism provided by the present application may also be implemented in the form of a program product, which includes program code. When the program product is run on a computer device, the program code is used to enable the computer device to execute the steps of a method for optimizing data services using a caching mechanism according to various exemplary implementations of the present application described above in this specification.

[0157] The program product may employ any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0158] The program product for optimizing data services using a cache mechanism of the embodiment of the present application can adopt a portable compact disk read-only memory (CD-ROM) and include program code, and can be run on an electronic device. However, the program product of the present application is not limited thereto. In this document, a readable storage medium can be any tangible medium containing or storing a program, which can be used by or in combination with an instruction execution system, an apparatus or a device.

[0159] The readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, wherein the readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The readable signal medium may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0160] The program code embodied on the readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the foregoing.

[0161] The program code for performing the operations of the present application can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and also conventional procedural programming languages ​​such as "C" language or similar programming languages. The program code can be executed entirely on the user electronic device, partially on the user device, as an independent software package, partially on the user electronic device and partially on the remote electronic device, or entirely on the remote electronic device or server. In the case of a remote electronic device, the remote electronic device can be connected to the user electronic device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external electronic device (for example, using an Internet service provider to connect through the Internet).

[0162] It should be noted that, although several units or subunits of the device are mentioned in the above detailed description, this division is merely exemplary and not mandatory. In fact, according to the embodiments of the present application, the features and functions of two or more units described above can be embodied in one unit. Conversely, the features and functions of one unit described above can be further divided into multiple units to be embodied.

[0163] In addition, although the operations of the method of the present application are described in a specific order in the drawings, this does not require or imply that the operations must be performed in this specific order, or that all the operations shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps.

[0164] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0165] The present application is described with reference to the flowcharts and block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and block diagram, as well as the combination of the processes and boxes in the flowchart and block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and block diagram. Figure 1 Process or multiple processes and boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0166] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 Process or multiple processes and boxes Figure 1 A function specified in one or more boxes.

[0167] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 Process or multiple processes and boxes Figure 1 A step that specifies a function in one or more boxes.

[0168] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of the present application.

[0169] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A method for optimizing data services using a cache mechanism, applied to an external node in a cluster system, wherein the cluster system includes a plurality of service nodes, a central node, and an external node, characterized in that: The method comprises: Using a Web application framework program to obtain cluster data from a first API interface of the monitoring and control center, the cluster data being data of the multiple service nodes collected by the monitoring and control center; Utilize the Web application framework program to cache the acquired cluster data in the cache storage system, and when the persistent storage condition is met, persistently store at least part of the cached cluster data in the persistent storage system; A second API interface for data access service is provided by using a Web application framework program, and the second API interface interacts with the front end to obtain cluster data from the cache storage system or the persistent storage system to provide an HTTP service for external access.

2. The method according to claim 1, characterized in that: The Web application framework program belongs to the backend service program written in Python code in the cluster system; The backend service program also includes an object document mapper, which is used to map the monitoring data in the document form in the cache storage system into database objects and perform persistent storage.

3. The method according to claim 1, characterized in that Use the Web application framework program to cache the acquired cluster data into the cache storage system, including: The acquired cluster data is formatted using a Web application framework program, and the formatted cluster data is cached in a cache storage system according to a predetermined data arrangement logic.

4. The method according to claim 1, characterized in that: When the persistent storage conditions are met, at least part of the cached cluster data is persistently stored in the persistent storage system, including: Statistics on the frequency of HTTP services accessing cached cluster data from the cache storage system for external access in the recent period; Target cluster data whose access frequency is lower than a set frequency threshold is determined, and the target cluster data is persistently stored in a persistent storage system.

5. The method according to claim 1, characterized in that: Interacting with the front end through the second API interface, obtaining cluster data from the cache storage system or the persistent storage system to provide an HTTP service for external access, including: Receiving a real-time data access request sent by the front end through a Web page through the second API interface, and obtaining requested cluster data from the cache storage system according to the real-time data access request; Receiving a non-real-time data access request sent by the front end through a Web page through the second API interface, and obtaining the requested cluster data from the persistent storage system according to the non-real-time data access request; The acquired cluster data is sent to a front end through a socket, and is displayed through a Web page of the front end.

6. The method according to claim 1, characterized in that When the persistent storage conditions are met, at least part of the cached cluster data is persistently stored in the persistent storage system, including: Periodically determine whether the persistent storage condition is met through a scheduled task, and when the persistent storage condition is met, persistently store at least part of the cached cluster data in the persistent storage system; or Through the distributed publish-subscribe messaging system Apache Kafka, when the subscription meets the persistent storage conditions, it notifies the execution to persistently store at least part of the cached cluster data in the persistent storage system in the form of a published message.

7. The method according to claim 1, characterized in that The persistent storage system includes any one of the following: Cassandra storage system, PostgreSQL storage system, MongoDB storage system; The cache storage system includes any one of the following: a Memcached cache storage system, a Redis cache storage system.

8. A device for optimizing data services using a cache mechanism, the device being used as an external node in a cluster system, the cluster system comprising a plurality of service nodes, a central node and external nodes, characterized in that: The device comprises: A cluster data acquisition module, used to acquire cluster data from the first API interface of the monitoring and control center using a Web application framework program, wherein the cluster data is data of the multiple service nodes collected by the monitoring and control center; The cluster data storage module is used to cache the acquired cluster data into the cache storage system by using the Web application framework program, and when the persistent storage condition is met, persistently store at least part of the cached cluster data into the persistent storage system; The external access service module is used to use the Web application framework program to provide a second API interface for data access services, interact with the front end through the second API interface, obtain cluster data from the cache storage system or the persistent storage system to provide HTTP services for external access.

9. A device for optimizing data services using a cache mechanism, characterized in that: It comprises at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method as described in any one of claims 1-7.

10. A computer storage medium, characterized in that: The computer storage medium stores a computer program, and the computer program is used to enable a computer to execute the method according to any one of claims 1 to 7.

Images