A message writing method, apparatus, device and computer program product

Abstract

The application provides a message writing method, device, equipment and computer program product; the method comprises the following steps: in the case that a consumer consumes a first message, acquiring first CMDB information corresponding to the first message; adding the first message and the first CMDB information to a first cache queue; the first cache queue is a cache queue corresponding to the type of the first message; in the case that first quantity information of the first message cached in the first cache queue is greater than a first value, writing the first message cached in the first cache queue into a database; the first value is determined according to the writing capability of the database. The rate of message writing can be improved.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a message writing method, apparatus, device, and computer program product. Background Technology

[0002] Given limited cluster resources, when massive amounts of heterogeneous, high-concurrency data generate write tasks, only by rationally allocating and rapidly executing these write tasks can the smooth writing of data be guaranteed. However, in related technologies, when writing massive amounts of data, producers encapsulate the massive data into messages, and consumers consume the encapsulated messages to write them to the database. But there are problems such as insufficient consumption capacity leading to data backlog and business interruption, resulting in a low message writing rate. Summary of the Invention

[0003] This application provides a message writing method, apparatus, device, and computer program product that can improve the message writing speed.

[0004] The technical solution of this application embodiment is implemented as follows: This application provides a message writing method, the method including: When consuming the first message on the consumer side, obtain the information of the first configuration management database (CMDB) corresponding to the first message; Add the first message and the first CMDB information to the first cache queue; the first cache queue is the cache queue corresponding to the type of the first message. If the first quantity of the first message cached in the first cache queue is greater than a first value, the first message cached in the first cache queue is written to the database; the first value is determined based on the write capability of the database.

[0005] This application provides a message writing device, including: The acquisition unit is used to acquire the first configuration management database (CMDB) information corresponding to the first message when the first message is consumed at the consumer end. An adding unit is used to add the first message and the first CMDB information to a first cache queue; the first cache queue is the cache queue corresponding to the type of the first message; The writing unit is used to write the first message cached in the first cache queue to the database when the first quantity information of the first message cached in the first cache queue is greater than a first value; the first value is determined according to the writing capability of the database.

[0006] This application provides an electronic device, the electronic component comprising: Memory is used to store executable instructions or computer programs. The processor, when executing computer-executable instructions or computer programs stored in the memory, implements the method provided in the embodiments of this application.

[0007] This application provides a computer program product, including a computer program or computer executable instructions. When the computer program or computer executable instructions are executed by a processor, they implement the message writing method provided in this application.

[0008] The embodiments of this application have the following beneficial effects: When consuming the first message at the consumer end, the first CMDB information corresponding to the first message is obtained, and the first message and the first CMDB information are added to the first cache queue simultaneously for subsequent query processing. The first cache queue is the cache queue corresponding to the type of the first message. If the first quantity information of the first message cached in the first cache queue is greater than the first value, the first message cached in the first cache queue is written to the database. That is, the message is classified according to its type, the first message is temporarily stored in the cache queue, and if the first quantity information is greater than the first value, the first message is written to the database simultaneously, thereby improving the message writing speed. Attached Figure Description

[0009] Figure 1 A flowchart illustrating a message writing method provided in an embodiment of this application; Figure 2 A schematic diagram of an exemplary high-performance data writing method system architecture provided in this application embodiment; Figure 3 A schematic diagram illustrating an exemplary monitoring object and its main functional indicators provided in this application embodiment; Figure 4 A schematic diagram illustrating an exemplary end-to-end latency for Kafka data writing, provided as an embodiment of this application; Figure 5 A flowchart illustrating an exemplary near real-time MQ update strategy provided in this application embodiment; Figure 6 A flowchart illustrating an exemplary timed update strategy provided in this application embodiment; Figure 7 A flowchart illustrating an exemplary data offloading and caching queue provided in this application embodiment; Figure 8 A schematic diagram illustrating an exemplary rate control provided for an embodiment of this application; Figure 9A schematic diagram of an exemplary message writing device provided in this application embodiment; Figure 10 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0010] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0011] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0012] In the following description, the terms "first, second, third" are used merely to distinguish similar objects and do not represent a specific ordering of objects. It is understood that "first, second, third" may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0013] Given limited cluster resources, when massive amounts of heterogeneous, high-concurrency data generate access tasks, only by rationally allocating and rapidly executing these tasks can the smooth access of data be guaranteed. Compared to traditional master-slave task allocation mechanisms, decentralized task allocation mechanisms offer better robustness. In reality, it is difficult to estimate the computational, communication, and data loads in advance. Therefore, dynamically adjusting the cluster's consumption capacity to ensure efficient utilization of cluster node resources and achieve reliable, high-concurrency access to massive amounts of data, meeting the operational and monitoring needs of large, medium, and small cloud data centers, is a crucial issue. Currently, the following solutions are available for these scenarios.

[0014] 1. Prometheus + Time Series Database: This method uses a specially designed time series database (such as InfluxDB, Graphite, etc.) to store and manage resource performance data. It is suitable for querying time series data, but may have commercial issues and long-term storage problems.

[0015] 2. RocketMQ + relational database: RocketMQ is used as a message queue to receive operation and maintenance data, and an exchange and multiple consumer queues are created. The exchange is used to configure different routing rules for each consumer queue and bind the correspondence between the routing rules and the consumer queues. After the consumer messages are processed, they are written to the relational database. However, the performance of writing and querying relational row data has become one of the bottlenecks in data processing.

[0016] 3. Message middleware + document database (MongoDB): Kafka is used as the aggregation node for distributed message data from different device access layers. The flexibility of the document data structure in MongoDB database collections is utilized to directly store semi-structured data, avoiding cumbersome data structure conversion and simplifying the data persistence process.

[0017] The above solutions suffer from high end-to-end latency, making it impossible to guarantee real-time time and achieve near real-time monitoring. When dealing with massive amounts of data (hundreds of millions) written to enterprise-level data centers, insufficient consumption capacity leads to data backlog and business interruptions. The solutions are limited to a single application scenario, failing to dynamically scale up or down consumption capabilities for different cluster sizes. They also cannot address the long-term storage of massive amounts of data in certain scenarios. Furthermore, inconsistent data sources and the lack of standardized operational data mean that operations personnel often lack data analysis skills, hindering their ability to deeply explore the value behind the data and resulting in its underutilization.

[0018] Based on this, embodiments of this application provide a message writing method. Figure 1 This is a flowchart illustrating a message writing method provided in an embodiment of this application; as shown below. Figure 1 As shown, the method includes: S101. When consuming the first message on the consumer side, obtain the first CMDB information corresponding to the first message.

[0019] It should be noted that "consumer" can be understood as a consumer service, a consumer, or a consumer program. "First message" can be understood as a Kafka message. The consumer-side consumption of the first message can be understood as follows: the producer encapsulates performance data into Kafka messages and sends them to the message middleware. The consumer pulls or passively receives Kafka messages from the message middleware. Performance data can also be understood as multi-source data covering Internet Protocol (IP) resource monitoring. The data source is the monitored object, and the monitoring layers are: Information Technology (IT) resource monitoring, cloud resource monitoring, log monitoring, middleware monitoring, Kubernetes container monitoring, application process monitoring, and custom monitoring. Each monitoring layer corresponds to a function / key indicator. The function / key indicator for IT resource monitoring is physical hardware, including physical servers, network devices, storage devices, and security devices. The function / key indicator for cloud resource monitoring is virtual resources, including virtual machines, cloud hosts, block / object / file storage, and network bandwidth. The function / key indicator for log monitoring is configuration logs, including service logs, operation logs, operating system logs, and device logs. The function / key indicator for middleware monitoring is middleware, including: Structured Query Language (MyStructured Query Language). Languages ​​such as MySQL, Remote Dictionary Server (Redis), MongoDB, RocketMQ, Kafka, ClickHouse, and Keepalived are included. Kubernetes container monitoring focuses on Kubernetes containers (pods), including Central Processing Unit (CPU), memory, network, and usage. Application process monitoring focuses on process count, including active processes, ports, CPU, memory, and I / O. Custom monitoring focuses on network performance, including response time, Domain Name System (DNS) resolution, Transmission Control Protocol (TCP) establishment, and access traffic. The first message can be any message encapsulated in a Kafka message, and the specific method can be determined based on the actual situation; no restrictions are imposed here.

[0020] It should be noted that the first CMDB information can be understood as the first CMDB cache information or the first CMDB attribute information. Obtaining the first CMDB information corresponding to the first message can be understood as obtaining the first CMDB information from the consumer's local cache, or obtaining the first CMDB information from the Redis cache one level above the consumer's local cache. The specific method of obtaining the information can be determined according to the actual situation and is not limited here.

[0021] It should be noted that the consumer program consumes the first message in parallel. The number of replicas corresponding to the consumer program can be one or more. The number of partitions corresponding to the first message is obtained. The number of threads for parallel consumption of each replica is determined based on the number of partitions and the number of replicas. The product of the number of threads for parallel consumption of a single replica and the number of replicas must be less than or equal to the number of partitions.

[0022] S102. Add the first message and the first CMDB information to the first cache queue; the first cache queue is the cache queue corresponding to the type of the first message.

[0023] It should be noted that the first cache queue is the cache queue corresponding to the type of the first message; adding the first message and the first CMDB information to the first cache queue can be understood as injecting the first CMDB information into the first message, and then adding the injected first message to the corresponding first cache queue according to the type of the first message.

[0024] S103. If the first quantity of the first message cached in the first cache queue is greater than the first value, the first message cached in the first cache queue is written to the database; the first value is determined based on the database's write capability.

[0025] It should be noted that the database can be any database. As an example, the database can be the write-side database, ClickHouse, which is a high-performance, online analytical processing (OLAP) columnar database suitable for large-scale and efficient processing of complex queries and real-time analysis. The first value is determined based on the database's write capacity. This value can be selected from the minimum and maximum write capacity of the database, and the specific value can be determined according to the actual situation, without limitation here. As an example, ClickHouse is suitable for single large-batch writes of 100,000-500,000 rows, and the first value can be 100,000 rows. If the first quantity of the first message cached in the first cache queue is greater than the first value, the first message cached in the first cache queue is written to the database. This can be understood as writing the first message cached in the first cache queue to the database all at once if the first quantity of the first message cached in the first cache queue is greater than the first value. If the number of the first messages cached in the first cache queue reaches the first value, then all cached first messages are written to the database at once.

[0026] In the scheme of this application embodiment, when the consumer consumes the first message, it obtains the first CMDB information corresponding to the first message and adds the first message and the first CMDB information to the first cache queue at the same time to facilitate subsequent query processing; and the first cache queue is the cache queue corresponding to the type of the first message. When the first quantity information of the first message cached in the first cache queue is greater than a first value, the first message cached in the first cache queue is written to the database. That is, the message is classified according to its type, the first message is temporarily stored in the cache queue, and when the first quantity information is greater than the first value, the first message is written to the database at the same time, thereby improving the message writing speed.

[0027] In this embodiment of the application, the process of obtaining the first CMDB information corresponding to the first message specifically includes: calling the CMDB service to obtain the first configuration data and caching the first configuration data to the local cache corresponding to the consumer; obtaining the identification information corresponding to the first message; and, if the local cache has not reached its expiration time and / or is not in a refresh cycle, searching in the first configuration data in the local cache based on the identification information to obtain the first CMDB information.

[0028] It should be noted that the first configuration data can be understood as the full configuration data; the local cache can be understood as the consumer's local cache (Caffeine). Calling the CMDB service to obtain the first configuration data and caching it in the corresponding local cache on the consumer can be understood as the consumer calling the CMDB service to obtain the full configuration data upon startup and preloading it into the local cache. The identification information can be understood as the universally unique identifier (UUID) and resource type corresponding to the first message. Obtaining the identification information corresponding to the first message can be understood as the consumer processing the first message (i.e., the Kafka message) and obtaining the tag UUID and resource type corresponding to it. The expiration time can be understood as the expiration time of the local cache, and the refresh cycle can be understood as the refresh cycle of the local cache. The specific values ​​of the expiration time and refresh cycle can be determined according to the actual situation and are not limited here. If the local cache has not reached its expiration time and / or is not in a refresh cycle, the first CMDB information is obtained by searching the first configuration data in the local cache based on the identification information. This can be understood as follows: if the local cache has not reached its expiration time and / or is not in a refresh cycle, the CMDB data object in the consumer's local cache is matched with the tag UUID and resource type corresponding to the first message to obtain the first CMDB information; and the CMDB-related attribute information of the message is filled in according to the matching result.

[0029] The solution in this application embodiment obtains the CMDB information corresponding to the first message, which facilitates subsequent query processing.

[0030] In this embodiment of the application, the method further includes: when the local cache has expired or is in a refresh cycle, searching the Redis cache; the Redis cache is the parent cache of the local cache; when the Redis cache is not empty, searching for the first CMDB information in the Redis cache based on the identification information.

[0031] It should be noted that the expiration time can be understood as the expiration time of the local cache, and the refresh cycle can be understood as the refresh cycle of the local cache. The specific values ​​of the expiration time and refresh cycle can be determined according to the actual situation and are not limited here. The Redis cache is the parent cache of the consumer's local cache, and can also be understood as a second-level cache Redis. When the local cache has reached its expiration time or is in the refresh cycle, the Redis cache is searched. This can be understood as triggering the local cache Caffeine's cacheLoader.reload(UUID) method to search the second-level cache Redis. The Redis cache is not empty, which can be understood as the Redis cache exists. When the Redis cache is not empty, the first CMDB information is searched in the Redis cache based on the identifier information. This can be understood as, when the Redis cache exists, the first CMDB message is searched in the Redis cache based on the tag UUID and resource type corresponding to the first message.

[0032] In this embodiment of the application, the method further includes: when the Redis cache is empty, waiting for the Redis cache to be refreshed and obtaining the second configuration data from the CMDB data source; and searching for the first CMDB information in the second configuration data in the Redis cache based on the identification information.

[0033] It's important to note that an empty Redis cache can be understood as the Redis cache not existing. In this case, Redis returns a null value to the consumer. The second configuration data can be understood as CMDB data retrieved from the CMDB data source. When the Redis cache is empty, it waits for a Redis cache refresh and retrieves the second configuration data from the CMDB data source. Searching for the first CMDB information within the second configuration data in the Redis cache based on the identifier information can be understood as, in the case of the Redis cache not existing, waiting for an automatic Redis refresh and loading data from the CMDB data source, then searching for the first CMDB information in the second configuration data based on the tag UUID and resource type corresponding to the first message. The local cache is updated based on the second configuration data from the Redis cache during the next local cache update.

[0034] The solution in this application embodiment ensures the accuracy and real-time performance of CMDB information by using device cache expiration time.

[0035] In this embodiment of the application, the method further includes: when the consumer consumes a CMDB update, updating the local cache corresponding to the consumer based on the updated CMDB.

[0036] It should be noted that CMDB updates include at least one of the following: adding, deleting, or modifying CMDB content. When a consumer receives a CMDB update, it updates its local cache accordingly. This can be understood as follows: when the consumer starts, it listens for a CMDB topic; when a CMDB update is received, the consumer updates its local cache based on the updated CMDB content. During a CMDB update, the CMDB service broadcasts RocketMQ messages to the consumer.

[0037] In this embodiment of the application, the process of updating the local cache corresponding to the consumer based on the updated CMDB specifically includes: determining the update type corresponding to the CMDB update; the update type includes at least one of addition, deletion, and modification; and updating the local cache corresponding to the consumer based on the update type.

[0038] It's important to note that determining the update type during a CMDB update can be understood as judging whether the update is an insert, delete, or modify (or update). Updating the consumer's local cache based on the update type can be understood as determining whether the update type is insert, update, or delete, and then updating the local cache using `caffeine Cache.put()` or `Cache.Evit()`; this ensures the real-time performance and consistency of the CMDB.

[0039] In the scheme of this application embodiment, the local cache can share the load of the central cache, especially in high-concurrency scenarios, and is close to the optimal cache library. At the same time, considering that the CMDB data may be in a state of dynamic updating, a near real-time update notification mechanism for RocketMQ and a timed cache refresh mechanism for caffine are designed to ensure the accuracy and real-time performance of CMDB information.

[0040] In this embodiment of the application, the first rate corresponding to the consumption of the first message by the consumer is less than the second rate corresponding to the caching of the first message by the first cache queue; the second rate is less than the third rate corresponding to the writing of the first message cached in the first cache queue to the database.

[0041] It should be noted that the first rate can be understood as the rate at which the consumer consumes the first message; the second rate can be understood as the business rate, i.e., the rate at which the cache queue increases; and the third rate can be understood as the rate at which the first message is written to the database, i.e., the ingestion rate. In practical applications, the first rate can be denoted as... The second speed can be denoted as The third speed can be denoted as The first rate can be achieved through... The calculation yields the result, where P is the number of partitions. R represents the maximum number of messages to fetch, where R is the number of replicas. For consumption time. The second rate can be achieved through... The calculation yields C, where C is the number of parallel consumption threads per replica. To retrieve the maximum number of messages, The time period for the cache queue; the third rate can be achieved through... The calculation shows that W represents the database's ingestion capacity (the optimal ingestion quantity, i.e., the first value). This refers to the interval period for warehousing.

[0042] The solution in this application embodiment balances the production consumption rate, business processing, and data entry performance rate, ensuring high throughput of Kafka, avoiding message backlog, and maximizing the high-performance data entry of ClickHouse.

[0043] To facilitate understanding, the above solution is illustrated here. The message writing method described above can be used as an example of a high-performance writing method for massive data to a cloud monitoring platform. This method mainly consists of several steps, including message production to Kafka, Kafka data consumption, CMDB data injection, rate balancing control, cache queue design, and load balancing ClickHouse sharding writing. Figure 2 A schematic diagram of an exemplary high-performance data writing method system architecture provided in this application embodiment; as shown Figure 2As shown, the data includes monitoring of information technology (IT) resources, container orchestration engine (Kubernetes, K8S) resources, development cloud platform (OpenStack) monitoring, and log monitoring. Each data can be obtained from different data sources. For example, IT resource monitoring can be obtained from source A, source B, and source C, while log monitoring can be obtained from source X, source Y, and source Z. Different data sources create independent one-to-one topics (Topic1-n) based on resource types. Messages of different data sources can then be assigned to different Topics. Each Topic maintains multiple partitions, and a Group can contain multiple Consumers. Each Consumer can consume messages from one or more partitions. Zookeeper's role is to configure the cluster; it is a distributed system coordination service widely used in big data and microservices. It is used to monitor the status of shard1, shard2, replica1, and replica2, and to manage the election of the leader partition. Zookeeper is responsible for monitoring the status of Kafka brokers and groups and coordinating communication between them. ClickHouse (a column-oriented database for data analytics) sharded cluster provides data sharding and replication mechanisms, leveraging the open-source cluster management program Zookeeper to achieve clustered database management.

[0044] To address the challenges of deep monitoring of multiple data sources and massive resources, this system provides real-time monitoring of application and IT infrastructure resources, applications, middleware, logs, and event reporting. It collects various metrics, logs, and event data, offering high-speed writing, real-time querying, and storage capabilities. This enables real-time decision analysis using monitoring data, improves event handling efficiency, and helps users easily manage daily operations and maintenance scenarios. A highly available Kafka cluster combined with a ClickHouse sharded replication cluster is used to achieve massive data access.

[0045] On the consumer side, different data sources create independent one-to-one topics based on resource types. Ultimately, messages of different data source types can be assigned to different topics. The Kafka Broker is the message storage server, storing received messages on disk and maintaining multiple partitions for each topic. Kafka Broker supports horizontal scaling; as the number of Kafka Broker nodes increases, the throughput of the Kafka cluster also increases. Groups are message consumers, subscribing to and consuming messages from the Kafka Broker using a pull model. A Group can contain multiple Consumers, and each Consumer can consume messages from one or more partitions. ZooKeeper's role is to configure the cluster, elect a leader, and manage partitions. ZooKeeper is responsible for monitoring the status of Kafka Brokers and Groups and coordinating communication between them.

[0046] On the write side, ClickHouse is a high-performance OLAP columnar database suitable for efficiently handling complex queries and real-time analysis on a large scale. To achieve high availability, load balancing, and horizontal scaling, ClickHouse provides data sharding and replication mechanisms, leveraging the open-source cluster management program ZooKeeper for clustered database management. Based on ClickHouse's ability to handle large batches of 100,000-500,000 rows of writes at a time, with a single table ideally requiring no more than one write request per second, we designed a Kafka data splitting and cache queue batch ingestion program. Corresponding to Kafka topics, we established multiple concurrent cache queues (ConcurrentLinkedQueue) at the smallest granularity of resource type. These queues act as buffers to store temporary data, awaiting the optimal ingestion quantity for a single write. Simultaneously, utilizing its sharding and replication features, under the basic two-shard, two-replica condition, we achieved over four write requests per second per table, writing 10,000-50,000 rows of data per batch, enabling millions of writes per second for a single topic.

[0047] The solution in this application embodiment uses a one-stop, three-dimensional operation and maintenance management platform to meet the near real-time data writing requirements of most sources; a single replica and single topic can handle hundreds of millions of data writes per day without message backlog or data loss; data is associated with the CMDB configuration database in real time, providing local high-speed cache writing data CMDB attribute backfill information; business processing speed matches consumption speed, without causing Kafka rebalancing or business interruption; Ckickhouse provides high-speed batch writing to avoid frequent data entry and cache queue data backlog leading to Out of Memory (OOM); and an end-to-end full-link monitoring system is built to achieve minute-level data awareness and minute-level data entry.

[0048] Monitoring objects and their main functional indicators.

[0049] The system's operation flow is mainly as follows: Kafka supports multiple data sources, including but not limited to, IT resource monitoring (network devices, physical machines, virtual machines, cloud hosts, SDN, Ceph, ESSD), application monitoring (K8S clusters), log monitoring (syslog), and middleware monitoring (MySQL, Redis, MongoDB, RocketMQ, Kafka, ClickHouse, Keepalived, etc.). Figure 3 Resource types and key indicators. Figure 3 A schematic diagram illustrating an exemplary monitoring object and its main functional indicators provided in this application embodiment; as shown Figure 3As shown; the data source is the monitored object, and the monitoring layers are: Information Technology (IT) resource monitoring, cloud resource monitoring, log monitoring, middleware monitoring, container orchestration engine (Kubernetes) container monitoring, application process monitoring, and custom monitoring. Each monitoring layer corresponds to a function / key indicator. The function / key indicator for IT resource monitoring is physical hardware, including physical servers, network devices, storage devices, and security devices. The function / key indicator for cloud resource monitoring is virtual resources, including virtual machines, cloud hosts, block / object / file storage, and network bandwidth. The function / key indicator for log monitoring is configuration logs, including service logs, operation logs, operating system logs, and device logs. The function / key indicator for middleware monitoring is middleware, including: Structured Query Language (MySQL), Remote Dictionary Service (Redis), MongoDB, RocketMQ, Kafka, ClickHouse, Keepalived, etc. The function / key indicator for container orchestration engine (Kubernetes) container monitoring is: container orchestration engine containers (Kubernetes). pods include central processing unit (CPU), memory, network, and usage; application process monitoring corresponds to the number of processes, including active processes, ports, central processing unit (CPU), memory, and read / write input / output (I / O); custom monitoring corresponds to network performance, including response time, domain name system (DNS) resolution, transmission control protocol (TCP) establishment, and access traffic.

[0050] For message middleware systems, end-to-end latency is a key performance indicator for message transmission. In this proposal, end-to-end refers to the latency throughout the entire process from Kafka message delivery to consumption, business processing, queue buffering, and disk storage. Improper use and configuration of Kafka middleware in business applications often negatively impacts end-to-end performance, leading to increased latency and message backlog. By analyzing the composition of Kafka data transmission end-to-end latency, higher throughput can be achieved through targeted configuration and scaling of business applications. Figure 4 This application provides an exemplary schematic diagram illustrating the end-to-end latency of Kafka data writing; as shown. Figure 4 As shown, for the server side of the Kafka consumer-to-database model, i.e., the end-to-end latency of Kafka data writing, the workflow for processing a complete I / O request is as follows: 1. Producer latency: The third-party data collector writes messages with resource type and CMDB characteristic information to the corresponding Kafka Topic. The message receiving thread preprocesses the requests, including the produce, public, and commit steps.

[0051] 2. Consumer delay: The backend consumer program starts multiple listeners according to message type, consumes messages in parallel using multiple threads, and periodically pulls messages in batches with a maximum quantity of batchSize; this includes catch-up (the time required for the consumer to catch up with the production schedule) and fetch (the time required to retrieve data).

[0052] 3. Business processing latency: For the retrieved messages, data parsing is performed, the cache is queried to match the CMDB identifier, the CMDB information is associated, and business processing such as data normalization and information transformation is carried out. The processed data is added to the corresponding cache queue according to the resource type; including the handler.

[0053] 4. Queue waiting latency: For each new type of cache queue, a corresponding concurrent cache queue is created in advance to collect and buffer data; including: Cache Queue.

[0054] 5. Message entry latency: For large data entry requests, a dedicated concurrent cache queue polling thread schedules the data entry work. When certain conditions are met, the thread will complete the reading of the queue data and then complete the batch entry and data disking operations, including batch insertion.

[0055] The latency cost Tcost of the Kafka consumer-database multi-threaded model before optimization can be expressed as formula (1), which is as follows: (1) in, It is the loss caused by the producer's delay. The delay time for the consumer thread to fetch. and This refers to the time consumed by Java business processing and the cache queue waiting for data to be added to the database. This refers to the time required for scheduled threads to perform tasks such as batch data entry.

[0056] Analysis shows that optimizing the front-thread model incurs a latency cost. It contains a large number of time-consuming operations, among which , , and , All of these can be optimized using appropriate configurations and processing.

[0057] The high-speed cache design satisfies CMDB backfilling and data cleaning.

[0058] CMDB (Configuration Management Database) stores and manages various configuration and topology information of devices in an enterprise IT architecture. CMDB provides reliable data support for upper-layer application scenarios. When cloud monitoring platform performance data, logs, and other information are involved, they must be linked with CMDB information to leverage the value of configuration information while ensuring the accuracy and validity of the data.

[0059] To handle the massive resource monitoring demands of cloud monitoring platforms, a high-speed cache must be designed to handle the association of billions of performance data points with the CMDB (CMDB) data daily. Traditional distributed caches alone cannot meet the high-frequency call demands. Therefore, a two-level cache is adopted, combining a distributed cache with a traditional distributed Caffeine layer. The local cache can offload the load from the central cache, making it near-optimal, especially in high-concurrency scenarios. Furthermore, considering that CMDB data may be dynamically updated, the near real-time update notification mechanism of RocketMQ and the timed cache refresh mechanism of Caffeine are used to ensure the accuracy and real-time nature of CMDB information. This application's embodiment designs a high-speed local CMDB cache and a near real-time CMDB update mechanism. Figure 5 A flowchart illustrating an exemplary near real-time update strategy for message queues (MQ) provided in this application embodiment; as shown... Figure 5 As shown, the specific steps are as follows: 1. CMDB service is running.

[0060] 2. Provides CRUD (Create, Read, Update, Delete) services.

[0061] 3. Has the CMDB data been updated?

[0062] It should be noted that if the CMDB data is updated, proceed to step 4.

[0063] 4. Input data from CMDB is added to a secondary cache, Redis.

[0064] 5. Send a RocketMQ broadcast message.

[0065] 6. Start the multi-copy service on the consumer side and listen for broadcast messages.

[0066] 7. Determine the type of addition, deletion, or modification, and update the local cache of Caffeine.

[0067] 8. End.

[0068] Figure 6 A flowchart illustrating an exemplary timed update strategy provided in this application embodiment; as shown Figure 6 As shown, the specific steps are as follows: 1. When the consumer service starts, the cache is preloaded.

[0069] 2. Parse the Kafka message and look up the cache based on the UUID and type.

[0070] 3. Check if the Caffeine cache has expired or if it is automatically refreshed periodically.

[0071] It should be noted that if the Caffeine cache has expired or is periodically refreshed automatically, proceed to steps 4-8; if the Caffeine cache has not expired or is not periodically refreshed automatically, proceed to steps 9-10.

[0072] 4. Trigger reload of the secondary cache Redis.

[0073] 5. Check if the Redis cache has expired or if a forced refresh is needed.

[0074] It should be noted that if the Redis cache has expired or a forced refresh has been performed, proceed to steps 6-8; if the Redis cache has not expired or a forced refresh has not been performed, proceed to step 8.

[0075] 6. Load the CMDB data source.

[0076] 7. Update the Redis second-level cache.

[0077] It should be noted that step 7 is executed in response to a cache query request.

[0078] 8. Read Redis second-level cache.

[0079] It should be noted that after step 8, step 10 is executed in response to the cache query request.

[0080] 9. Read the local cache of Caffeine.

[0081] 10. Performance data is backfilled into CMDB information.

[0082] 11. Operation and maintenance data cache queue is waiting to be stored.

[0083] To facilitate understanding, the above scheme is explained in detail here to ensure the real-time performance and consistency of potentially dynamically changing CMDB data and monitoring data. This application embodiment employs a dual-caching refresh mechanism, adding long-period timed refreshes while maintaining real-time performance, thus incorporating redundancy. The design to ensure real-time performance is as follows: 1. Under normal CMDB service conditions, provide CRUD (Create, Read, Update, Delete) services to external users.

[0084] 2. When a user initiates a CMDB update, delete, modify, or change, a RocketMQ message should be sent via Broadcasting after the data change is completed. It cannot be sent via Cluster, otherwise, in a multi-replica deployment, only one replica will be able to consume the message.

[0085] 3. After the backend consumption service starts, it listens to the CMDB Topic.

[0086] 4. After the backend consumption service consumes CMDB add, delete, and modify messages, it determines the operation type (insert / update / delete) and updates the local cache using caffeine Cache.put() or Cache.Evit() to ensure CMDB real-time performance and consistency.

[0087] Another redundant update strategy is designed, which does not rely on RocketMQ messages for updates. The steps are as follows: 1. When the consumer program starts, it calls the CMDB service to obtain the full configuration data and preloads it into the local cache Caffeine.

[0088] 2. The consumer processes Kafka messages, matches them with locally cached CMDB data objects based on the message's tag UUID and resource type, and populates the message's CMDB-related attributes based on the matching results.

[0089] 3. The message indicating that the CMDB configuration information has been filled is used as the data to be imported into the database and added to the corresponding cache queue ConcurrentLinkedQueue according to the resource type.

[0090] 4. When the local cache expires or is automatically refreshed periodically, the caffeine's cacheLoader.reload(UUID) method is triggered to look up the second-level cache Redis.

[0091] 5. If the second-level cache exists, add it to the Caffeine cache. If it does not exist, return an empty value, wait for Redis to automatically refresh, load the data from the CMDB data source, and update the local cache in response to the next Caffeine update.

[0092] In theory, setting a cache expiration time can guarantee eventual consistency between the database and the cache. However, in some data consistency-sensitive scenarios, even the cache expiration period can cause business anomalies. Therefore, it is necessary to make every effort to maintain consistency between the database and the cache.

[0093] Flow control and rate matching.

[0094] This application's embodiments address high-throughput scenarios with massive data volumes. In the application of the message middleware Kafka, Kafka decouples the message production and consumption processes. However, to prevent service crashes due to resource exhaustion, it should be designed appropriately, implementing flow control during message transmission to protect the message broker and consumer clients downstream of the message flow.

[0095] 1. From the producer's perspective In scenarios with multiple data sources and multiple resource sub-types, topics are created based on resource sub-types as the smallest unit. Messages are published to the broker using a push model, with each message appended to different partitions within the same topic, representing sequential disk writes. Theoretically, for large amounts of data, only adding broker nodes is needed for the entire Kafka cluster to adapt to data of any size. In production, adjusting the `linger.ms`, `batch.size`, and `compression.type` parameters can control the batch sending time and batch size for the same partition and perform message data compression.

[0096] 2. From the consumer's perspective Without partitions, a consumer can only consume one topic. To improve Kafka throughput, multiple partitions are set for each topic. Consumers are grouped to consume different partitions of a partition, which can improve concurrent consumption capabilities and greatly enhance Kafka's consumption capacity. Therefore, the initial partitions can be dynamically adjusted to dynamically adjust the consumption capacity.

[0097] 3. From and , Based on different business and data support needs, a model can be designed where multiple consumers form a consumer group. By setting up batch consumption on the consumer side and initially configuring the consumer program with 3 replicas in a distributed environment, the number of parallel consumption threads can be dynamically calculated. The following relationship exists in the parallel consumption scenario.

[0098] Configuration principle: Number of machines (replicas) * concurrency <= number of partitions; otherwise, it will cause Kafka rebalancing. During Kafka rebalancing, the coordinator notifies consumers to start rebalancing. Rebalancing may lead to data loss, therefore the rebalancing process has a significant impact on optimizing the stability of Kafka consumer groups.

[0099] If the number of K8S POD replicas for consumer business processing is RThe number of partitions for a single topic is P The number of concurrent consumer threads for each replica is [number] times the total number of threads is [number]. C Then there is R Number of partitions per replica P Effective concurrent thread count C The relationship is as shown in formula (2), which is shown below.

[0100] (2) Where P is the number of partitions in a single topic; C is the number of parallel consumption threads in a single replica multi-threaded system; and R is the number of replicas. The number of parallel consumption threads C in a single replica can be obtained by formula (3), as shown below.

[0101] (3) Where C represents the number of parallel consumption threads per replica; R represents the number of replicas; and P represents the number of partitions in a single topic. By setting the Kafka consumer listener type to batch (listener.type=batch), the maximum number of records retrieved per poll (max-poll-records) is... (Configurable) To achieve batch consumption, with the number of partitions remaining unchanged, add multi-threading to the consumer group. The setting is: batch message container concurrency (batchcontainer.setConcurrency) = number of partitions. Then, the number of partitions can be calculated using formula (4). The total number of copies N to be consumed in a single batch within a period is given by formula (4) as shown below.

[0102] (4) Where N is the total number of fetches in a single batch for a single replica, P is the number of partitions in a single topic, R is the number of replicas, and λ is the number of fetches per thread. This refers to the consumption cycle, or the consumption rate per unit of time. It can be calculated using formula (5), which is shown below.

[0103] (5) in, Here, N represents the consumption rate, P represents the total number of data fetched in a single batch on a single replica, R represents the number of partitions in a single topic, λ represents the number of replicas, and λ represents the number of data fetched per thread. For unit consumption time, This refers to the consumption time. The above settings support parallel data extraction from multiple partitions using multi-threading, enabling high-concurrency processing of massive data throughput and consumption. It also achieves load balancing and horizontal scaling of Kafka to a certain extent.

[0104] Design of a program for batch data splitting and cache queue insertion into Kafka.

[0105] Given the characteristics of Kafka's small-batch, high-frequency data retrieval and ClickHouse's large-batch, low-frequency data writes, there is a time lag between peak business processing and scheduled data ingestion. Properly configuring a shared buffer can store temporary data, cached data, or intermediate results, reducing frequent access to the database or other backend systems. This improves database access efficiency and response time while reducing the load on the ClickHouse backend system from frequent writes.

[0106] The shared buffer uses the JDK's built-in unbounded thread-safe queue, ConcurrentLinkedQueue. ConcurrentLinkedQueue is characterized by its unbounded default capacity, ensuring data caching is non-blocking and doesn't impact speed. A second key feature is its support for concurrent scenarios; when a large number of threads read and write to the queue simultaneously, it maintains high throughput without additional locking mechanisms, handling frequent insertions and deletions, which aligns with business needs. While the queue itself is unbounded, its length must be controlled in practice. Otherwise, in extreme cases, the queue may grow indefinitely, leading to resource exhaustion and service outages. During business processing, the queue length range can be calculated based on estimated consumption traffic. When the number of consuming threads and the amount pulled per transaction increase, indicating a heavy network load, the queue length should increase. In this case, the inbound scheduling thread cycle should be shortened, and the inbound frequency increased to ensure data in the queue is processed without backlog. Conversely, when consumption traffic decreases, the queue length should decrease, allowing for a longer inbound scheduling thread cycle and a reduced inbound frequency, thus lowering CPU resource consumption.

[0107] Assume the initial length of the cache queue is It is also the minimum length of the cache queue. The maximum length of the cache queue is The preset length of the queue can be calculated using formula (6), which is based on the current speed (number of threads * quantity). Formula (6) is shown below.

[0108] (6) Where C represents the number of parallel consumer threads in a single-copy multi-threaded system, and λ represents the number of fetches per thread. T represents the period of the inbound scheduling thread. This is the minimum length of the cache queue. Let L be the preset length of the cache queue. After calculation, L is guaranteed to satisfy the interval constraint, which can be expressed by formula (7), as shown below.

[0109] (7) in, This is the minimum length of the cache queue. This represents the maximum length of the cache queue. Let L be the preset length of the cache queue, and L be the actual length of the cache queue. The algorithm described above can dynamically calculate the length L of each cache queue, ensuring that its length remains within a controllable range and does not actually reach an unbounded state.

[0110] This application uses ClickHouse as the backend storage. ClickHouse is suitable for writing large batches of 10,000-50,000 rows at a time, with a single table ideally requiring no more than one write request per second. Temporary data is periodically read from the buffer to anticipate the optimal number of rows to be written at once. Leveraging its sharded replication feature, under the basic two-shard, two-replica condition, this allows for more than four write requests per second per table, writing 10,000-50,000 rows at a time, achieving millions of writes per second for a single topic. This ensures high-speed writing and prevents cache queue backlog. Assuming each interval... If T is used for one entry into the warehouse and the entry capacity is W, then the entry rate can be expressed by formula (8) as follows.

[0111] (8) in, W represents the data ingestion rate, and W represents the database's data ingestion capacity (the optimal number of data ingested, i.e., the first value). This refers to the inbound interval. In other words, to ensure the cache queue doesn't back up, there must be an inbound capacity greater than the queue length within the Δt3 period. This is specifically achieved through the rate balancing design described above.

[0112] Since ConcurrentLinkedQueue can only hold a single queue, messages from different Topics, differentiated by resource type, ultimately enter different cache queues. To facilitate searching for queues within the same resource type, ConcurrentHashMap is used to manage resource types and queues. Key code examples include: Map. <String,ConcurrentLinkedQueue <string>>queueMap = new ConcurrentHashMap<>(), the following uses a single cache queue to illustrate the design process of a data splitting cache queue. Figure 7 This application provides an exemplary flowchart of a data offloading and caching queue; as shown in the embodiments. Figure 7 As shown, the specific steps are as follows: 1. The consumer service starts and initializes the cache queue.

[0113] 2. Periodic multi-threaded consumption, fetching once per second, batch of 2000 records.

[0114] 3. Business processing: CMDB cache association, add to cache queue.

[0115] 4. Has the scheduled cache entry time been reached?

[0116] It should be noted that if the warehousing time has been reached, proceed to step 5; otherwise, proceed to step 6.

[0117] 5. Is the cache queue empty?

[0118] It should be noted that if the cache queue is empty, proceed to step 6; if the cache queue is not empty, proceed to step 7.

[0119] 6. Wait for the next scheduled inbound shipment.

[0120] 7. The amount of data in the cache queue is greater than the amount of data added in a single batch.

[0121] 8. Nginx load balancing.

[0122] 9. Write to the ClickHouse sharded replication cluster.

[0123] 10. Business processing.

[0124] 11. Single warehousing completed.

[0125] To facilitate understanding, an example of the above solution is provided here. Taking single-type, single-topic data processing as an example, this can be extended to scenarios with multiple data sources and multiple data types. The specific functions and steps are as follows: 1. When the consumer service starts, initialize the cache queue (one cache queue for each type of resource).

[0126] 2. The consumer segment periodically consumes messages and performs business processing (multi-threaded consumption of the same Topic).

[0127] 3. After the business process injects the CMDB cache information, it is added to the concurrent cache queue ConcurrentLinkedQueue.

[0128] 4. When the scheduled task in Spring @Schedule expires, check the number of items in the cache queue. If the cache queue is empty, wait for the next scheduled task. Otherwise, retrieve a batch of data from the queue (batchSize) and provide a one-time batch write to ClickHouse via the native HTTP interface and load balancing cluster sharding.

[0129] 5. Loop through and check the amount of cached data in the Queue. If the amount is greater than the batch write size (BatchSize), write it immediately without waiting for the next scheduled task. This will run for a long time and may cause data backlog in the cache queue, occupying machine memory and potentially leading to OutOfMemoryError (OOM).

[0130] 6. At this point, clickHouse's single batch write is complete, and the next business processing cycle begins.

[0131] The design incorporates load balancing of consumption capacity and warehousing capacity, along with rate control.

[0132] For high-concurrency consumption of massive amounts of data from Kafka and frequent batch writes to ClickHouse, it's necessary to limit the number of concurrent requests and the ingestion rate. The Kafka message retrieval rate should be less than (or roughly equal to) the rate at which the cache queue writes to ClickHouse. In other words, the average amount of data retrieved from Kafka per minute should be less than the average amount of data ingested per minute. Otherwise, data backlog in the cache queue will occur, consuming machine memory and potentially leading to OutOfMemoryError (OOM). Assume the initial length of the cache queue is... It is also the minimum length of the cache queue. The maximum length of the cache queue is According to the current The preset length of the queue is calculated based on the time velocity (number of threads * batch fetch quantity). It is calculated using formula (9), which is as follows.

[0133] (9) in, The preset length of the cache queue. This is the minimum length of the cache queue. Let C be the maximum length of the cache queue, C be the number of threads consuming the cache in a single copy using multi-threaded parallel processing, and λ be the number of threads that a single thread can fetch. The current time is used. The business rate, which is the rate at which the cache queue increases, can be calculated using formula (10), as shown below.

[0134] (10) in, Where C is the business rate, C is the number of parallel consumer threads in a single replica multi-threaded system, and λ is the number of fetches per thread. The time period of the cache queue is defined by the above design and calculation. The amount of data in the cache queue is quantifiable, that is, the consumption rate of each Topic must be less than the data ingestion capacity. This can be expressed by formula (11), which is shown below.

[0135] (11) in, For consumption rate, For business speed, The inbound rate is given by formula (5), (8) and (10). Substituting these into formula (11) yields formula (12), which is shown below.

[0136] (12) Where P is the number of partitions. R represents the maximum number of messages to fetch, where R is the number of replicas. C represents the number of parallel consumption threads per replica, where C is the unit of time period. W represents the time period of the cache queue, and W represents the database's ingestion capacity (the optimal ingestion quantity, i.e., the first value). This refers to the interval period for warehousing. For ease of understanding, Figure 8 A schematic diagram illustrating an exemplary rate control provided in an embodiment of this application; as shown Figure 8 As shown, taking a single topic as an example, Kafka consumers pull services from Kafka producers, process them through a Java business logic program, enter a Java cache queue, and then insert them into the database via ClickHouse for batch processing. To achieve a balance between consumption rate and database insertion rate, a certain consumption rate is required. Less than the service rate Less than the inbound rate .

[0137] In Java programming, the reporting source, listener, cache queue, and database insertion thread, and their corresponding relationships (should maintain a 1:1:1:1 ratio), should be defined as follows, based on the above reasoning and combining best practices of ClickHouse and Kafka, to maximize the utilization of ClickHouse's database insertion capabilities: With a single Topic partition number of P=12 and a Kubernetes three-replica deployment of R=3, the number of concurrent threads per replica can be obtained from formula (2) and can be expressed by formula (13), as shown below.

[0138] = 4 (13) The number of rows that ClickHouse can write (per batch) is W = 120,000 (Rows) (adjustable); the write frequency of ClickHouse (per batch) is Δt3 = 20 seconds (Second, S) (adjustable).

[0139] The Java business processing rate (single copy) can be expressed by formula (14), which is shown below.

[0140] (14) Kafka batch consumption frequency (single topic) C1 = 1S (sleep time is adjustable); the number of messages pulled in a single time can be obtained from formula (12) as λ, which can be expressed by formulas (15) and (16), as shown below.

[0141] (15) (16) At this time, under the deployment of a single Topic with three replicas R=3, the maximum message processing capacity per minute can be expressed by formula (17), which is shown below.

[0142] *60=3*4*1500*60=1080000 (17) This means a single topic can process 1.08 million data entries per minute, easily handling 1 billion data entries per day. Furthermore, it ensures that the consumption rate matches the business processing and data insertion capabilities without message backlog, cache queue backlog, or performance degradation due to frequent ClickHouse writes.

[0143] Based on the above balanced design of production and consumption rates, and balanced design of business processing and data writing rates, we can avoid Kafka message backlog, Kafka rebalance caused by untimely business processing affecting normal consumption, data backlog in cache queues, and occupation of machine memory, which may seriously lead to OOM.

[0144] 1. The backend consumer thread can dynamically set the sleep time of the consumer thread through formula (11), adjust the amount of data pulled by the consumer thread each time, and ensure sufficient business processing time.

[0145] 2. The backend consumer thread can dynamically set the periodic update time (sleep time) of ClickHouse to reduce the frequency of ClickHouse data entry and avoid performance degradation caused by frequent writes.

[0146] 3. Configure the backend consumer thread to handle the number of Kafka writes in a single batch, leveraging ClickHouse's advantage of handling low-frequency, large-scale data writes all at once.

[0147] 4. The number of concurrent Kafka consumer threads can be set in the backend to handle business in a multi-threaded manner, avoiding Kafka rebalance caused by pulling too many messages at once and delaying business processing.

[0148] 5. The above parameters can be used to dynamically adjust the number of messages that the Kafka consumer thread pulls in batches, thus adjusting Kafka's consumption capacity.

[0149] 6. By deploying Kubernetes containers, you can deploy and expand multiple replicas to double the consumption capacity.

[0150] For monitoring objects of different sizes, simply adjust the above parameters and add replicas, and configure the relevant parameters when starting the connection to the Kafka cluster to achieve a 10-50 times increase in consumption processing capacity, which basically meets the operation and maintenance monitoring needs of large, medium and small cloud data centers.

[0151] The following is the code for the specific settings.

[0152] @Bean public Map<String, Object> consumerConfigs() {; Map<String, Object> props = new HashMap<>(); props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG,kafkaConfig.getServer()); / / The maximum number of messages a Consumer instance can pull from the broker at one time; the official default value is 500. props.put(ConsumerConfig.MAX_POLL_RECORDS_CONFIG,calculateMaxPollRecords()); props.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest"); return props; }; / / Calculate the maximum number of messages to fetch in one operation; private Integer calculateMaxPollRecords() {; if (kafkaConfig.getKafkaMaxPollRecords()>0) {; return kafkaConfig.getKafkaMaxPollRecords(); }; / / The maximum number of entries to be added to the cache queue is initially 50,000; Integer cacheQueueBatchInsertSize = kafkaConfig.getCacheBatchInsertSize(); / / The initial time interval for entering data into the ClickHouse table is 30 seconds; Integer batchInsertInterval = kafkaConfig.getBatchInsertInterval(); / / The initial time interval for batch consumption and fetching from Kafka is 1 second; Integer kafkaListenInterval = kafkaConfig.getKafkaListenInterval() / 1000; / / The principle for setting the number of concurrent consumption threads is: number of machines (replicas) * concurrency <= number of partitions; Integer concurrency = kafkaConfig.getConcurrentKafkaListenerSize(); Integer maxPollRecords = (cacheQueueBatchInsertSize *kafkaListenInterval) / (batchInsertInterval * concurrentKafkaListener); return maxPollRecords; }

[0153] The advantages of the embodiments of this application are as follows: 1. Kafka offers high throughput, reduces end-to-end latency, and maximizes Kafka consumption capabilities.

[0154] 2. High-speed business processing, operation and maintenance monitoring data combined with CMDB database attribute backfilling.

[0155] 3. The design balances production consumption, business processing, and inbound rates, and combines concurrent consumption to achieve streaming processing, avoiding data backlog, business interruption, and memory overflow problems caused by rate mismatch.

[0156] 4. ClickHouse cache queue design: Data is distributed according to type, and a concurrent queue buffer is designed to store temporary data. Low-frequency, large-batch data is added to the database to improve ClickHouse write performance.

[0157] 5. Enables dynamic code configuration and Kubernetes multi-replica scaling, allowing for dynamic adjustment of consumption and ingestion capabilities to address various operational scenarios in large, medium, and small cloud data centers.

[0158] This application provides a message writing device. Figure 9 This application provides a schematic diagram of the structure of an exemplary message writing device; as shown in the embodiments of this application. Figure 9 As shown, the message writing device 900 includes: The acquisition unit 901 is used to acquire the first CMDB information corresponding to the first message when the first message is consumed at the consumer end; Adding unit 902 is used to add the first message and the first CMDB information to the first cache queue; the first cache queue is the cache queue corresponding to the type of the first message. The writing unit 903 is used to write the first message cached in the first cache queue to the database when the first quantity information of the first message cached in the first cache queue is greater than a first value; the first value is determined according to the writing capability of the database.

[0159] In some embodiments, the acquisition unit 901 is further configured to call the CMDB service to acquire first configuration data and cache the first configuration data in the local cache corresponding to the consumer; acquire the identification information corresponding to the first message; and, if the local cache has not reached its expiration time and / or is not in a refresh cycle, search the first configuration data in the local cache based on the identification information to obtain the first CMDB information.

[0160] In some embodiments, the message writing device 900 further includes a lookup unit, configured to look up a remote dictionary service Redis cache when the local cache has reached the expiration time or is in the refresh cycle; the Redis cache is the parent cache of the local cache; and when the Redis cache is not empty, to look up the first CMDB information in the Redis cache based on the identification information.

[0161] In some embodiments, the lookup unit is further configured to wait for the Redis cache to be refreshed and obtain second configuration data from the CMDB data source when the Redis cache is empty; and search for the first CMDB information in the second configuration data in the Redis cache based on the identification information.

[0162] In some embodiments, the message writing device 900 further includes an update unit, configured to update the local cache corresponding to the consumer based on the updated CMDB when the consumer consumes a CMDB update.

[0163] In some embodiments, the update unit is further configured to determine the update type corresponding to the CMDB update; the update type includes at least one of addition, deletion and modification; and update the local cache corresponding to the consumer based on the update type.

[0164] In some embodiments, the first rate at which the consumer consumes the first message is less than the second rate at which the first cache queue caches the first message; the second rate is less than the third rate at which the first message cached in the first cache queue is written to the database.

[0165] This application also provides an electronic device. Figure 10 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application; as shown below. Figure 10 As shown, the electronic device 1000 includes a processor 1001 and a memory 1003. Optionally, the electronic device 1000 may also include a communication bus 1002.

[0166] In specific embodiments, the processor 1001 can be at least one of the following: Application Specific Integrated Circuit (ASIC), Digital Signal Processor (DSP), Digital Signal Processing Device (DSPD), Programmable Logic Device (PLD), Field Programmable Gate Array (FPGA), CPU, controller, microcontroller, and microprocessor. It is understood that for different devices, the electronic device used to implement the above processor function can also be other types, and this embodiment does not specifically limit it.

[0167] In this embodiment, the communication bus 1002 is used to establish communication between the processor 1001 and the memory 1003; when the processor 1001 executes the running program stored in the memory 1003, it implements the following message writing method: When consuming the first message at the consumer end, the first CMDB information corresponding to the first message is obtained; the first message and the first CMDB information are added to the first cache queue; the first cache queue is the cache queue corresponding to the type of the first message; if the first quantity information of the first message cached in the first cache queue is greater than a first value, the first message cached in the first cache queue is written to the database; the first value is determined according to the write capability of the database.

[0168] Furthermore, the processor 1001 is also configured to call the CMDB service to obtain first configuration data and cache the first configuration data in the local cache corresponding to the consumer; obtain the identification information corresponding to the first message; and, if the local cache has not reached its expiration time and / or is not in a refresh cycle, search the first configuration data in the local cache based on the identification information to obtain the first CMDB information.

[0169] Furthermore, the processor 1001 is also configured to, when the local cache has reached the expiration time or is in the refresh cycle, search the remote dictionary service Redis cache; the Redis cache is the parent cache of the local cache; and when the Redis cache is not empty, search the first CMDB information in the Redis cache based on the identification information.

[0170] Furthermore, the processor 1001 is also configured to, when the Redis cache is empty, wait for the Redis cache to be refreshed and obtain the second configuration data from the CMDB data source; and search for the first CMDB information in the second configuration data in the Redis cache based on the identification information.

[0171] Furthermore, the processor 1001 is also configured to update the local cache corresponding to the consumer based on the updated CMDB when the consumer consumes a CMDB update.

[0172] Furthermore, the processor 1001 is also used to determine the update type corresponding to the CMDB update; the update type includes at least one of addition, deletion and modification; and update the local cache corresponding to the consumer based on the update type.

[0173] Furthermore, the first rate at which the consumer consumes the first message is less than the second rate at which the first cache queue caches the first message; the second rate is less than the third rate at which the first message cached in the first cache queue is written to the database.

[0174] This application provides a storage medium storing a computer program thereon. The computer-readable storage medium stores one or more programs, which can be executed by one or more processors. The computer program implements the message writing method described above.

[0175] Based on the above embodiments, this application provides a computer program product, including a computer program that can be executed by one or more processors, and the computer program implements the message writing method described above.

[0176] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0177] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this disclosure, in essence, or the part that contributes to the related technology, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes several instructions to cause an image display device (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of this disclosure.

[0178] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, and improvements made within the spirit and scope of this application are included within the scope of protection of this application.< / string>

Claims

1. A message writing method, characterized in that, The method includes: When consuming the first message on the consumer side, obtain the first configuration management database (CMDB) information corresponding to the first message; The first message and the first configuration management database (CMDB) information are added to the first cache queue; the first cache queue is the cache queue corresponding to the type of the first message. If the first quantity of the first message cached in the first cache queue is greater than a first value, the first message cached in the first cache queue is written to the database; the first value is determined based on the write capability of the database.

2. The method according to claim 1, characterized in that, The step of obtaining the first configuration management database (CMDB) information corresponding to the first message includes: Call the CMDB service to obtain the first configuration data, and cache the first configuration data in the local cache corresponding to the consumer. Obtain the identification information corresponding to the first message; If the local cache has not expired or is not in a refresh cycle, the first configuration management database (CMDB) information is obtained by searching the first configuration data in the local cache based on the identification information.

3. The method according to claim 2, characterized in that, The method further includes: If the local cache has reached its expiration time or is in the refresh cycle, the remote dictionary service Redis cache is searched; the Redis cache is the parent cache of the local cache. If the Redis cache is not empty, the first configuration management database (CMDB) information is searched within the Redis cache based on the identification information.

4. The method according to claim 3, characterized in that, The method further includes: If the Redis cache is empty, wait for the Redis cache to be refreshed and obtain the second configuration data from the CMDB data source; Based on the identification information, the first configuration management database (CMDB) information is retrieved from the second configuration data in the Redis cache.

5. The method according to claim 1, characterized in that, The method further includes: If the consumer receives a CMDB update, the local cache corresponding to the consumer is updated according to the updated CMDB.

6. The method according to claim 5, characterized in that, The step of updating the local cache corresponding to the consumer based on the updated CMDB includes: Determine the update type corresponding to the CMDB update; the update type includes at least one of add, delete, and modify. Update the local cache corresponding to the consumer based on the update type.

7. The method according to claim 1, characterized in that, The first rate at which the consumer consumes the first message is less than the second rate at which the first cache queue caches the first message; the second rate is less than the third rate at which the first message cached in the first cache queue is written to the database.

8. A message writing device, characterized in that, The device includes: The acquisition unit is used to acquire the first configuration management database (CMDB) information corresponding to the first message when the first message is consumed at the consumer end. The adding unit is used to add the first message and the first configuration management database (CMDB) information to the first cache queue; the first cache queue is the cache queue corresponding to the type of the first message; The writing unit is used to write the first message cached in the first cache queue to the database when the first quantity information of the first message cached in the first cache queue is greater than a first value; the first value is determined according to the writing capability of the database.

9. An electronic device, characterized in that, The electronic device includes: Memory is used to store executable instructions or computer programs. A processor, when executing computer-executable instructions or computer programs stored in the memory, implements the method according to any one of claims 1 to 7.

10. A computer program product comprising a computer program or computer-executable instructions, characterized in that, When the computer program or computer-executable instructions are executed by a processor, they implement the method described in any one of claims 1 to 7.

Images