Control and management method and system based on Kafka message queue

Abstract

The present invention relates to the field of data processing technology, specifically to a control and management method and system based on Kafka message queues. The invention constructs a time feature vector based on historical message traffic and event tags, combines the Kafka Broker network topology, and predicts the node load factor through a graph neural network; performs queue scheduling based on message priority and node load factor to optimize the message processing order; decouples computing, storage, and network resources based on resource usage indicators to determine whether to trigger expansion and contraction, replica migration, or current limiting adjustments; deploys lightweight Kafka instances at edge nodes, implements layered storage at the edge and center based on data activity, and asynchronously completes data synchronization; and constructs a recovery strategy model through reinforcement learning to achieve automatic identification and control responses for node anomalies.

Description

Technical Field

[0001] The present invention relates to the field of data processing, and in particular to a control and management method and system based on a Kafka message queue. Background Art

[0002] With the widespread adoption of distributed stream processing systems in scenarios such as finance, e-commerce, and the Industrial Internet, Kafka, as a high-throughput, scalable message queue middleware, has become a core data communication component. Kafka demonstrates excellent performance in processing large-scale data streams, supporting a publish / subscribe model, partitioning, and replica disaster recovery strategies, making it suitable for high-concurrency, high-availability data collection and transmission requirements.

[0003] In actual production environments, Kafka's operational efficiency and system stability are limited by multiple factors, including uneven node loads caused by message traffic fluctuations, resource bottlenecks limiting system throughput, data transmission latency in edge scenarios, and recovery decisions when abnormal events occur. To improve the adaptability and robustness of Kafka systems, the industry has proposed a series of optimization strategies, including rule-based scheduling algorithms, Pod scaling mechanisms, hotspot partition replica migration, and edge storage deployment.

[0004] However, in the face of complex and ever-changing business environments and dynamically changing resource systems, existing methods mostly rely on static threshold configuration or manual intervention, making it difficult to achieve highly automated and intelligent resource control. Furthermore, the scheduling priorities during system operation fail to reflect business urgency and node load in real time, resulting in some high-priority messages not being processed promptly when resources are limited, impacting overall service quality. Furthermore, existing systems still lack intelligent optimization capabilities at the policy level for fault response and self-healing, resulting in inefficient recovery processes and difficulty supporting higher levels of service continuity requirements.

[0005] To this end, a control and management method and system based on Kafka message queue are proposed. Summary of the Invention

[0006] The present invention provides a control and management method and system based on Kafka message queues. By constructing time feature vectors combined with graph neural networks to predict node loads, priority queues and service level agreements are used to achieve dynamic scheduling, and reinforcement learning strategy models are used to achieve abnormal self-healing, thereby improving system stability and processing efficiency.

[0007] To achieve the above object, the present invention provides the following technical solutions:

[0008] The control and management method based on Kafka message queue includes:

[0009] Based on historical message traffic data and event labels, a time feature vector is constructed. This time feature vector is used as the node input of a pre-trained graph neural network. The graph structure is constructed in combination with the Kafka Broker network topology to predict and sort the load factor of each node.

[0010] Messages to be processed at each node are categorized by labels and mapped to logical priority queues, and dynamically scheduled according to the service level agreement.

[0011] Based on the system resource usage indicators of each node, the scheduling control of computing resources, storage resources, and network resources is logically decoupled to determine whether to trigger expansion and contraction, replica migration, or message throttling, and then perform system resource adjustments;

[0012] Deploy lightweight Kafka Broker instances on edge nodes, implement tiered storage between the edge and the center based on the access frequency of pending messages and data activity, and synchronize data through an asynchronous mechanism.

[0013] Continuously monitor system abnormal indicators, build strategy models based on reinforcement learning, automatically decide on recovery actions and perform closed-loop adjustments to achieve system self-healing.

[0014] Furthermore, constructing the time feature vector includes:

[0015] Obtain historical message traffic data of each Kafka Broker node and aggregate it by time granularity;

[0016] Extracting event tags corresponding to the time granularity, including holiday tags, weather conditions, and advertising delivery status;

[0017] Constructing a sliding time window based on the historical message traffic data and calculating statistical indicators within the past 1 hour, 6 hours, and 24 hours, including traffic mean, maximum, and volatility;

[0018] The event label and the statistical indicator are concatenated to form a time feature vector of the node.

[0019] Furthermore, labeling and mapping the messages to be processed to logical priority queues includes:

[0020] Receive each pending message and extract its business type, task urgency, and data timeliness;

[0021] Prioritize messages based on pre-set priority rules, including real-time, near-real-time, and batch processing;

[0022] Build logical queues based on priority tags and bind messages with different priority tags to corresponding logical queues.

[0023] Furthermore, the service level agreement determines the scheduling priority of each logical queue according to the sum of the logical priority number of the message to be processed and the load factor number of the node where it is located, and preferentially processes the one with the largest sum of numbers.

[0024] Furthermore, logical decoupling of the scheduling and control of computing resources, storage resources, and network resources includes:

[0025] Periodically collect the computing resource utilization of the node. If the computing resource utilization exceeds the set computing resource threshold, the automatic expansion of the Pod is triggered; otherwise, the Pod reduction operation is triggered;

[0026] Monitor the node's storage resource usage in real time. If the storage resource usage exceeds the set storage resource threshold, the partition replica migration is triggered.

[0027] Monitor the network resource status between nodes. If the network resource status is greater than the network resource set threshold, low-priority message flow control is triggered.

[0028] Furthermore, edge and center tiered storage includes:

[0029] Determine the dividing line between hot data and cold data based on message access frequency and data activity;

[0030] Keep a complete copy of hot data on edge nodes, and store cold data only in the central cluster;

[0031] The data from the edge nodes is transmitted to the center in an asynchronous and synchronous manner.

[0032] Furthermore, the strategy model adopts a deep Q network, and its training process includes:

[0033] Define the state space, including the node load factor, system anomaly indicators and system resource usage indicators;

[0034] Define the action space, including logical priority queue reordering, system resource adjustment, and service degradation;

[0035] Set up a reward function with message integrity and business continuity as positive indicators;

[0036] Use historical load factors and system exception handling records as training samples to train the deep Q network model;

[0037] When the system is running, the current state vector of the node is continuously input, and the optimal control action is selected according to the model strategy.

[0038] The present invention also proposes a control and management system based on Kafka message queue, including:

[0039] The load prediction module is used to construct a time feature vector based on historical message traffic data and event labels, and use the time feature vector as the node input of the pre-trained graph neural network. It combines the Kafka Broker network topology to build a graph structure, predict the load factor of each node, and sort it;

[0040] Dynamic scheduling module, used to classify the messages to be processed by each node by label and map them to logical priority queues, and dynamically schedule them according to the service level agreement;

[0041] The resource decoupling module is used to logically decouple the scheduling and control of computing resources, storage resources, and network resources based on the system resource usage indicators of each node, determine whether to trigger expansion and contraction, replica migration, or message throttling, and perform system resource adjustments;

[0042] The data synchronization module is used to deploy lightweight Kafka Broker instances on edge nodes, implement tiered storage between the edge and the center based on the access frequency of pending messages and data activity, and complete data synchronization through an asynchronous mechanism.

[0043] The strategy generation module is used to continuously monitor system abnormal indicators, build a strategy model based on reinforcement learning, automatically decide on recovery actions and perform closed-loop adjustments to achieve system self-healing.

[0044] The beneficial effects of the present invention are:

[0045] 1. This method combines historical message flow data with event labels to construct time feature vectors, uses them as node inputs in a graph neural network, and integrates them with the Kafka cluster topology to accurately predict and rank the load level of each node. This method not only enhances the ability to model inter-node associations but also provides a reliable basis for subsequent scheduling and resource allocation, improving the system's load balancing and response efficiency.

[0046] 2. This invention continuously monitors system anomalies and employs a deep Q-network to build a policy model. This model automatically makes optimal control decisions and implements closed-loop adjustments in situations such as node anomalies and resource bottlenecks. This mechanism, with its continuous learning and optimization capabilities, helps enhance the system's autonomous decision-making and stable operation in complex scenarios.

[0047] 3. This paper utilizes a modular design to construct a Kafka control and management system that includes load forecasting, dynamic scheduling, resource decoupling, data synchronization, and policy generation. This system covers the entire process of message flow processing, resource scheduling, and exception recovery, and offers excellent scalability and flexibility. The system supports layered deployment at the edge and center, and implements refined scheduling based on logical priorities and service-level agreements. This improves overall processing efficiency and system stability in high-concurrency, large-scale data processing scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0049] Figure 1 This is a flow chart of the control and management method based on Kafka message queue provided by the present invention;

[0050] Figure 2 It is a structural diagram of the control and management system based on Kafka message queue provided by the present invention. DETAILED DESCRIPTION

[0051] The preferred embodiments of the present invention are described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present invention, and are not used to limit the present invention.

[0052] Example 1

[0053] The control and management method based on Kafka message queue, such as Figure 1 Shown, including:

[0054] S100: Construct a time feature vector based on historical message traffic data and event labels, and use the time feature vector as a node input for a pre-trained graph neural network. Combined with the Kafka Broker network topology, the graph structure is constructed to predict and sort the load factor of each node.

[0055] Furthermore, constructing the time feature vector includes:

[0056] Obtain historical message traffic data of each Kafka Broker node and aggregate it by time granularity;

[0057] Extracting event tags corresponding to the time granularity, including holiday tags, weather conditions, and advertising delivery status;

[0058] Constructing a sliding time window based on the historical message traffic data and calculating statistical indicators within the past 1 hour, 6 hours, and 24 hours, including traffic mean, maximum, and volatility;

[0059] The event label and the statistical indicator are concatenated to form a time feature vector of the node.

[0060] Specifically, the system regularly collects message traffic data from each Kafka broker node in the Kafka cluster over historical periods. This data includes performance metrics such as the receive rate, send rate, and message processing volume per unit time. This data is aggregated according to a preset time granularity (e.g., every 1 minute, every 5 minutes, or every 15 minutes) to generate time series data aligned with timestamps. To enhance the time feature vector's ability to perceive external contextual changes, the system obtains event labels aligned with this time granularity from the business side or external services. Event labels include: holiday flags, which indicate whether the current time falls on a statutory holiday, weekend, or major event; weather conditions, which capture weather characteristics in the current city, such as sunny, rainy, or snowy; and advertising status, which identifies whether online advertising or major promotions have been launched during the corresponding period. These events typically significantly impact traffic levels. Each label is encoded as a numeric vector, for example, a holiday flag with 1 representing a holiday and 0 representing a weekday. Weather conditions can be represented as a fixed-dimensional binary vector using one-hot encoding, for example, 0001 for sunny days. To capture traffic trends at different time scales, the system uses a sliding time window mechanism to calculate the following statistical indicators: mean, maximum, and standard deviation of traffic over the past hour, six hours, and 24 hours. The event label vector is then concatenated with the statistical feature vector to form a complete set of node time feature vectors.

[0061] By integrating historical message traffic data with event labels and using a sliding window multi-scale statistical feature extraction method, we can comprehensively characterize the load behavior of Kafka nodes over different time periods, while also enhancing the model's ability to perceive the impact of external events. By standardizing these features and inputting them into a graph neural network, we improve the accuracy and robustness of load forecasting and provide reliable data support for subsequent dynamic scheduling and resource control.

[0062] Furthermore, based on the deployment and connection relationship of each Broker node in the Kafka cluster, a network topology diagram is constructed, in which each node corresponds to a Kafka Broker, and each edge represents a communication link or replica relationship between Broker nodes. Subsequently, the time feature vector constructed in the previous step is used as the feature input of each node, and the pre-trained graph neural network model is introduced for training and prediction. During the training phase, the node load data collected historically is used as a label, and the graph neural network is pre-trained through supervised learning. During deployment and operation, the time feature vector of each node at the current moment is input into the pre-trained graph neural network model to obtain the standardized predicted load coefficient of each node, and the nodes are sorted according to the load coefficient.

[0063] S200: Label and classify the messages to be processed by each node and map them to logical priority queues, and dynamically schedule them according to the service level agreement;

[0064] Furthermore, labeling and mapping the messages to be processed to logical priority queues includes:

[0065] Receive each pending message and extract its business type, task urgency, and data timeliness;

[0066] Prioritize messages based on pre-set priority rules, including real-time, near-real-time, and batch processing;

[0067] Build logical queues based on priority tags and bind messages with different priority tags to corresponding logical queues.

[0068] Specifically, the Kafka message processing module first monitors the message stream to be processed. Every time a message is received, the business-related context information is extracted, which mainly includes: business type, such as transaction, log, monitoring, order, etc.; task urgency, which identifies the sensitivity of the task to processing timeliness, which is divided into high, medium and low; data timeliness, which refers to whether the message content has an expiration limit in the business process. The system combines the above three attributes to form a priority label based on the preset priority rules. In a feasible implementation method, the priority rules are defined as: real-time class (high priority): business type is transaction or alarm, task urgency is high, and data timeliness is less than 1 minute; quasi-real-time class (medium priority): such as monitoring messages, task urgency is medium, and data timeliness is between 1-10 minutes; batch processing class (low priority): such as periodic tasks such as log archiving and report generation, with weak data timeliness or no time constraints.

[0069] Based on the message priority tag, the system maps the message to the corresponding logical priority queue.

[0070] By extracting the business type, task urgency and data timeliness of the messages to be processed, and combining preset rules to classify the messages into priority labels such as real-time, quasi-real-time and batch processing, and mapping them to the corresponding logical priority queues, the importance and timeliness of the messages can be identified and hierarchically managed.

[0071] Furthermore, the service level agreement determines the scheduling priority of each logical queue according to the sum of the logical priority number of the message to be processed and the load factor number of the node where it is located, and preferentially processes the one with the largest sum of numbers.

[0072] Specifically, a logical priority number is assigned to each pending message. This number can be defined based on the priority tag to which the message belongs, for example: real-time message number is 3, quasi-real-time is 2, and batch is 1. The larger the number, the more important and urgent the message is. Then, combined with the current operating status of the Kafka Broker node, the load factor number of the node is obtained. This number can be obtained by sorting the load factors predicted by the graph neural network. The load factor number indicates that the message congestion at the node is more serious. Next, the logical priority number is summed with the load factor number of the node to obtain a comprehensive scheduling priority value. The system sorts all pending messages according to the comprehensive priority values corresponding to them, and prioritizes the messages with the largest number and the largest scheduling number, thereby improving resource utilization efficiency and alleviating system bottlenecks while taking into account both business importance and node load conditions.

[0073] By combining the logical priority number of a message with the load factor number of the node it resides on, this method quantifies the relationship between business importance and system operational status in scheduling strategies. Compared to traditional decision-making based solely on priority or resource metrics, this method prioritizes messages with greater system impact and criticality to business continuity in scenarios where high-priority messages and high-load nodes intersect, alleviating node congestion, reducing latency, and improving system throughput.

[0074] S300: Based on the system resource usage indicators of each node, the scheduling control of computing resources, storage resources, and network resources is logically decoupled to determine whether to trigger expansion and contraction, replica migration, or message throttling, and then perform system resource adjustments;

[0075] Furthermore, logical decoupling of the scheduling and control of computing resources, storage resources, and network resources includes:

[0076] Periodically collect the computing resource utilization of the node. If the computing resource utilization exceeds the set computing resource threshold, the automatic expansion of the Pod is triggered; otherwise, the Pod reduction operation is triggered;

[0077] Monitor the node's storage resource usage in real time. If the storage resource usage exceeds the set storage resource threshold, the partition replica migration is triggered.

[0078] Monitor the network resource status between nodes. If the network resource status is greater than the network resource set threshold, low-priority message flow control is triggered.

[0079] Specifically, the system obtains the CPU or memory utilization of the current node at set intervals and compares it with a preset upper threshold for compute resources. If compute resource utilization exceeds the threshold (for example, 70%), Kubernetes' auto-scaling mechanism is triggered, adding new Kafka pod replicas to distribute the load. If resource utilization remains below the scaling-down threshold (for example, 30%) for a sustained period, pod scaling is triggered to free up resources. The system monitors the disk usage of Kafka partitions in real time. When a node's disk utilization exceeds a set threshold (for example, 80%), the system migrates the partition replicas occupying more space on the current node to nodes with more available disk resources. Replica migration is performed by the Kafka Controller or an extended control component, and the replica status during the migration process is recorded to ensure data integrity and consistency. Network metrics such as throughput, latency, and packet loss rate are collected between nodes. If a network resource metric exceeds a set threshold (for example, egress bandwidth utilization exceeds 90%), the system initiates message priority throttling. This throttling strategy prioritizes message queues with lower logical priorities, reducing network pressure and ensuring that the transmission needs of higher-priority services are not affected.

[0080] By logically decoupling the scheduling and control of computing, storage, and network resources, the system can identify and respond to different types of resource bottlenecks, enabling more targeted and flexible resource management strategies. This approach avoids misjudgments and scheduling conflicts caused by resource interference, improving resource utilization and operational stability of Kafka clusters in high-concurrency, multi-service scenarios.

[0081] S400: Deploy lightweight Kafka Broker instances on edge nodes, implement tiered storage between the edge and the center based on the access frequency of pending messages and data activity, and synchronize data through an asynchronous mechanism.

[0082] Furthermore, edge and center tiered storage includes:

[0083] Determine the dividing line between hot data and cold data based on message access frequency and data activity;

[0084] Keep a complete copy of hot data on edge nodes, and store cold data only in the central cluster;

[0085] The data from the edge nodes is transmitted to the center in an asynchronous and synchronous manner.

[0086] Specifically, during the operation of Kafka Broker, the system continuously records the access frequency and the most recent active time of each message, and comprehensively calculates the "data activity" value of the message. The access frequency can be obtained by counting the number of times each message is read by consumers in a unit of time, and the data activity is obtained by evaluating the access frequency and the access time interval. In a feasible implementation method, the data activity is ,in Indicates the access frequency of the message per unit time. Indicates the time interval between the message and the last access. and Represent the access frequency and time interval weights respectively. Then, all messages are sorted according to the access frequency distribution and activity statistics within a period of time, and the quantile algorithm is used to delineate the dividing line between hot data and cold data. For example, the top 20% access frequency and activity messages can be classified as "hot data", and the rest as "cold data". For the identified hot data, the system will retain its complete copy in the lightweight Kafka Broker instance of the edge node to ensure low-latency access to hot messages by the edge node; while cold data is only stored in the central cluster to avoid edge resources occupying too much storage space. In order to ensure data consistency, an asynchronous synchronization mechanism is adopted between the edge node and the central node: the edge node records the change information in the local log queue, and regularly synchronizes the message data to the central cluster through incremental replication.

[0087] By evaluating message access frequency and data activity, we effectively distinguish between hot and cold data, retaining copies of frequently accessed, real-time data on edge nodes to reduce data access latency and improve system responsiveness. Meanwhile, we centrally store cold data in a central cluster, conserving edge storage resources and alleviating the burden of data redundancy. Asynchronous synchronization maintains data consistency between the edge and the center, enabling coordinated optimization of resource utilization and access performance, enhancing system scalability and scheduling flexibility.

[0088] S500: Continuously monitors system abnormality indicators, builds a policy model based on reinforcement learning, automatically decides on recovery actions and performs closed-loop adjustments to achieve system self-healing.

[0089] Furthermore, the strategy model adopts a deep Q network, and its training process includes:

[0090] Define the state space, including the node load factor, system anomaly indicators and system resource usage indicators;

[0091] Define the action space, including logical priority queue reordering, system resource adjustment, and service degradation;

[0092] Set up a reward function with message integrity and business continuity as positive indicators;

[0093] Use historical load factors and system exception handling records as training samples to train the deep Q network;

[0094] When the system is running, the current state vector of the node is collected and input into the pre-trained deep Q network, and the optimal control action is selected according to the model strategy.

[0095] Specifically, the state vector of the system is defined as the input of the model, which includes the following information items: the load factor of each KafkaBroker node, which can be obtained from the output of the prediction module; system anomaly indicators, including node downtime rate, response timeout, retry frequency, alarm log density, etc.; system resource usage indicators, including computing resources, storage resources and network resources. Define the optional control operations (i.e. actions) of the model, including but not limited to: reordering of logical priority queues, such as adjusting the priority of specific logical queues to alleviate the pressure on high-load nodes; system resource adjustments, such as Pod scaling, partition replica migration, and message throttling; service degradation control, such as suspending low-priority consumption, shutting down non-core modules, and enabling backup nodes. Guided by the self-healing effect of the system, a reward function in reinforcement learning is set. When the control action improves message integrity and business continuity, a positive reward is given; otherwise, a penalty is given. Among them, message integrity can be obtained by comparing the message counts on the production and consumer ends through Kafka's own consumer site management mechanism. The calculation method is: ,in, Indicates the message integrity indicator, Indicates the message count on the production side. Indicates the message count on the consumer side. Business continuity measures latency by accessing a link tracing system (such as Zipkin or Jaeger). The calculation method is: ,in, represents the business continuity indicator, Indicates the time when the consumer side obtains the message. represents the time when the producer sends a message. The reward function is designed as: ,in, represents the reward function, and is the preset weight, Indicates the current time point for calculating message integrity and business continuity, This represents the time point at which the last round of message integrity and business continuity calculations was performed. Training samples are extracted from the system's historical operational data. These samples consist of (state vector, executed action, reward value, and next state vector). In actual operation, the system continuously extracts the current state vector. The policy model adjusts and selects the optimal action based on the current state. The system executes the action and monitors its effectiveness, feeding the results back to the model for continuous training and adjustment, achieving closed-loop optimization.

[0096] By introducing a reinforcement learning strategy model based on a deep Q-network, the system can automatically learn and execute optimal control actions under complex conditions, enabling intelligent management of Kafka message queues. This approach effectively improves the real-time and accuracy of control decisions, optimizes core performance indicators such as message integrity and business continuity, and enhances the system's adaptability and self-healing capabilities in abnormal scenarios.

[0097] Example 2

[0098] This embodiment provides a specific application of a control and management system based on Kafka message queue to illustrate the collaborative working process of each functional module in the system. Figure 2 As shown in the figure, the system is deployed in a city-level smart transportation platform to process a large number of asynchronous messages from various sensors, intersection cameras and traffic signal equipment.

[0099] During operation, the load prediction module first collects the past seven days of historical message traffic data from each Kafka broker node, extracts event labels for corresponding time periods (such as peak periods, severe weather, and holidays), and generates time feature vectors based on a sliding time window. This vector is then input into the graph neural network model. Based on the Kafka cluster network topology, the model predicts the load factor of each node and ranks them from highest to lowest load, assisting in subsequent scheduling and resource management decisions.

[0100] The dynamic scheduling module extracts the urgency and timeliness of each message based on the service type (e.g., traffic violation detection, signal control, real-time road condition feedback), categorizes it into three levels: real-time, near-real-time, and batch processing, and maps them to corresponding logical priority queues. Based on the service level agreement, the system calculates a comprehensive priority number for each queue, prioritizing lighter-loaded, higher-priority message queues for processing.

[0101] The resource decoupling module regularly collects CPU, memory, disk, and network usage from Kafka nodes. When a node's CPU usage exceeds a set threshold (e.g., 80%), it automatically triggers Pod expansion. If storage utilization exceeds a threshold, it automatically initiates migration of the corresponding topic partition replicas. If inter-node network bandwidth is saturated, it throttles low-priority batch messages to ensure the transmission of high-priority tasks.

[0102] The data synchronization module deploys lightweight Kafka Broker edge instances at transportation hubs to cache data collected by local cameras and sensors. The system identifies hot data (such as sudden congestion information) based on access frequency and data activity, retaining a complete copy of it at the edge node and asynchronously transferring cold data back to the central cluster for archiving.

[0103] The policy generation module monitors system anomaly indicators (such as message accumulation, node downtime, and abnormal processing delays) in real time. Based on a policy model trained using a deep Q network, it evaluates the node state space. When a node experiences high load and anomalies, the system automatically selects a combination of actions, such as downgrading low-priority services, replica migration, and scheduling reordering, to promptly mitigate the issue and improve system recoverability and stability.

[0104] Finally, it should be noted that the above descriptions are merely preferred embodiments of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art will be able to modify the technical solutions described in the aforementioned embodiments or substitute equivalents for some of the technical features. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention shall be included within the scope of protection of the present invention.

Claims

1. A control and management method based on Kafka message queue, characterized in that: include: Based on historical message traffic data and event labels, a time feature vector is constructed. This time feature vector is used as the node input of a pre-trained graph neural network. The graph structure is constructed in combination with the Kafka Broker network topology to predict and sort the load factor of each node. Messages to be processed at each node are categorized by labels and mapped to logical priority queues, and dynamically scheduled according to the service level agreement. Based on the system resource usage indicators of each node, the scheduling control of computing resources, storage resources, and network resources is logically decoupled to determine whether to trigger expansion and contraction, replica migration, or message throttling, and then perform system resource adjustments; Deploy lightweight Kafka Broker instances on edge nodes, implement tiered storage between the edge and the center based on the access frequency of pending messages and data activity, and synchronize data through an asynchronous mechanism. Continuously monitor system abnormal indicators, build a strategy model based on reinforcement learning, automatically decide on recovery actions and perform closed-loop adjustments to achieve system self-healing; Tagging and classifying pending messages and mapping them to logical priority queues includes: Receive each pending message and extract its business type, task urgency, and data timeliness; Prioritize messages based on pre-set priority rules, including real-time, near-real-time, and batch processing; Build logical queues based on priority tags and bind messages with different priority tags to corresponding logical queues; The service level agreement determines the scheduling priority of each logical queue based on the sum of the logical priority number of the message to be processed and the load factor number of the node where it is located, and preferentially processes the queue with the largest sum of numbers; Logically decoupling the scheduling and control of computing resources, storage resources, and network resources includes: Periodically collect the computing resource utilization of the node. If the computing resource utilization exceeds the set computing resource threshold, the automatic expansion of the Pod is triggered; otherwise, the Pod reduction operation is triggered; Monitor the node's storage resource usage in real time. If the storage resource usage exceeds the set storage resource threshold, the partition replica migration is triggered. Monitor the network resource status between nodes. If the network resource status is greater than the network resource set threshold, low-priority message flow control is triggered.

2. The control and management method based on Kafka message queue according to claim 1 is characterized in that: Constructing a time feature vector includes: Obtain historical message traffic data of each Kafka Broker node and aggregate it by time granularity; Extracting event tags corresponding to the time granularity, including holiday tags, weather conditions, and advertising delivery status; Constructing a sliding time window based on the historical message traffic data and calculating statistical indicators within the past 1 hour, 6 hours, and 24 hours, including traffic mean, maximum, and volatility; The event label and the statistical indicator are concatenated to form a time feature vector of the node.

3. The control and management method based on Kafka message queue according to claim 1 is characterized in that: Edge and center tiered storage includes: Determine the dividing line between hot data and cold data based on message access frequency and data activity; Keep a complete copy of hot data on edge nodes, and store cold data only in the central cluster; The data from the edge nodes is transmitted to the center in an asynchronous and synchronous manner.

4. The control and management method based on Kafka message queue according to claim 1 is characterized in that: The strategy model uses a deep Q network, and its training process includes: Define the state space, including the node load factor, system anomaly indicators and system resource usage indicators; Define the action space, including logical priority queue reordering, system resource adjustment, and service degradation; Set up a reward function with message integrity and business continuity as positive indicators; Use historical load factors and system exception handling records as training samples to train the deep Q network; When the system is running, the current state vector of the node is collected and input into the pre-trained deep Q network, and the optimal control action is selected according to the model strategy.

5. The control and management system based on Kafka message queue is characterized by: include: The load prediction module is used to construct a time feature vector based on historical message traffic data and event labels, and use the time feature vector as the node input of the pre-trained graph neural network. It combines the Kafka Broker network topology to build a graph structure, predict the load factor of each node, and sort it; Dynamic scheduling module, used to classify the messages to be processed by each node by label and map them to logical priority queues, and dynamically schedule them according to the service level agreement; The resource decoupling module is used to logically decouple the scheduling and control of computing resources, storage resources, and network resources based on the system resource usage indicators of each node, determine whether to trigger expansion and contraction, replica migration, or message throttling, and perform system resource adjustments; The data synchronization module is used to deploy lightweight Kafka Broker instances on edge nodes, implement tiered storage between the edge and the center based on the access frequency of pending messages and data activity, and complete data synchronization through an asynchronous mechanism. The strategy generation module is used to continuously monitor system abnormal indicators, build a strategy model based on reinforcement learning, automatically decide on recovery actions and perform closed-loop adjustments to achieve system self-healing; Tagging and classifying pending messages and mapping them to logical priority queues includes: Receive each pending message and extract its business type, task urgency, and data timeliness; Prioritize messages based on pre-set priority rules, including real-time, near-real-time, and batch processing; Build logical queues based on priority tags and bind messages with different priority tags to corresponding logical queues; The service level agreement determines the scheduling priority of each logical queue based on the sum of the logical priority number of the message to be processed and the load factor number of the node where it is located, and preferentially processes the queue with the largest sum of numbers; Logically decoupling the scheduling and control of computing resources, storage resources, and network resources includes: Periodically collect the computing resource utilization of the node. If the computing resource utilization exceeds the set computing resource threshold, the automatic expansion of the Pod is triggered; otherwise, the Pod reduction operation is triggered; Monitor the storage resource usage of the node in real time. If the storage resource usage exceeds the set storage resource threshold, the partition replica migration is triggered. Monitor the network resource status between nodes. If the network resource status is greater than the set threshold of network resources, low-priority message flow control is triggered.

Images