Regulation and control management method and system based on Kafka message queue

Abstract

The invention relates to the technical field of data processing, in particular to a regulation and control management method and system based on a Kafka message queue. According to the method, a time feature vector is constructed based on historical message traffic and event tags, and a node load coefficient is predicted through a graph neural network in combination with a Kafka Broker network topology; performing queue scheduling according to the message priority and the node load coefficient, and optimizing a message processing sequence; performing decoupling management on calculation, storage and network resources according to the resource use indexes, and judging whether capacity expansion and shrinkage, copy migration or current limiting adjustment is triggered or not; a lightweight Kafka instance is deployed at an edge node, hierarchical storage of the edge and the center is realized in combination with data activeness, and data synchronization is asynchronously completed; and constructing a recovery strategy model through reinforcement learning, and realizing automatic identification and regulation response of node abnormity.

Description

Technical Field

[0001] The present invention relates to the field of data processing, and specifically to a regulation and management method and system based on Kafka message queues. Background Art

[0002] With the wide application of distributed stream processing systems in scenarios such as finance, e-commerce, and industrial Internet, Kafka, as a high-throughput and scalable message queue middleware, has become one of the core data communication components. Kafka exhibits good performance in processing large-scale data streams, supports publish / subscribe mode, partition mechanism, and replica disaster recovery strategy, and is suitable for high-concurrency and highly available data collection and transmission requirements.

[0003] In the actual production environment, the operating efficiency and system stability of Kafka are limited by multiple factors, such as uneven node load caused by message traffic fluctuations, system throughput limitations due to resource bottlenecks, data transmission latency in edge scenarios, and recovery decisions during abnormal events. To improve the adaptability and robustness of the Kafka system, the industry has proposed a series of optimization strategies, including rule-based scheduling algorithms, Pod scaling mechanism, hot partition replica migration, and edge storage deployment.

[0004] However, in the face of complex and changing business environments and system conditions with dynamic resource changes, most existing methods rely on static threshold configuration or manual intervention, making it difficult to achieve highly automated and intelligent resource regulation. In addition, the scheduling priority during system operation is difficult to reflect the business urgency and node load in real time, resulting in some high-priority messages not being processed in a timely manner when resources are limited, affecting the overall service quality. At the same time, existing systems still lack intelligent optimization capabilities at the policy level in terms of fault response and self-healing, and the recovery process is not efficient enough to support higher-level service continuity requirements.

[0005] Therefore, a regulation and management method and system based on Kafka message queues are proposed. Summary of the Invention

[0006] The present invention provides a regulation and management method and system based on Kafka message queues, which predicts node load by constructing time feature vectors combined with graph neural networks, realizes dynamic scheduling by using a priority queue and service level agreement, and uses a reinforcement learning policy model to achieve abnormal self-healing, improving system stability and processing efficiency.

[0007] To achieve the above object, the present invention provides the following technical solutions: A regulation and management method based on Kafka message queues, comprising: Construct a time feature vector based on historical message traffic data and event tags, and use the time feature vector as the node input of a pre-trained graph neural network. Combine the Kafka Broker network topology to construct a graph structure, predict the load coefficients of each node and sort them; Classify the tags of the messages to be processed by each node and map them to a logical priority queue, and perform dynamic scheduling according to the service level agreement; According to the system resource usage metrics of each node, logically decouple the scheduling control of computing resources, storage resources, and network resources, determine whether to trigger scaling, replica migration, or message rate limiting, and execute system resource adjustment; Deploy lightweight Kafka Broker instances on edge nodes, realize hierarchical storage between the edge and the center in combination with the access frequency and data activity of the messages to be processed, and complete data synchronization through an asynchronous mechanism; Continuously monitor system anomaly metrics, build a policy model based on reinforcement learning, automatically make decisions to execute recovery actions and perform closed-loop adjustment to achieve system self-healing.

[0008] Furthermore, constructing the time feature vector includes: Obtain the historical message traffic data of each Kafka Broker node and perform aggregation processing according to the time granularity; Extract the corresponding event tags within the time granularity, including holiday markers, weather conditions, and advertising placement status; Construct a sliding time window based on the historical message traffic data, and calculate the statistical metrics within the past 1 hour, 6 hours, and 24 hours, including traffic mean, maximum value, and volatility; Concatenate the event tags and the statistical metrics to form the time feature vector of the node.

[0009] Furthermore, classifying the tags of the messages to be processed and mapping them to a logical priority queue includes: Receive each message to be processed and extract its business type, task urgency, and data timeliness; Label the messages with priority tags according to the preset priority rules, including real-time, near-real-time, and batch processing; Construct a logical queue according to the priority tags, and bind the messages with different priority tags to the corresponding logical queues.

[0010] Furthermore, the service level agreement determines the scheduling priority of each logical queue according to the sum of the logical priority numbers of the messages to be processed and the load coefficient numbers of the nodes where they are located, and preferentially processes the one with the largest sum of numbers.

[0011] Furthermore, logically decoupling the scheduling control of computing resources, storage resources, and network resources includes: Periodically collect the computing resource utilization rate of nodes. If the computing resource utilization rate is greater than the set threshold of computing resources, trigger the automatic scaling-out operation of Pods; otherwise, trigger the scaling-in operation of Pods. Real-time monitor the storage resource usage rate of nodes. If the storage resource usage rate is greater than the set threshold of storage resources, trigger the partition replica migration. Monitor the network resource status among nodes. If the network resource status is greater than the set threshold of network resources, trigger the traffic throttling of low-priority messages.

[0012] Furthermore, the edge and center hierarchical storage includes: Determine the boundary between hot data and cold data according to the message access frequency and data activity. Keep the complete copies of hot data on edge nodes, and only store cold data in the central cluster. Use the asynchronous synchronization method to transfer the data of edge nodes to the center.

[0013] Furthermore, the policy model adopts a deep Q network, and its training process includes: Define the state space, including the load factor of nodes, system anomaly indicators, and system resource usage indicators. Define the action space, including the reordering of the logical priority queue, system resource adjustment, and service degradation. Set the reward function, with message integrity and business continuity as positive indicators. Use the historical load factor and system anomaly handling records as training samples to train the deep Q network model. During the system operation, continuously input the current state vector of nodes, and select the optimal control action according to the model policy.

[0014] The present invention also proposes a regulation and management system based on the Kafka message queue, including: A load prediction module, which is used to construct a time feature vector based on historical message traffic data and event labels, use the time feature vector as the node input of the pre-trained graph neural network, combine the Kafka Broker network topology to construct a graph structure, predict the load factor of each node and sort them. A dynamic scheduling module, which is used to classify the labels of the messages to be processed by each node and map them to the logical priority queue, and perform dynamic scheduling according to the service level agreement. A resource decoupling module, which is used to logically decouple the scheduling control of computing resources, storage resources, and network resources according to the system resource usage indicators of each node, judge whether to trigger scaling, replica migration, or message throttling, and perform system resource adjustment. The data synchronization module is used to deploy lightweight Kafka Broker instances at edge nodes, achieve hierarchical storage between the edge and the center by combining the access frequency of messages to be processed and data activity, and complete data synchronization through an asynchronous mechanism; The policy generation module is used to continuously monitor system anomaly metrics, build a policy model based on reinforcement learning, automatically make decisions to execute recovery actions and perform closed-loop adjustment to achieve system self-healing.

[0015] The beneficial effects of the present invention are as follows: 1. The present invention constructs a time feature vector by combining historical message traffic data and event tags, and uses it as the node input of the graph neural network, which is fused with the Kafka cluster topology structure, so as to accurately predict the load levels of each node and sort them. This method not only enhances the ability of correlation modeling between nodes, but also provides a reliable basis for subsequent scheduling and resource allocation, improving the load balancing and response efficiency of the system.

[0016] 2. The present invention continuously monitors the anomaly metrics of the system, builds a policy model using a deep Q-network, and can automatically make optimal regulation decisions and perform closed-loop adjustment in the case of node anomalies, resource bottlenecks, etc. This mechanism has the ability of continuous learning and optimization, which helps to improve the system's autonomous decision-making and stable operation capabilities in complex scenarios.

[0017] 3. The present invention constructs a Kafka regulation and management system including load prediction, dynamic scheduling, resource decoupling, data synchronization and policy generation through modular design, covering the entire process of message flow processing, resource scheduling and anomaly recovery, and has good scalability and flexibility. The system supports hierarchical deployment between the edge and the center, and performs refined scheduling by combining logical priorities and service level agreements, and can improve the overall processing efficiency and system stability in high-concurrency and large-scale data processing scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The drawings are used to provide a further understanding of the present invention, and constitute a part of the specification. They are used together with the embodiments of the present invention to explain the present invention, and do not constitute a limitation to the present invention. In the drawings: Figure 1 is a flowchart of the regulation and management method based on the Kafka message queue provided by the present invention; Figure 2 is a schematic structural diagram of the regulation and management system based on the Kafka message queue provided by the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0019] The following describes the preferred embodiments of the present invention with reference to the drawings. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present invention, and are not used to limit the present invention. Embodiment 1

[0020] A regulation management method based on Kafka message queue, as Figure 1 shown, includes: S100: Construct a time feature vector based on historical message traffic data and event tags, and use the time feature vector as the node input of a pre-trained graph neural network. Combine the Kafka Broker network topology to construct a graph structure, predict the load coefficients of each node and sort them; Further, constructing the time feature vector includes: Obtain the historical message traffic data of each Kafka Broker node and perform aggregation processing according to the time granularity; Extract the corresponding event tags within the time granularity, including holiday marks, weather conditions, and advertising placement status; Construct a sliding time window based on the historical message traffic data, and calculate statistical metrics within the past 1 hour, 6 hours, and 24 hours, including traffic mean, maximum value, and volatility; Concatenate the event tags and the statistical metrics to form the time feature vector of the node.

[0021] Specifically, the system regularly collects the message traffic data of each Kafka Broker node in the historical period from the Kafka cluster. The message traffic data includes performance metrics such as the receiving rate, sending rate, and message processing volume per unit time. According to the preset time granularity (for example: every 1 minute, every 5 minutes, or every 15 minutes), the above data is aggregated to generate time series data aligned by timestamp. To enhance the perception ability of the time feature vector to external context changes, the system obtains event tags aligned with the time granularity from the business side or external services. Event tags include: holiday marks, used to indicate whether the current time is during legal holidays, weekends, or important events; weather conditions, used to collect the weather characteristics of the current city, such as sunny, rainy, or snowy; advertising placement status, used to identify whether there are online advertisements or major promotion activities launched during the corresponding period. Such events usually significantly affect the traffic level. Each type of tag is encoded in the form of a numerical vector. For example, the holiday mark is represented by 1 for holidays and 0 for weekdays; the weather condition can be represented as a binary vector of a fixed dimension through one-hot encoding. For example, sunny is represented by 0001. To capture the traffic change trends at different time scales, the system calculates the following statistical metrics based on the sliding time window mechanism: the traffic mean, maximum value, and standard deviation within the past 1 hour, 6 hours, and 24 hours. Concatenate the event tag vector and the statistical feature vector to form a complete set of node time feature vectors.

[0022] By fusing historical message traffic data with event tags and combining a multi-scale statistical feature extraction method with a sliding window, it is possible to comprehensively characterize the load behavior of Kafka nodes in different time periods and enhance the model's perception of the impact of external events. After standardizing these features and inputting them into a graph neural network, the accuracy and robustness of load prediction are improved, providing reliable data support for subsequent dynamic scheduling and resource regulation.

[0023] Furthermore, based on the deployment and connection relationships of each Broker node in the Kafka cluster, a network topology graph is constructed, where each node corresponds to a Kafka Broker and each edge represents the communication link or replica relationship between Broker nodes. Subsequently, the time feature vector constructed in the previous step is used as the feature input for each node, and a pre-trained graph neural network model is introduced for training and prediction. In the training phase, the node load data collected historically is used as the label, and the graph neural network is pre-trained through supervised learning. During deployment and operation, the time feature vectors of each node at the current moment are input into the pre-trained graph neural network model to obtain the standardized predicted load coefficient for each node, and the nodes are sorted according to their load coefficients.

[0024] S200: Classify the messages to be processed by each node and map them to a logical priority queue, and perform dynamic scheduling according to the service level agreement; Furthermore, classifying the messages to be processed and mapping them to a logical priority queue includes: Receive each message to be processed and extract its service type, task urgency, and data timeliness;

[0025] Assign a priority label to the message according to the preset priority rules, including real-time, near real-time, and batch processing; Construct a logical queue according to the priority label and bind the messages with different priority labels to the corresponding logical queues.

[0026] Specifically, the Kafka message processing module first listens to the message stream to be processed. For each received message, it extracts the context information related to the business, mainly including: business type, such as transactions, logs, monitoring, orders, etc.; task urgency, which indicates the sensitivity of the task to the processing time limit and is divided into high, medium, and low; data timeliness, which refers to whether there is an expiration limit for the message content in the business process. The system combines the above three attributes to form a priority label according to the preset priority rules. In a feasible implementation, the priority rules are defined as follows: real-time class (high priority): the business type is transaction or alarm, the task urgency is high, and the data timeliness is less than 1 minute; quasi-real-time class (medium priority): such as monitoring messages, the task urgency is medium, and the data timeliness is between 1 and 10 minutes; batch processing class (low priority): such as periodic tasks like log archiving and report generation, with weak data timeliness or no time constraint.

[0027] According to the message priority label, the system maps the message to the corresponding logical priority queue.

[0028] By extracting the business type, task urgency, and data timeliness of the message to be processed, classifying the message into priority labels such as real-time, quasi-real-time, and batch processing according to the preset rules, and mapping it to the corresponding logical priority queue, the recognition and hierarchical management of the importance and timeliness of the message are realized.

[0029] Furthermore, the service level agreement determines the scheduling priority of each logical queue according to the sum of the logical priority number of the message to be processed and the load factor number of the node where it is located, and gives priority to processing the one with the largest sum of the numbers.

[0030] Specifically, a logical priority number is assigned to each message to be processed. This number can be defined according to the priority label to which the message belongs. For example: the real-time message number is 3, the quasi-real-time is 2, and the batch processing is 1. The larger this number, the higher the importance and urgency of the message. Then, combined with the running state of the current Kafka Broker node, the load factor number of this node is obtained. This number can be obtained by sorting the load factors predicted by the graph neural network. The load factor number indicates the more serious the message congestion on this node. Then, the logical priority number and the load factor number of the node where it is located are summed to obtain a comprehensive scheduling priority value. The system sorts according to the comprehensive priority values corresponding to all messages to be processed, and preferentially schedules the message with the largest sum of the numbers, so as to improve the resource utilization efficiency and relieve the system bottleneck on the premise of taking into account both the business importance and the node load status.

[0031] By comprehensively calculating the logical priority number of a message and the load factor number of its node, a quantitative calculation of the scheduling policy between business importance and system operating status is achieved. Compared with the traditional method of making decisions based on a single dimension of priority or resource metrics, this method can preferentially schedule messages that have a greater impact on the system and are more critical to business continuity in the cross-scenario of high-priority messages and high-load nodes, alleviating node congestion, reducing latency, and improving system throughput efficiency.

[0032] S300: According to the system resource usage metrics of each node, logically decouple the scheduling control of computing resources, storage resources, and network resources, determine whether to trigger scaling, replica migration, or message rate limiting, and perform system resource adjustment; Furthermore, logically decoupling the scheduling control of computing resources, storage resources, and network resources includes: Periodically collect the computing resource utilization rate of the node. If the computing resource utilization rate is greater than the set threshold of computing resources, trigger the automatic expansion operation of the Pod; otherwise, trigger the Pod scaling-down operation; Real-time monitor the storage resource usage rate of the node. If the storage resource usage rate is greater than the set storage resource threshold, trigger partition replica migration; Monitor the network resource status between nodes. If the network resource status is greater than the set network resource threshold, trigger low-priority message rate limiting.

[0033] Specifically, the system obtains the CPU usage rate or memory usage rate of the current node at regular time intervals and compares it with the preset upper limit threshold of computing resources. If the computing resource utilization rate exceeds this threshold (e.g., 70%), trigger the automatic expansion mechanism of Kubernetes to add Kafka Pod replicas to share the load; if the resource utilization rate is lower than the scaling-down threshold (e.g., 30%) for a continuous period of time, trigger Pod scaling-down to release resources. Real-time monitor the usage rate of the disk where the Kafka partition is located. When the disk usage rate of a certain node exceeds the set threshold (such as 80%), the system migrates the partition replicas that occupy a large amount of space in the current node to a node with rich disk resources; the replica migration is executed by the Kafka Controller or an extended control component, and the replica status during the migration process is recorded to ensure data integrity and consistency. Collect network throughput, latency, packet loss rate and other metrics between nodes. If the network resource metrics exceed the set threshold (such as the export bandwidth usage rate exceeding 90%), the system will start the message priority rate-limiting mechanism; the rate-limiting policy preferentially regulates the message queues with low logical priorities to reduce network pressure and ensure that the transmission requirements of high-priority services are not affected.

[0034] By logically decoupling the scheduling control of computing resources, storage resources, and network resources, the system can separately identify and respond to different types of resource bottlenecks, enabling a more targeted and flexible resource management strategy. This method avoids misjudgment and scheduling conflicts caused by resource interference, improving the resource utilization and operational stability of the Kafka cluster in high-concurrency and multi-service scenarios.

[0035] S400: Deploy lightweight Kafka Broker instances on edge nodes, implement hierarchical storage between the edge and the center in combination with the access frequency and data activity of the messages to be processed, and complete data synchronization through an asynchronous mechanism; Furthermore, the hierarchical storage between the edge and the center includes: Determine the dividing line between hot data and cold data based on the message access frequency and data activity; Keep a complete copy of the hot data on the edge nodes, and only store the cold data in the central cluster; Use an asynchronous synchronization method to transfer the data on the edge nodes to the center.

[0036] Specifically, during the operation of the Kafka Broker, the system continuously records the access frequency and the most recent active time of each message, and comprehensively calculates the "data activity" value of the message. The access frequency can be obtained by counting the number of times each message is read by consumers within a unit of time, and the data activity is evaluated through the access frequency and the access time interval. In a feasible implementation, the data activity , where represents the access frequency of the message within a unit of time, represents the time interval since the last access of the message, and represent the weights of the access frequency and the time interval respectively. Then, based on the access frequency distribution and activity statistics over a period of time, all messages are sorted, and the quantile algorithm is used to demarcate the dividing line between hot data and cold data. For example, the messages with the top 20% access frequency and activity can be classified as "hot data", and the rest as "cold data". For the identified hot data, the system will keep a complete copy of it in the lightweight Kafka Broker instance on the edge nodes to ensure low-latency access to hot messages by the edge nodes; while the cold data is only stored in the central cluster to avoid excessive storage space occupation by edge resources. To ensure data consistency, an asynchronous synchronization mechanism is adopted between the edge nodes and the central node: the edge nodes record the change information in the local log queue and periodically synchronize the message data to the central cluster through incremental replication.

[0037] By evaluating the access frequency of messages and the data activity, hot data and cold data are effectively distinguished. Copies of data with high access frequency and high real-time requirements are retained at the edge nodes to reduce data access latency and improve the system response speed. At the same time, cold data is centrally stored in the central cluster to save edge storage resources and reduce the burden of data redundancy. The data consistency between the edge and the center is maintained through asynchronous synchronization to achieve the collaborative optimization of resource utilization and access performance, and enhance the scalability and scheduling flexibility of the system.

[0038] S500: Continuously monitor the system's abnormal indicators, build a policy model based on reinforcement learning, automatically make decisions to execute recovery actions and perform closed-loop adjustment to achieve system self-healing.

[0039] Furthermore, the policy model adopts a deep Q network, and its training process includes: Define the state space, including the load factor of nodes, system abnormal indicators, and system resource usage indicators; Define the action space, including reordering the logical priority queue, adjusting system resources, and service degradation; Set the reward function, with message integrity and business continuity as positive indicators; Use the historical load factor and system abnormal handling records as training samples to train the deep Q network; During system operation, collect the current state vector of nodes and input it into the pre-trained deep Q network, and select the optimal control action according to the model policy.

[0040] Specifically, define the state vector of the system as the input of the model, which contains the following information items: the load factor of each KafkaBroker node, which can be obtained from the output result of the prediction module; system abnormal indicators, including node downtime rate, response timeout, retry frequency, alarm log density, etc.; system resource usage indicators, including computing resources, storage resources, and network resources. Define the adjustable control operations (i.e., actions) that the model can select, including but not limited to: reordering the logical priority queue, such as adjusting the priority of specific logical queues to relieve the pressure on high-load nodes; adjusting system resources, such as scaling Pods, migrating partition replicas, and message rate limiting; service degradation control, such as suspending low-priority consumption, closing non-core modules, and enabling standby nodes. Oriented by the system self-healing effect, set the reward function in reinforcement learning. When the control action improves message integrity and business continuity, give a positive reward; otherwise, give a penalty term. Among them, message integrity can be obtained by comparing the message counts at the production end and the consumption end through the consumption offset management mechanism of Kafka itself. Its calculation method is: , where, represents the message integrity index, represents the message count at the production end, Indicates the consumer - side message count. Business continuity calculates latency through an access - link tracing system (such as Zipkin, Jaeger), and its calculation method is: , where represents the business - continuity metric, represents the time point when the consumer - side obtains the message, represents the time point when the producer - side sends the message. The reward function is designed as: , where represents the reward function, and are preset weights, represents the current time point for calculating message integrity and business continuity, represents the time point for calculating message integrity and business continuity in the previous round. Training samples are extracted from the historical operation data of the system. The samples are composed of (state vector, executed action, reward value, next - state vector). In actual operation, the system continuously extracts the current state vector. The policy model adjusts and selects the optimal action according to the current state. The system executes the action and monitors its effect, and feeds the results back to the model for continuous training and adjustment to achieve closed - loop optimization.

[0041] By introducing a reinforcement - learning policy model based on the deep Q - network, the system can automatically learn and execute optimal control actions in complex states, realizing intelligent management of the Kafka message queue. This method effectively improves the real - time performance and accuracy of control decisions, optimizes core performance indicators such as message integrity and business continuity, and enhances the system's adaptability and self - healing ability in abnormal scenarios. Embodiment 2

[0042] This embodiment provides a specific application method of a control and management system based on the Kafka message queue to illustrate the collaborative working process of each functional module in the system. The system structure is as shown in Figure 2 . The system is deployed in an urban - level intelligent transportation platform and is used to process a large number of asynchronous messages from various sensors, intersection cameras, and traffic signal devices.

[0043] During operation, the load - prediction module first collects the historical message - traffic data of each Kafka Broker node in the past 7 days, extracts the event tags (such as peak periods, bad weather, holidays, etc.) for the corresponding time period, and generates a time - feature vector based on a sliding time window, and inputs it into the graph - neural - network model. Combining the Kafka - cluster network topology, the model predicts the load coefficients of each node, sorts the node loads from high to low, and assists subsequent scheduling and resource - management decisions.

[0044] The dynamic scheduling module extracts the task urgency and data timeliness according to the service types carried by each message (such as traffic violation detection, signal light control, real-time traffic condition feedback), divides them into three levels: real-time, quasi-real-time, and batch processing, and maps them to the corresponding logical priority queues. The system calculates the comprehensive priority number of each queue according to the service level agreement, and preferentially schedules the message queue with lighter load and higher priority for processing.

[0045] The resource decoupling module regularly collects the CPU, memory, disk, and network utilization rates of Kafka nodes. When the CPU utilization rate of a node exceeds the set threshold (such as 80%), it automatically triggers the expansion of Pods; if the storage utilization rate exceeds the threshold, it automatically initiates the migration operation of the corresponding Topic partition replicas; if the network bandwidth between nodes is saturated, it limits the traffic of low-priority batch processing messages to ensure the transmission of high-priority tasks.

[0046] The data synchronization module deploys lightweight Kafka Broker edge instances at transportation hubs to cache the data collected by local cameras and sensors. The system determines which data are hot data (such as sudden congestion information) based on the access frequency and data activity, and retains its complete copy at the edge node, while transmitting the cold data back to the central cluster for archiving through an asynchronous mechanism.

[0047] The policy generation module monitors the system anomaly metrics in real time (such as message backlog, node downtime, processing delay anomaly, etc.), and evaluates the node state space based on the policy model trained by the deep Q network. When a node has a high load and is accompanied by abnormal backlog, the system can automatically select a combination of actions such as "degrading low-priority services + replica migration + scheduling reordering" to relieve the problem in a timely manner and improve the recoverability and stability of the system.

[0048] Finally, it should be noted that the above are only the preferred embodiments of the present invention and are not used to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements for some of the technical features. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A regulation management method based on Kafka message queue, characterized in that Including: Construct a time feature vector based on historical message traffic data and event tags, and use the time feature vector as the node input of a pre-trained graph neural network. Combine the Kafka Broker network topology to construct a graph structure, predict the load coefficients of each node, and sort them. Classify the tags of the messages to be processed at each node and map them to a logical priority queue, and perform dynamic scheduling according to the service level agreement. According to the system resource usage metrics of each node, logically decouple the scheduling control of computing resources, storage resources, and network resources, determine whether to trigger scaling, replica migration, or message rate limiting, and execute system resource adjustment. Deploy lightweight Kafka Broker instances at edge nodes, realize hierarchical storage between the edge and the center by combining the access frequency and data activity of the messages to be processed, and complete data synchronization through an asynchronous mechanism. Continuously monitor system anomaly metrics, build a policy model based on reinforcement learning, automatically make decisions to execute recovery actions and perform closed-loop adjustment to achieve system self-healing.

2. The regulation management method based on the Kafka message queue according to claim 1, characterized in that, Constructing a time feature vector includes: Obtain the historical message traffic data of each Kafka Broker node and perform aggregation processing according to the time granularity. Extract the corresponding event tags within the time granularity, including holiday marks, weather conditions, and advertising placement status. Construct a sliding time window based on the historical message traffic data, and calculate the statistical metrics within the past 1 hour, 6 hours, and 24 hours, including traffic mean, maximum value, and volatility. Concatenate the event tags and the statistical metrics to form the time feature vector of the node.

3. The regulation management method based on Kafka message queue according to claim 1, characterized in that Classifying the tags of the messages to be processed and mapping them to a logical priority queue includes: Receive each message to be processed and extract its service type, task urgency, and data timeliness. Label the messages with priority tags according to the preset priority rules, including real-time, near real-time, and batch processing. Construct a logical queue according to the priority tags, and bind the messages with different priority tags to the corresponding logical queues.

4. The regulation management method based on the Kafka message queue according to claim 1, characterized in that, The service level agreement determines the scheduling priority of each logical queue according to the sum of the logical priority number of the message to be processed and the load coefficient number of the node where it is located, and preferentially processes the one with the largest sum of numbers.

5. The regulation management method based on the Kafka message queue according to claim 1, characterized in that Logically decouple the scheduling control of computing resources, storage resources, and network resources includes: Periodically collect the computing resource utilization rate of the node. If the computing resource utilization rate is greater than the set threshold of the computing resource, trigger the automatic scaling-out operation of the Pod; otherwise, trigger the scaling-in operation of the Pod. Real-time monitor the storage resource usage rate of the node. If the storage resource usage rate is greater than the set threshold of the storage resource, trigger partition replica migration. Monitor the network resource status between nodes. If the network resource status is greater than the set threshold of the network resource, trigger low-priority message rate limiting.

6. The regulation management method based on Kafka message queue according to claim 1, characterized in that Hierarchical storage between the edge and the center includes: Determine the boundary between hot data and cold data according to the message access frequency and data activity. Keep the complete copies of hot data at edge nodes, and only store cold data in the central cluster. Use an asynchronous synchronization method to transfer the data of edge nodes to the center.

7. The regulation management method based on Kafka message queue according to claim 1, characterized in that The policy model uses a deep Q network, and its training process includes: Define the state space, including the node load factor, system exception indicators, and system resource usage indicators; Define the action space, including logical priority queue reordering, system resource adjustment, and service degradation; Set the reward function, with message integrity and business continuity as positive indicators; Use historical load factors and system exception handling records as training samples to train the deep Q network; During system operation, collect the current state vector of the node and input it into the pre-trained deep Q network, and select the optimal control action according to the model strategy.

8. A regulation and management system based on Kafka message queue, characterized in that, Include: A load prediction module, which is used to construct a time feature vector based on historical message traffic data and event labels, and use the time feature vector as the node input of the pre-trained graph neural network. Combine the Kafka Broker network topology to construct a graph structure, predict the load factor of each node and sort them; A dynamic scheduling module, which is used to classify the tags of the messages to be processed by each node and map them to the logical priority queue, and perform dynamic scheduling according to the service level agreement; A resource decoupling module, which is used to logically decouple the scheduling control of computing resources, storage resources, and network resources according to the system resource usage indicators of each node, judge whether to trigger scaling, replica migration, or message rate limiting, and execute system resource adjustment; A data synchronization module, which is used to deploy lightweight Kafka Broker instances on edge nodes, realize hierarchical storage of edge and center combined with the access frequency and data activity of the messages to be processed, and complete data synchronization through an asynchronous mechanism; A policy generation module, which is used to continuously monitor system exception indicators, construct a policy model based on reinforcement learning, automatically decide to execute recovery actions and perform closed-loop adjustment to achieve system self-healing.

Images