Message-in-message station resource scheduling method, device thereof, equipment and medium

Abstract

The invention discloses a message channel resource scheduling method and device, equipment and a medium, and the method comprises the steps: obtaining a business isolation list which comprises business types, business priorities and resource groups corresponding to a plurality of businesses, and the setting degree of each resource group is matched with the upper limit, lower limit and health degree of a health index of the business priority; dynamically adjusting the health degree of the message node according to a plurality of real-time load indexes of the message node in the resource group; querying a service isolation list according to a service type in the service theme creation request to determine a target resource group, wherein a plurality of target nodes of which the health degrees are ranked in the front and the health degrees are higher than corresponding upper limits are determined in the target resource group to create a partition of a theme specified by the request; and when the total health degree of each resource group is lower than the corresponding lower limit, preferentially preempting the message node for the resource group with higher service priority according to the service priority. According to the invention, resource scheduling is carried out by an ingenious strategy to realize load balancing, and the stability of the middle station can be improved.

Description

Technical Field

[0001] The present application relates to network system resource optimization, and in particular to a message middle station resource scheduling method and its device, equipment, and medium. Background Art

[0002] In today's Internet era, various online business systems such as risk control business systems, advertising business systems, online live broadcast systems, e-commerce platforms, etc., are extremely dependent on the message platform to process and transmit large amounts of data. As the core component of these systems, the message platform is responsible for ensuring the timely delivery and processing of information. However, with the continuous growth of business volume and the diversification of business types, traditional message platform technology faces many challenges.

[0003] First, traditional resource scheduling methods often lack flexibility and scalability. In a multi-business environment, fixed resource allocation patterns are difficult to adapt to the rapid changes in business needs, resulting in low resource utilization and even some nodes being overloaded while other nodes are idle. This load imbalance problem seriously affects the stability and performance of the system.

[0004] Secondly, traditional technologies are also unable to cope with different business priorities. Critical and non-critical businesses are often mixed together and share the same resources, which may cause performance impairment of critical businesses under high load and cannot meet business needs with strict requirements on real-time and stability.

[0005] In addition, as the business scale expands, traditional technologies are also stretched in resource group management. How to effectively divide and expand resource groups to adapt to changing business needs has become a difficult problem. Especially during business peaks, how to quickly respond to changes in resource demand and maintain high system availability and service continuity is a problem that existing technologies cannot effectively solve.

[0006] In this context, although some systems try to alleviate the pressure by increasing hardware resources, this not only increases costs but also fails to fundamentally solve the problem. Summary of the invention

[0007] The purpose of the present application is to solve the above-mentioned problems and to provide a message middle station resource scheduling method and its corresponding device, equipment, and non-volatile readable storage medium.

[0008] According to one aspect of the present application, a message center resource scheduling method is provided, comprising the following steps:

[0009] Obtain a business isolation list, the business isolation list including business types, business priorities and resource groups corresponding to multiple businesses, each resource group including multiple message nodes for distributed storage of message flows of each topic of its corresponding business, and each resource group having a matching health indicator upper limit, health indicator lower limit and initialized health degree corresponding to its business priority;

[0010] According to multiple real-time load indicators of the message nodes in each resource group, the health of each message node is dynamically adjusted, and the order of each message node in the node queue corresponding to each resource group is adjusted according to the adjusted health;

[0011] In response to a business topic creation request, query the business isolation list according to the business type in the request to determine the corresponding target resource group, determine multiple message nodes with a high health ranking and a health level higher than the corresponding health indicator upper limit as target nodes in the node queue of the target resource group, and create a partition of the topic specified by the request between the target nodes;

[0012] The overall health of the resource group is determined based on the health of all message nodes in each resource group. When the overall health is lower than the corresponding lower limit of the health indicator, new message nodes are configured from the resource pool for resource groups with higher business priorities according to the business priorities of the resource groups.

[0013] According to another aspect of the present application, a message center resource scheduling device is provided, including:

[0014] A list acquisition module is configured to acquire a business isolation list, wherein the business isolation list includes business types, business priorities, and resource groups corresponding to multiple businesses, each resource group includes multiple message nodes for distributed storage of message flows of each topic of its corresponding business, and each resource group is provided with a matching health indicator upper limit, health indicator lower limit, and initialized health degree corresponding to its business priority;

[0015] A sorting and scheduling module is configured to dynamically adjust the health of each message node according to multiple real-time load indicators of the message nodes in each resource group, and adjust the sorting of each message node in the node queue corresponding to each resource group according to the adjusted health;

[0016] A topic creation module is configured to respond to a business topic creation request, query the business isolation list according to the business type in the request to determine the corresponding target resource group, determine multiple message nodes with a high health ranking and a health level higher than the corresponding health indicator upper limit as target nodes in the node queue of the target resource group, and create a partition of the topic specified by the request between the target nodes;

[0017] The resource expansion module is configured to determine the overall health of the resource group based on the health of all message nodes in each resource group. When the overall health is lower than the corresponding lower limit of the health indicator, new message nodes are configured from the resource pool for resource groups with higher business priorities according to the business priorities of the resource groups.

[0018] According to another aspect of the present application, a message middle station resource scheduling device is provided, including a central processing unit and a memory, wherein the central processing unit is used to call and run a computer program stored in the memory to execute the steps of the message middle station resource scheduling method described in the present application.

[0019] According to another aspect of the present application, a non-volatile readable storage medium is provided, which stores a computer program implemented according to the message middle station resource scheduling method in the form of computer-readable instructions, and when the computer program is called and executed by a computer, the steps included in the method are executed.

[0020] Compared with traditional technologies, this application achieves efficient management of business isolation and load balancing by introducing health monitoring and upper and lower limits of health indicators in the resource scheduling of the message middle station. This application cleverly uses the same health indicator to serve both the expansion decision at the business level and the message node sorting at the business theme level for node scheduling, thereby meeting the needs at different levels. At the business level, the system intelligently judges and prioritizes the expansion of high-priority businesses based on the overall health of the message nodes in the resource group and the lower limit of the health indicator to ensure the performance and stability of key businesses. At the business theme level, the system sorts the health of each message node in the resource group and schedules the optimal message node for each business theme, thereby achieving efficient storage and processing of message flows. This method not only improves the flexibility and accuracy of resource scheduling, but also improves the automation and intelligence level of the system, and effectively solves the load balancing problem of the message middle station in the face of multi-business and high-concurrency scenarios. Through this systematic approach, this application achieves the optimal allocation and utilization of resources while ensuring business isolation, significantly improving the overall performance and stability of the message middle station. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 A flow chart of an embodiment of a message middle station resource scheduling method of the present application;

[0022] Figure 2 This is a principle block diagram of the message middle station resource scheduling device of this application;

[0023] Figure 3 A structural diagram of a message middle station resource scheduling device adopted in this application. DETAILED DESCRIPTION

[0024] In order to facilitate the understanding of the various embodiments of the present application, a network architecture of the message middle station of the present application is illustrated as follows. This message middle station is designed as a highly available and scalable system, aiming to provide reliable message stream processing services for different types of business systems. The core of the network architecture is a central message middle station server cluster, which consists of multiple message processing nodes, each of which can store and process message streams of different business topics in a distributed manner.

[0025] In this architecture, the message middleware server cluster is further divided into multiple resource groups, each of which is targeted at a specific business type or business priority, and there is a one-to-one correspondence between business type and business priority. For example, one resource group may specifically serve high-priority risk control system messages, while another resource group may serve regular priority e-commerce platform messages. For another example, one resource group can serve order messages that the e-commerce platform needs to respond quickly, and another resource group can serve other messages that the e-commerce platform does not need to process quickly. Each resource group contains multiple message nodes, which are managed according to the upper and lower limits of health indicators defined in the business isolation list.

[0026] The front-end server of the message center is responsible for receiving configuration information and business topic creation requests from different business systems. When a new business system needs to be connected to the message center, the front-end server will map it to the corresponding resource group according to the business type and business priority of the new business, and create a topic partition for the new business in the resource group. In this process, it is necessary to monitor the health of the message nodes in each resource group in real time, and optimize resource allocation based on the dynamic adjustment of the health.

[0027] In addition, to achieve resource scheduling, the message center also includes an intelligent resource scheduling module, which is responsible for dynamically adjusting the size of the resource group according to changes in business load. When the overall health of a resource group is lower than the preset lower threshold, the module will trigger the resource expansion process to seize or add message nodes from the resource pool to ensure load balancing and high availability of the system.

[0028] To achieve these functions, each message node in the message middleware runs a monitoring agent, which is responsible for collecting the real-time load indicators of the node and sending these data to the central monitoring system. The central monitoring system uses these data to perform anomaly detection, calculate the health probability of each load indicator, and adjust the weight of each indicator accordingly. These adjusted weights are used to calculate the health of the message node, thereby achieving refined resource scheduling.

[0029] Overall, the network architecture of this message center is designed to provide efficient and stable message flow processing services for various business systems through dynamic resource scheduling and business isolation, while ensuring the performance and stability of key businesses. Through this architecture, the message center can adapt to changing business needs and achieve optimal resource allocation and load balancing.

[0030] The resource scheduling process of the message middle station can be implemented by the message middle station resource scheduling method of the present application. By implementing the message middle station resource scheduling method as a computer program product, installing it in the server of the message middle station, and forming a message middle station system, the purpose of resource scheduling can be achieved.

[0031] See also Figure 1 According to a message middle station resource scheduling method provided by the present application, a computer program product can be installed and run in a server to provide a message middle station. In some embodiments thereof, the following steps are included:

[0032] Step S3100: Obtain a business isolation list, wherein the business isolation list includes business types, business priorities, and resource groups corresponding to multiple businesses, each resource group includes multiple message nodes for distributed storage of message flows of each topic of its corresponding business, and each resource group is set with a matching health indicator upper limit, health indicator lower limit, and initialized health degree corresponding to its business priority;

[0033] The business isolation list is a core data structure in the resource scheduling method of the message middle platform. It contains relevant information of multiple businesses to guide the allocation and scheduling of resources. The list covers three key data: business type, business priority, and resource group. Business type is a classification identifier of the nature of the business, such as a risk control system, an advertising business system, or a live broadcast system; business priority is a priority level set according to the different performance and stability requirements of the business, usually expressed in numerical values. High-priority businesses generally have stricter requirements for real-time and stability; resource groups are a collection of message nodes, which are allocated to businesses of a specific business type to achieve distributed storage and processing of business messages. Different resource groups achieve business isolation by serving different business types respectively.

[0034] Each resource group has a series of health indicators, including the upper limit of health indicators, the lower limit of health indicators and the initial health. These health indicators can also be used by each message node in the resource group. The upper limit of health indicators is an indicator used to measure whether the message node has sufficient load redundancy, while the lower limit of health indicators is the lowest threshold for measuring the overall health of the entire resource group. A value lower than this usually means that the resource group has performance problems or needs to be expanded. The initial health is a preset starting value used to set a baseline health status for the node at the beginning of resource scheduling.

[0035] The upper limit and lower limit of the health index can be pre-set based on experience. The upper limit of the health index is greater than the lower limit of the health index, and the interval between the two is an elastic range, which can be adjusted according to actual needs.

[0036] There are many specific implementation methods for obtaining the business isolation list. For example, the list can be generated through manual configuration, and the operation and maintenance personnel can pre-define the resource group and corresponding health indicators of each business according to business needs and resource conditions. In addition, the list can also be generated by automated tools, and the message center can automatically allocate resource groups and set health indicators for newly connected businesses based on historical data and preset algorithms. It is also possible to combine manual and automated methods, with the operation and maintenance personnel first defining the basic classification and priority of the business, and then the message center automatically adjusting the size and health indicators of the resource group based on real-time load data.

[0037] In actual applications, the business isolation list can be stored in the form of a database table on the server of the message middle station, or in the form of a configuration file. Regardless of the storage method adopted, the list should be able to be quickly accessed and updated by the resource scheduling system to ensure the real-time and accuracy of resource scheduling. For example, when a new business system needs to be connected to the message middle station, the message middle station can determine the resource group corresponding to the business by querying the business isolation list according to the business type of the new business, and allocate resources according to the health of the message nodes in the resource group. If the overall health of a resource group is lower than the preset lower limit, the message middle station will automatically seize and add message nodes from the resource pool according to the business priority to ensure the performance and stability of the business.

[0038] Step S3200: dynamically adjust the health of each message node according to multiple real-time load indicators of the message nodes in each resource group, and adjust the order of each message node in the node queue corresponding to each resource group according to the adjusted health;

[0039] The real-time load index is the key data to measure the current working status of the message node, and refers to the load index obtained by real-time collection of the message node. This real-time performance can be reflected in the timing collection according to a shorter preset time slot.

[0040] In one embodiment, the real-time load indicators include, but are not limited to, CPU usage, memory occupancy, disk I / O throughput, and network bandwidth utilization. Further embodiments may also selectively include the number of topics and partitions in the message node according to actual needs, wherein the number of partitions specifically includes the number of primary partitions and the number of replica partitions. These indicators can reflect the current load of the node and are an important basis for health assessment.

[0041] Computing health requires quantitative assessment of these real-time load indicators. For example, CPU usage can be expressed as a percentage, and memory occupancy can be measured as the ratio of used memory to total memory. Each indicator can be assigned different weights according to its impact on system performance. The determination of weights can be achieved through historical data analysis, expert experience, or machine learning models. In different embodiments, the setting of weights can be determined as preset values ​​based on expert experience, or can be determined based on a preset algorithm formed by historical data analysis or machine learning models.

[0042] When setting the weights of the load indicators in different resource groups, the differences in resource requirements between different business types can be considered, so that when the message nodes of different resource groups calculate their health, the weight combinations used for their multiple load indicators can reflect the differences in weight combinations corresponding to different resource groups. For example, when the business type of a resource group represents a business that requires high real-time response, CPU usage and network bandwidth utilization can occupy a relatively high weight, and when the business type of a resource group represents a business that requires a high storage space, disk I / O throughput and memory occupancy can be given a relatively high weight.

[0043] In a specific implementation, the calculation of health can be completed by weighted summation. For example, if CPU usage and memory occupancy have a greater impact on health, they can be given a higher weight. The specific calculation formula can be:

[0044] Health = w1*CPU usage + w2*memory usage + w3*disk I / O throughput + w4*network bandwidth utilization

[0045] Among them, w1, w2, w3, and w4 are the weights of the corresponding indicators, and their sum is 1.

[0046] In one embodiment, the dynamic adjustment of health also includes the handling of abnormal situations. For example, if the disk I / O throughput of a message node, a real-time load indicator, suddenly increases, it usually indicates that the node is processing a large amount of data and needs more resources. The message middle station can respond to this change by increasing the health weight of the real-time load indicator.

[0047] A node queue is a queue that is sorted according to the health of the message nodes in a specific resource group. Each resource group maintains a node queue, which contains all the message nodes in the resource group. The order of the node queue is based on the health of the node. The nodes with higher health are ranked at the front of the queue, so that when processing new business topics or message flows, the message middle station will give priority to the nodes at the front of the queue to assign tasks. This mechanism ensures that message nodes with lighter loads can receive new partitions or message flows first, thereby achieving load balancing and avoiding the situation where some nodes are overloaded while other node resources are idle.

[0048] The dynamic adjustment of the node queue is carried out in real time according to the changes in the node health. Specifically, after the health of the message node is calculated according to the real-time load indicators of each message node in the resource group, the health reflects the real-time load status and resource availability of the corresponding message node, so that the message nodes in the node queue can be reordered according to the health of each message node. For example, when a message node has processed a large number of messages, its health will be improved accordingly, and its position in the node queue will also move forward accordingly, indicating that it can receive more messages or tasks.

[0049] Step S3300: In response to a business topic creation request, query the business isolation list according to the business type in the request to determine the corresponding target resource group, determine multiple message nodes with a high health ranking and a health level higher than the corresponding health indicator upper limit as target nodes in the node queue of the target resource group, and create a partition of the topic specified by the request between the target nodes;

[0050] Business topic creation requests are usually initiated by business systems to meet their messaging and data processing needs. In the message middle platform architecture, different business systems, such as e-commerce platforms, online games, video live broadcast platforms, etc., need to process a large number of messages, such as order notifications, real-time interactive information, user behavior logs, etc. In order to process these messages in a timely and efficient manner, these business systems need to create corresponding business topics on the message middle platform. Business topics are a way of classifying messages, which allows the message middle platform to route messages to the correct processing flow based on the type of message or business relevance. Therefore, whenever a new business demand emerges or an existing business needs to expand its message processing capabilities, a business topic creation request is initiated to allocate resources and set the corresponding processing logic on the message middle platform to ensure that business data can be correctly stored, processed and forwarded.

[0051] This application responds to business topic creation requests to ensure that new message flows can be processed effectively. When a business system initiates a new business topic creation request, the message center responds quickly to allocate appropriate resources for the topic.

[0052] First, the message center system will query the business isolation list to determine the corresponding target resource group based on the business type contained in the request, thereby determining which resource group the new business topic will be assigned to, which in turn affects the processing efficiency and stability of the message flow. As a predefined data structure, the business isolation list contains the mapping relationship between business types and resource groups, allowing the message center to quickly locate the correct resource group.

[0053] After determining the target resource group, the message middle-office system will check the node queue of the resource group and select multiple message nodes with high health ranking and higher than the upper limit of the health index as the target nodes. This screening process is based on the health calculated in the previous step, ensuring that the selected target nodes can handle new business topics without causing overload. Nodes with health higher than the upper limit of the health index indicate that they are currently under low load and have sufficient processing power to accommodate new message flows. Compared with only considering the case of high health ranking, this scheduling mechanism can better ensure the robustness of the message flow of the message middle-office in processing business topics.

[0054] Determining the number of target nodes can be achieved based on the backup mechanism and data redundancy strategy preset by the message middle station system. This strategy aims to enhance the reliability and fault tolerance of the message middle station by distributing data among multiple message nodes. Specifically, when creating a partition, each business topic will decide how many message nodes are involved according to the number of replicas required. For example, if each message master partition is required to have two replica partitions to ensure high availability, the number of target nodes should be three. In this way, when a node fails, the replicas on other nodes can take over the work to ensure the continuous processing of the message flow and the stability of the service. In practice, the number of replica partitions can be adjusted according to the business's requirements for data consistency and availability and the resource status of the message middle station. High-priority businesses may require more replica partitions to reduce the risk of data loss, while low-priority businesses may use fewer replica partitions to reduce costs. Through flexible partitioning strategies and replica mechanisms, the message middle station can adapt to the needs of different businesses and achieve optimal resource configuration.

[0055] After selecting multiple target nodes, the message middleware system will create partitions for the new business topic between these nodes, thereby distributing the message flow of the new topic to multiple nodes to achieve load balancing and high availability.

[0056] In one embodiment, if the message nodes in the target resource group are insufficient to meet the needs of the new business topic, or the health of the message nodes is generally low, the message middle platform system will trigger the resource expansion process. By seizing and adding message nodes from the resource pool and initializing the configuration of these message nodes, they can quickly join the resource group and start processing message flows.

[0057] The entire process above is realized by the automation of the intelligent scheduling process of this application. This design enables the message center to flexibly respond to changes in business needs while ensuring efficient processing of message flows and system stability. In this way, the message center can provide reliable and efficient message processing services for different business systems to meet the needs of various business scenarios.

[0058] Step S3400: determine the overall health of the resource group based on the health of all message nodes in each resource group. When the overall health is lower than the corresponding lower limit of the health indicator, configure new message nodes from the resource pool for resource groups with higher business priorities according to the business priorities of the resource groups.

[0059] Determining the overall health of a resource group plays a role in comprehensively evaluating the health of all message nodes within the resource group. The purpose is to monitor the operating status of the entire resource group in real time, ensure that it can meet the needs of the business it serves, and dynamically adjust resources when necessary.

[0060] The calculation of the overall health is based on the real-time health of each message node in the resource group, which has been quantitatively evaluated by real-time load indicators. The overall health is an indicator that reflects the overall performance and stability of the resource group. It integrates the health information of all nodes and provides a macro perspective on the overall status of the resource group. In one embodiment, the overall health of the resource group can be obtained by calculating the arithmetic mean of the health calculated in real time for each message node in the resource group.

[0061] When the overall health of a resource group is lower than the preset health indicator lower limit, it usually means that the message nodes of the resource group are generally facing high load pressure and cannot effectively handle current or expected business traffic. In this case, the message middleware system needs to take measures to increase resources to ensure business continuity and performance.

[0062] According to the business priority of the resource group, the message center system will first seize and add message nodes from the resource pool for the resource group with high business priority. This decision-making process takes into account the different requirements of different businesses for performance and stability, ensuring that key businesses can get sufficient resource support. The newly added message nodes will be initialized and configured, and quickly added to the resource group to share the load of existing nodes, thereby improving the overall health of the resource group.

[0063] For example, if a resource group serves a high-priority risk control system, and the overall health of the resource group is below the lower limit of the health indicator, the message middle office system will automatically identify this situation and quickly allocate new nodes from the resource pool to ensure that the messages of the risk control system can be processed in a timely manner to avoid business risks caused by insufficient resources.

[0064] In one embodiment, the message middle office system will pay more attention to resource groups with higher business priorities. These resource groups usually include businesses with strict requirements on real-time and reliability, such as financial transaction systems, online payment platforms, or mission-critical notification services. The message middle office system will automatically identify these high-priority resource groups and quickly seize available message nodes from the resource pool and assign them to these resource groups.

[0065] In one embodiment, the preemption process may include reallocating message nodes in a resource pool that are being used for low-priority services to a high-priority resource group, or starting a new message node instance to increase the capacity of the resource pool, and using the new message node instance as a newly added message node of the resource group. Once new nodes are allocated or started, they undergo a rapid initialization process, including configuring appropriate network settings, storage resources, and processing capabilities, so that they can quickly join the target resource group and begin processing message flows.

[0066] Through this dynamic resource scheduling mechanism, the message center can flexibly adapt to the changing needs of different businesses, ensure efficient use of resources, and improve the stability and reliability of the message center. This method not only improves the performance of a single resource group, but also optimizes the resource allocation strategy of the entire message center, providing an effective solution to resource competition and conflict in multiple business scenarios.

[0067] According to the embodiment composed of steps S3100 to S3400, it can be seen that the present application cleverly uses the same health index to serve both the capacity expansion decision at the business level and the message node sorting at the business theme level for node scheduling, thereby meeting the needs at different levels. At the business level, the message middle station intelligently judges and prioritizes capacity expansion for high-priority businesses based on the overall health of the message nodes in the resource group, combined with the lower limit of the health index, to ensure the performance and stability of key businesses. At the business theme level, the message middle station sorts the health of each message node in the resource group, and schedules the optimal message node for each business theme, thereby achieving efficient storage and processing of message flows. This method not only improves the flexibility and accuracy of resource scheduling, but also improves the automation and intelligence level of the message middle station, and effectively solves the load balancing problem of the message middle station in the face of multi-business and high-concurrency scenarios. Through this systematic approach, the present application achieves the optimal allocation and utilization of resources while ensuring business isolation, significantly improving the overall performance and stability of the message middle station.

[0068] It can be further known that the application of this application has many technical advantages, including but not limited to:

[0069] First, this application makes resource scheduling more flexible and precise by dynamically adjusting the health level and setting the upper and lower limits of health indicators. This dynamic adjustment mechanism can respond to changes in business load in real time and ensure load balancing, thereby significantly improving resource utilization and system stability. This advantage is particularly evident in scenarios with high concurrency and large business fluctuations.

[0070] Secondly, this application achieves physical isolation between different businesses through the setting of business isolation lists and resource groups. This not only avoids resource competition, but also ensures that key businesses can obtain necessary resource guarantees, thereby improving the performance and stability of key businesses. This isolation mechanism is particularly important for businesses such as risk control that have extremely high requirements for real-time performance and stability.

[0071] Furthermore, the present application can automatically expand capacity according to business priority when the overall health of the resource group is lower than the lower limit of the health index, which enables the message center to quickly respond to changes in business needs, especially during business peaks, to ensure the high availability of the message center and service continuity. This adaptive expansion strategy significantly improves the scalability of the message center. At the same time, the present application also reduces dependence on hardware resources and avoids the practice of simply increasing hardware resources to relieve pressure. Through more intelligent resource scheduling and management, unnecessary cost expenditures are reduced and cost-effectiveness optimization is achieved.

[0072] In addition, this application also improves the automation and intelligence level of the message center. Through the automated monitoring, adjustment and expansion mechanism, manual intervention is reduced and operation and maintenance work is simplified. At the same time, the intelligent scheduling strategy also improves the response speed and processing capacity of the message center, allowing the message center to respond to changes in demand in various business scenarios more quickly and accurately.

[0073] Based on any embodiment of the method of the present application, the health of each message node is dynamically adjusted according to multiple real-time load indicators of the message nodes in each resource group, and the order of each message node in the node queue corresponding to each resource group is adjusted according to the adjusted health, including:

[0074] Step S3210: for each resource group, automatically collect a plurality of pre-specified real-time load indicators of each message node therein according to a preset time slot;

[0075] In order to achieve dynamic monitoring of health, it is necessary to regularly collect real-time load indicators of each message node in each resource group. To this end, multiple key performance indicators of all message nodes in each resource group are monitored in real time and data is collected. These indicators include but are not limited to CPU usage, memory occupancy, disk I / O throughput, and network bandwidth utilization. For example, in another embodiment, it may further include the number of topics and partitions in the message node. The collection of these data can be automatically triggered by the time slot preset by the message center. The time slot can be 1 second, 3 seconds, 5 seconds, 10 seconds, 30 seconds, 1 minute, 1 hour, etc., which can be determined on demand, thereby ensuring the real-time and accuracy of the data.

[0076] In specific implementation, a timed task or event trigger mechanism can be used to realize automatic data collection. For example, the message center can configure a timer to automatically execute data collection tasks at certain time intervals (such as every second, every minute, or every hour). In each preset time slot, the data collection module will traverse all message nodes in each resource group and collect the load index data at the current moment as the corresponding real-time load index. These data will be used for subsequent health calculations to provide real-time reference information for resource scheduling.

[0077] The specific implementation methods of data collection can be diversified. For example, the values ​​of performance counters can be obtained directly from the operating system, or application-level monitoring data can be obtained through API calls. In some cases, existing monitoring tools and platforms (such as Prometheus, Grafana, etc.) can also be used to assist in completing data collection.

[0078] In addition, to ensure the accuracy and representativeness of the data, in one embodiment, the real-time load index collected can be the average value or maximum value within a period of time, generally the time range corresponding to the time slot, rather than the instantaneous value of a single sampling. For example, the average value of the CPU usage rate in the past five minutes can be calculated to reduce the impact of occasional peaks.

[0079] Step S3220: According to a preset health calculation formula, weighted sum of the real-time load indicators is performed to obtain the current health of each message node;

[0080] The health calculation formula is used to integrate multiple real-time load indicators of the message node to obtain the corresponding health value. Taking one implementation method as an example, the real-time load indicators include CPU usage, memory occupancy, disk I / 0 throughput and network bandwidth utilization. Each indicator is assigned different weights according to its impact on system performance. These weights reflect the relative importance of different resources in different business scenarios. They can be preset on demand or automatically determined. For example, different weight combinations can be given according to the business priority and resource configuration of the resource group. For example, for compute-intensive businesses, the weight of CPU usage can be higher; and for data-intensive businesses, the weight of disk I / 0 throughput can be larger.

[0081] An exemplary health calculation formula is as follows:

[0082]

[0083] In this formula, W represents the health of the message node, α, β, γ, and δ are the weights of CPU, memory, disk IO, and network bandwidth, respectively. These weights are allocated according to actual needs to reflect their importance. usage MEM usage 、DISK usage , NET usage is the actual measured resource usage, while CPU max MEM max 、DISK max , NET max is the maximum capacity of these resources, and the corresponding real-time load indicator can be obtained by subtracting the corresponding resource usage from the maximum capacity and then dividing it by the maximum capacity. The maximum capacity is usually determined based on the hardware configuration upper limit of a single message node.

[0084] The formula normalizes the data of different resource types and makes them comparable by dividing the margin corresponding to the actual usage of each resource item by its maximum capacity to obtain a percentage value. Such normalization allows the message middle station to fairly compare the load surplus of different nodes, regardless of their hardware configuration. The load surplus situation can also reflect the health of the message node operation.

[0085] Through this formula, the message center can calculate the current health W of each message node based on the real-time load index. This health reflects the overall load idleness of the corresponding message node at the current moment. The higher the health W, the lighter the load of the message node, so new business topics can be assigned to it; on the contrary, nodes with lower health W need relatively fewer new business topics to be assigned to them to achieve load balancing.

[0086] Of course, in other embodiments, when more load indicators such as the number of topics and the number of partitions are introduced into the health calculation formula, the corresponding weights and indicator calculation methods can naturally be introduced according to the same principles as above to become new resource items in the formula, so that the health is associated with more load indicators and the health of the message node is measured more comprehensively.

[0087] It can be seen that this health-based calculation method enables the message center to more accurately evaluate the load status of each message node, and perform intelligent resource scheduling based on this, thereby optimizing the resource utilization efficiency and business processing capabilities of the entire message center.

[0088] Step S3230: adjust the order of each message node in the node queue of the resource group according to the current health of each message node.

[0089] After the health calculation is completed, all message nodes in the resource group can be sorted according to the calculation results. In this sorting process, message nodes with higher health will be placed in front of the node queue, while message nodes with lower health will be placed at the back of the node queue. This sorting mechanism ensures that when processing new business topics or message flows, the message middle station can give priority to message nodes with lighter loads, thereby achieving load balancing.

[0090] For example, if there are four message nodes A, B, C, and D in a resource group, their healths are H_A, H_B, H_C, and H_D respectively. Based on this, these nodes are sorted in descending order of health. If H_A>H_B>H_C>H_D, node A will be placed at the front of the queue, and node D will be placed at the back of the queue.

[0091] This sorting mechanism not only takes into account the current load status of the node, but also the capacity and business needs of the node. Therefore, it can more accurately reflect the availability and processing power of the node, providing a more reliable basis for resource scheduling.

[0092] In addition, this sorting method is also very flexible and scalable. When the business needs of the message center change, or the load status of the node changes, the message center can quickly recalculate the health and update the sorting of the node queue to quickly adapt to these changes.

[0093] In actual implementation, the order of node queues can be managed by a dedicated order scheduling module. This module will trigger the recalculation of health and reordering of node queues regularly or as needed to ensure that the node queues are always kept up to date.

[0094] The embodiment defined by steps S3210 to S3230 realizes the refined management of message nodes in the resource group by real-time monitoring and dynamic evaluation of the load status of each message node. The unique technical effect of this embodiment is that it can ensure that the resource scheduling system can respond quickly and automatically adjust resource allocation in the face of business traffic fluctuations, thereby maintaining the stability and efficiency of the message center. Specifically, by regularly collecting real-time load indicators and calculating health, the message center can accurately evaluate the load margin of each node, and then sort them according to the health of the nodes, and give priority to allocating new business topics to nodes with lighter loads. This method not only optimizes resource utilization and reduces the risk of overload, but also improves the response speed and processing power of the message center through automated resource scheduling, and significantly improves the load balancing performance of the message center in a multi-business environment. In addition, this embodiment also enhances the scalability and flexibility of the message center, enabling it to adapt to changing business needs, ensure the performance and stability of key businesses, and reduce operation and maintenance costs and complexity.

[0095] On the basis of any embodiment of the method of the present application, according to a preset health calculation formula, before obtaining the current health of each message node by weighted summing of the real-time load indicators, the method includes:

[0096] Step S3211: perform anomaly detection based on the historical data of the load indicators recorded in the system operation log of each message node in each resource group, and determine the health probability corresponding to each load indicator being in a normal state;

[0097] Anomaly detection of load indicators can further ensure the accuracy of health calculation. The purpose of anomaly detection is to identify those load indicator readings that do not meet expectations. These readings may be caused by system failures, external attacks, configuration errors, or other abnormal factors. Through anomaly detection, these erroneous data can be prevented from affecting the health calculation, thereby ensuring the accuracy of resource scheduling decisions.

[0098] Abnormal situations may include, but are not limited to, sudden load peaks, sustained high load conditions, abnormally low resource usage, or significant deviations from historical data. For example, if a message node's CPU usage suddenly soars to 100% and persists for a period of time, this may be caused by a process entering an infinite loop. Another example is that if a message node's network bandwidth utilization suddenly drops to near zero, this may be a problem with the network connection.

[0099] To achieve anomaly detection, machine learning models such as Isolation Forest, Autoencoders, or Long Short-Term Memory Networks (LSTM) can be used. These models can learn the patterns of normal load indicators and identify those anomalies that deviate significantly from the normal pattern. The training of anomaly detection models requires the use of labeled normal and abnormal data samples, which can be obtained through historical operation logs, where abnormal samples can be marked by system administrators based on past system failure records.

[0100] When implementing anomaly detection, the model analyzes the historical data of load indicators recorded in the system operation log of each message node in each resource group and outputs the health probability of each indicator. The health probability is a quantitative indicator that indicates the confidence that the current load indicator reading is within the normal range. For example, a CPU usage reading with a health probability of 0.9 indicates that there is a 90% confidence that the reading is normal.

[0101] In one embodiment, the anomaly detection model combines a long short-term memory network (LSTM) and a classifier to identify anomalies of the load index. The anomaly detection model is trained in advance with training samples so that it has the ability to predict the health probability corresponding to the load index.

[0102] The data content of the training samples may consist of historical data of a single load indicator recorded in the system operation log of each message node in each resource group, including but not limited to CPU usage, memory occupancy, disk I / O throughput, and network bandwidth utilization, etc. These historical data are used as input to the model to capture typical behavior patterns during normal operation.

[0103] The supervision label is used to indicate whether each input training sample of the anomaly detection model is abnormal, which can be determined by prior labeling. When labeling these labels, it is necessary to determine whether each load indicator reading is normal. This is usually done by the system administrator based on past system operation records and fault reports. For example, if a CPU usage reading rarely reaches above 95% historically, the reading above this threshold can be marked as abnormal.

[0104] The LSTM network is responsible for extracting features from time series data, and it is able to capture the long-term dependencies of load indicators over time. This is because the LSTM network is designed with a gating mechanism, which enables it to effectively remember long-term information when processing time series data and ignore unimportant short-term fluctuations.

[0105] The classifier is responsible for predicting the health probability of each load indicator based on the features extracted by the LSTM network. This classifier can be a simple logistic regression model or a more complex neural network such as a multi-layer perceptron (MLP) or a support vector machine (SVM). The goal of the classifier is to learn how to classify load indicator readings as normal or abnormal based on the input features.

[0106] During training, the LSTM network and the classifier together form an end-to-end model that is optimized by minimizing the difference between the predicted health probability and the true label. This optimization can be achieved using various machine learning algorithms, such as gradient descent or its variants.

[0107] In this way, the anomaly detection model can learn the patterns of normal load indicators and identify those anomalies that deviate significantly from the normal pattern. After the model is trained, it can automatically analyze new load indicator data and output the health probability of each indicator. These health probabilities are then used to calculate the health of the message node and provide an accurate basis for resource scheduling.

[0108] The output of the anomaly detection model, that is, the health probability of each load indicator, will be used in the subsequent health calculation. This health probability reflects the reliability of each indicator, so when calculating health, those indicators with higher health probabilities will have a greater influence. In this way, the message middle station can more accurately evaluate the current health status of each message node and make more reasonable resource scheduling decisions based on this.

[0109] Step S3212: adjusting the preset weight of each load indicator according to its health probability to obtain an adjusted weight of each load indicator;

[0110] After completing anomaly detection and calculating the health probability of each load indicator, the health probability can be used to adjust the preset weight of the corresponding real-time load indicator. This is because different load indicators may have different effects on the health of the message node, and the importance of each indicator may change over time. For example, if a load indicator shows a high probability of anomaly and a low probability of health, then its weight in the health calculation should be reduced accordingly to reduce its impact on the health assessment.

[0111] To this end, in this step, the preset weight of each load indicator is adjusted according to its health probability. The adjusted weight reflects the contribution of each indicator to the health at the current moment. This adjustment process can be achieved through a simple mathematical formula. For example, the preset weight of each indicator can be multiplied by its health probability to obtain the adjusted weight. In one embodiment, if the health probability is lower than a certain threshold, the weight of the indicator can be set to zero or a very small value, indicating that it should not affect the calculation of health.

[0112] Step S3213: normalize the adjusted weight of each load indicator to obtain the actual weight of each load indicator;

[0113] Normalizing the adjusted weights ensures that the actual weights of all load metrics sum to a fixed value, usually 1. This step ensures that the weights of the individual metrics are comparable and their combined effect is balanced when calculating health. Normalization can be achieved by dividing each adjusted weight by the sum of all weights. This way, the actual weight of each load metric reflects its relative importance in the overall health calculation.

[0114] Step S3214: Substitute the actual weight of each load indicator into the preset health calculation formula to calculate the current health by weighted summing of each real-time load indicator.

[0115] After obtaining the actual weights corresponding to each load indicator, substitute these actual weights into the preset health calculation formula. This formula can combine the actual weight of each load indicator and the real-time load indicator to calculate the current health of each message node by weighted summation. For example, if the CPU usage and memory occupancy of a message node are the two main real-time load indicators currently considered, then their actual weights will be used together with their real-time values ​​to calculate the health of the node. This calculation result will directly affect the ranking of the node in the node queue within the resource group and the possibility of it being assigned a new business topic partition.

[0116] The embodiment defined by steps S3211 to S3213 brings unique technical effects to the message middle station resource scheduling method by introducing anomaly detection and weight adjustment mechanisms.

[0117] First, by performing anomaly detection on each load indicator, the impact of abnormal data on health assessment can be identified and reduced, thereby improving the accuracy and reliability of health calculation. This is often overlooked by existing technologies, but it is particularly important in scenarios where the system operating environment is complex or vulnerable to attacks, because it reduces false positives and false negatives and ensures system stability.

[0118] Secondly, by dynamically adjusting the weight of each load indicator according to the health probability, the real impact of each indicator on the health of the message node can be more accurately reflected. This dynamic weight adjustment mechanism enables the message middle station to adaptively respond to changes in resource usage patterns, whether these changes are due to normal business fluctuations or due to performance issues of the message middle station. This approach improves the flexibility and intelligence of resource scheduling, making resource allocation more in line with the current system status.

[0119] Then, the normalization process ensures that all adjusted weights are re-uniformed in scale, making it possible to compare and weighted sum different indicators. This not only improves the scientificity and rationality of health calculation, but also provides a solid data foundation for subsequent resource scheduling decisions.

[0120] It can be seen that this embodiment significantly improves the accuracy and efficiency of the message center in resource scheduling based on the organic coordination of the above process. This not only optimizes resource utilization, but also enhances the robustness of the message center to abnormal situations, thereby providing a more effective solution to resource competition and conflict in multiple business scenarios.

[0121] On the basis of any embodiment of the method of the present application, according to multiple real-time load indicators of the message nodes in each resource group, the health of each message node is dynamically adjusted, and after the sorting of each message node in the node queue corresponding to each resource group is adjusted according to the adjusted health, it includes:

[0122] Step S4100: Calculate the average deviation of the current period relative to the historical period based on the health change data of the message nodes in the current resource group between the current period and the historical period;

[0123] The health of each message node represents the load level of the message node under normal operation. In order to measure historical changes, it can be examined in units of preset cycles, which can be measured in days or weeks. The current cycle refers to the ongoing cycle, while the historical cycle refers to the same time period in the past. For example, if today is the current cycle and the current time is 2 pm, the load indicator data at 2 pm today can be compared with the same time points in the past few days to calculate the deviation between the health of each message node in the current cycle and its historical cycle data.

[0124] The purpose of calculating the average deviation is to quantify the trend of message node health over time. Specifically, statistical analysis is performed on multiple real-time load indicators of each message node in the current cycle, and compared with historical data of the same period. The values ​​representing the differences are determined by comparison, and then the average of these differences is calculated to obtain the average deviation of each message node. This average deviation is a quantitative indicator that shows the degree of change in the health of the message node relative to historical data.

[0125] For example, if a message node's CPU usage averaged 50% at the same time in the past week, and the average usage in the current cycle is 60%, then the average deviation of the node's CPU usage is 10%. If this deviation exceeds the threshold preset by the message center, such as 15%, then it can be considered that the node's CPU resource usage is increasing, indicating that more resources may be needed to maintain its performance.

[0126] Step S4200, fitting a health curve of the current cycle based on the health change data generated in the historical cycle and the current cycle, and correcting the health change data not generated in the health curve according to the average deviation;

[0127] After the calculation of the average deviation of the health change of the message node is completed, the average deviation can be used to predict the health change trend in the current cycle. To this end, a health curve needs to be constructed, which is based on the collected real-time load indicator data and reflects the change of the health of the message node over time.

[0128] The construction of health curves usually uses mathematical or statistical methods, such as polynomial regression, moving average, or time series analysis, to fit the health change data generated in historical cycles and the current cycle. This curve is a quantitative representation of the trend of node health changes, which can help predict the health of the node in the future time period. For example, if the curve shows that the health is decreasing over time, this usually indicates that the resource usage of the node is increasing, and more resources are needed to maintain its performance.

[0129] After the health curve is constructed, it needs to be corrected to improve the accuracy of the prediction. The correction process takes into account the average deviation calculated previously. If the average deviation shows that the health of the message nodes is generally lower than the historical level, the curve will be adjusted accordingly to reflect this trend. This correction can be achieved by adjusting the slope or shape of the curve to ensure that the curve is closer to the actual observed data points.

[0130] For example, if the CPU usage of a message node averaged 50% at the same time in the past week, and the average usage in the current period is 60%, then the average deviation of the CPU usage of this node is 10%. When fitting the health curve, this deviation will be used to adjust the curve to make it more accurately predict the changes in CPU usage in future time periods.

[0131] The corrected health curve will be used to predict changes in node health in future time periods. These prediction data will be used to assess whether the node may face the risk of insufficient resources in the future, thereby providing a basis for resource scheduling decisions. For example, if the forecast shows that the health of a node will continue to decline in the future time period, the message middle station can take measures in advance, such as increasing resource allocation or load balancing, to ensure that the node can continue to effectively process message flows.

[0132] Step S4300: Calculate the average of the corrected health change data corresponding to the future time period. When the average is lower than the lower limit of the health indicator of the current resource group, mark the current message node as being in the expected busy state; otherwise, clear the expected busy state.

[0133] Based on the modified health curve, the node health change data in a certain time period in the future can be predicted and the average of these predicted data can be calculated. This time period can be set as needed, such as the next 1 hour, several hours or longer.

[0134] Calculating the predicted mean can be used to assess whether each message node may face the risk of insufficient resources in the future. If the predicted health mean is lower than the preset health indicator lower limit of the resource group, the message center will mark the message node as expected to be busy in the future time period. This mark will serve as a reference for resource scheduling, prompting the message center to reserve more resources or adjust the load according to the load of the message node to avoid performance degradation or service interruption of the entire resource group. Here, the lower limit of the health indicator of the resource group is once again used to measure the predicted health mean, realizing the multi-functional reuse of the same indicator. Its design is ingenious and very intuitive.

[0135] For example, if the lower limit of the health indicator of a resource group is 70%, and the message center predicts that the average health of a message node in the next hour will be 65%, this indicates that the node will not be able to meet resource requirements in the future, and the message center will mark the node as expected busy. In order to maintain this status information in a timely manner, when the predicted average is not lower than the lower limit of the health indicator of the current resource group, the expected busy state of the current message node can be cleared in time.

[0136] Step S4400, count the proportion of the number of message nodes in the current resource group that are expected to be busy to the total number of message nodes in the current resource group. When the proportion is higher than the preset threshold, mark the current resource group as being in a resource emergency state, lock the message nodes in the resource pool to achieve preemption, otherwise clear the resource emergency state of the current resource group, and release the current resource group's lock on the message nodes in the resource pool.

[0137] For each resource group, you can also count the number of message nodes in the resource group that are marked as expected busy, and calculate the percentage of this number in the total number of message nodes in the entire resource group. Based on this percentage, you can determine whether the entire resource group is facing resource shortage. If the number of expected busy nodes is high and exceeds the threshold preset by the message center, for example, more than 20%, the message center will mark the entire resource group as being in a resource emergency state. This state indicates that most message nodes in the resource group will need more resources in the future to maintain normal operation.

[0138] Once a resource group is marked as in a resource-critical state, the message center will take measures to lock the message nodes in the resource pool to achieve resource preemption, so that idle nodes in the resource pool can be allocated to the resource group in resource crisis, or the resource allocation strategy between resource groups can be adjusted to ensure the performance and stability of key businesses.

[0139] If the number of nodes in resource emergency state is lower than the preset threshold, the message center will consider the resource group's resource usage to be normal and will not take additional locking measures. Resources can continue to be allocated on demand between different resource groups. If there is still a resource emergency state mark in the resource group in this case, it can be cleared accordingly.

[0140] It can be seen that through the embodiments of step S4100 to step S4400, the message middle station of the present application can identify and respond to changes in resource requirements in advance, thereby achieving more refined and forward-looking resource scheduling. This not only improves resource utilization, but also enhances the message middle station's ability to predict potential problems, ensuring the continuity and stability of key businesses.

[0141] Based on the previous embodiment of the method of the present application, the overall health of the resource group is determined according to the health of all message nodes in each resource group. When the overall health is lower than the corresponding lower limit of the health index, according to the service priority of the resource group, a new message node is configured from the resource pool for the resource group with a higher service priority, including:

[0142] Step S5100: Detect whether the current resource group is in a resource emergency state. If it is in a resource emergency state, determine the message node locked by the current resource group in the resource pool, and configure the message node as a new message node of the current resource group;

[0143] According to the strategy of this application, a new message node is added to each resource group. The overall health of the resource group is determined based on the health of all message nodes in each resource group. When the overall health is lower than the corresponding lower limit of the health indicator, that is, when the overall health of the resource group is lower than the corresponding lower limit of the health indicator, it is necessary to configure a new message node for this resource group to achieve effective capacity expansion.

[0144] Multiple resource groups may need to configure new message nodes at the same time. In this case, the message nodes can be preempted and added to the resource groups according to the service priorities of the resource groups. The previous embodiment actually achieves the preemption of the message nodes by locking the message nodes of the resource pool in advance. Therefore, in this case, further detection of each resource group, such as a current resource group, to see if it is in a resource emergency state can directly call the message nodes that have been preempted in advance to configure them in the current resource group.

[0145] Specifically, when the current resource group is detected to be marked as in a resource emergency state, the message center first determines the message nodes in the resource pool that have been locked by the current resource group. The locked message nodes indicate that they have been reserved for the current resource group for rapid resource expansion when resources are in an emergency.

[0146] Next, the message center configures these locked message nodes as new message nodes of the current resource group. This configuration process includes updating system records, marking the message nodes as part of the current resource group, and can adjust network settings, allocate storage resources, or configure specific message processing logic to ensure that the new nodes can immediately start processing message flows.

[0147] Step S5200: When the resource is not in an emergency state, determine an idle message node that is not locked by other resource groups from the resource pool, first lock the message node, and then configure it as a new message node of the current resource group;

[0148] If the current resource group is not in a resource emergency state, the message center will select idle message nodes from the resource pool that are not locked by other resource groups. These idle message nodes are message nodes in the resource pool that are not currently preempted and can be quickly put into service. To avoid possible preemption conflicts, the message center will first lock these message nodes to prevent them from being occupied by other resource groups, and then configure them to become part of the current resource group.

[0149] During the configuration process, the message center can perform a series of configuration tasks, including setting network parameters, allocating storage resources, configuring message processing logic, etc. These tasks ensure that the new message node can work with other nodes in the resource group to jointly process the message flow.

[0150] Step S5300: When configuration of a new message node fails, an alarm notification message is sent to a preset communication interface.

[0151] If any problems are encountered during the configuration of a new message node, such as node unavailability, configuration errors, or network problems, the message center will not be able to successfully add the node to the resource group. In this case, the message center will send an alarm notification message to the preset communication interface. This alarm notification can trigger an automated error handling process or notify the operation and maintenance personnel to intervene manually to quickly resolve the problem and ensure that the resource group can obtain the required resources in a timely manner.

[0152] The unique technical effect of this embodiment compared to other embodiments is that it not only allows the message middle station to dynamically allocate new message nodes to resource groups according to business priorities, but also introduces resource emergency status for resource groups, so that even resource groups with lower business priorities can seize the required message nodes when facing actual resource pressure. This mechanism ensures the timeliness and fairness of resource allocation by monitoring the overall health of the resource group and triggering resource preemption when it is lower than the preset lower limit of the health indicator. In this way, the resource group can not only obtain resources according to the relative importance of the business, but also dynamically adjust according to the actual resource usage and demand, thereby optimizing resource allocation, improving resource utilization, and ensuring the stability of the message middle station and the continuity of the business. In addition, this resource scheduling strategy based on comprehensive consideration of actual load and business priority makes resource allocation more flexible and accurate, and provides a more scientific and reasonable resource management method for the message middle station.

[0153] Based on any embodiment of the method of the present application, the overall health of the resource group is determined according to the health of all message nodes in each resource group. When the overall health is lower than the corresponding health index lower limit, according to the business priority of the resource group, a new message node is configured from the resource pool for the resource group with a higher business priority, including:

[0154] Step S6100: Identify multiple message nodes that are not locked in the resource pool as potential new nodes;

[0155] During the resource scheduling process of the message center, once it is detected that the overall health of the resource group is lower than the preset health indicator lower limit, the server can start the resource expansion process.

[0156] Considering that the message nodes in the resource pool may be locked for some reason, the server first identifies multiple message nodes that are not locked in the resource pool. These nodes are potential new nodes that can be used to expand the processing capacity of the resource group. The resource pool is a collection of all available message nodes in the message center, and the unlocked message nodes mean that they are not currently used by any resource group, so they can be candidates for new resources.

[0157] The process of identifying these potential new nodes can be achieved by querying the status information of the resource pool, specifically by maintaining a node status list of the resource pool. This list records the current status of each message node, including whether it is locked, current load, processing capacity, and health status. By checking this list, find all message nodes marked as unlocked and list them as potential new nodes.

[0158] In a specific implementation, a background service can be run to periodically scan the status of the resource pool to update the list of unlocked message nodes. In addition, an event trigger mechanism can be configured so that when the status of a message node changes (for example, from locked to unlocked), the server will be notified and update the list of potential new nodes.

[0159] Step S6200: Perform a stress test on each potential new node, and select the potential new node with the best stress test result as the target new node of the resource group with the highest service priority that needs to be expanded;

[0160] To ensure availability, you can perform stress testing on the newly added nodes. The purpose of stress testing is to evaluate the actual processing capacity and stability of each node, so as to select the node that best meets the current resource group requirements for expansion.

[0161] Stress testing is to observe the performance of nodes under extreme conditions by simulating high-load conditions. It simulates the performance of message nodes under high-traffic conditions by sending a large number of messages or requests to the nodes. In this process, the response time, processing speed, error rate and other key performance indicators of the corresponding nodes can be monitored as test results. Through these indicators, it is determined whether the node can withstand the expected load and whether it can provide services stably.

[0162] For example, a gradually increasing load is applied to each potential new node until the node's performance begins to decline or errors occur. During this process, the maximum load that each node can handle and the performance data under high load are recorded as test results. These data will be used to evaluate the performance of the node and serve as the basis for selecting nodes for expansion.

[0163] After the stress test is completed, the node with the best performance is selected according to the test results. For example, in one embodiment, the specific key performance indicators in the test results of the message node can be weighted and fused to obtain comprehensive indicator data, and compared with a preset threshold. When the comprehensive indicator data is the highest among all message nodes in the resource pool and is higher than the threshold, the node is selected as the best node.

[0164] These optimal nodes are not only able to handle high loads, but also able to maintain stable performance under pressure. The selected nodes will be configured as new nodes in the current resource group to provide additional processing power.

[0165] Step S6300: After the target newly added node completes the expansion of the resource group to which it belongs, the load of each message node in the resource group is redistributed.

[0166] The selected target new node will be formally configured as part of the resource group. This includes updating system records to mark the new node as a member of the resource group, thereby ensuring that the target new node can work with other nodes in the resource group to process message flows.

[0167] Once the target new node is successfully added to the resource group, it triggers the redistribution of the load within the resource group. Specifically, new business topics or message flows can be assigned to the target new node, or existing message flows can be redistributed to balance the load of all nodes in the resource group. The purpose of load redistribution is to ensure that each node in the resource group operates in an optimal manner to avoid overloading some nodes while other node resources are idle.

[0168] After the load redistribution is completed, the performance of the target newly added node can be continuously monitored to ensure that it can stably process messages. If the target newly added node exhibits problems in actual operation, it can be adjusted, such as replacing the target newly added node.

[0169] The unique technical effect of this embodiment compared to other embodiments is: First, by identifying unlocked message nodes as potential new nodes, the message middle station can quickly respond to the capacity expansion needs of the resource group. Next, by stress testing these potential nodes, the message middle station can evaluate and select the nodes that are most capable of high-load conditions, thereby ensuring that the new nodes can immediately provide the required processing power after joining the resource group. Finally, by redistributing the load within the resource group, the message middle station not only balances the workload between nodes, but also optimizes the processing efficiency of the entire resource group. This coherent process not only improves resource utilization, but also enhances the ability of the message middle station to adapt to business peaks, ensuring the continuity and stability of key businesses, but also improves the flexibility and automation level of resource scheduling.

[0170] Based on any embodiment of the method of the present application, the method further includes:

[0171] Step S7100: receiving configuration information of a new service, and determining a service type and a service priority of the new service from the configuration information;

[0172] When a new service needs to be connected, a corresponding service configuration request will be triggered. To this end, the message middle station first receives and processes the configuration information of the corresponding service in response to the request. In one embodiment, the configuration information is a set of data that describes the characteristics and requirements of the new service in detail, including service type and service priority. The service type is a classification of the nature of the service, for example, it can be e-commerce, online payment, real-time communication, etc., while the service priority reflects the urgency and importance of the service's demand for resources.

[0173] The process of receiving new business configuration information can be achieved through a predefined interface that can receive data from the business management system or operation team. This data can be input through automated configuration management tools or directly entered into the system through manual operations. The message center will parse this configuration information and extract key data such as business type and business priority.

[0174] For example, if the new service is a high-priority financial service system, its configuration information may include service type "financial service" and service priority "high". Based on this information, the message center will determine the resource requirements of the new service and prepare to connect it to the appropriate resource group.

[0175] In a specific implementation, the configuration page of the message center can provide a user interface to allow business administrators to enter the configuration information of new businesses, or automatically receive this information through API calls. After verifying the integrity and accuracy of this information, the message center can store it accordingly for use in subsequent steps.

[0176] In one embodiment, a verification step may also be included to ensure that the received configuration information complies with the predetermined format and business rules. If the configuration information does not meet the requirements, the message center will provide feedback and require the correct information to be provided.

[0177] Step S7200: query the resource group corresponding to the new service from the service isolation list according to the determined service priority;

[0178] The business isolation list contains the correspondence between different business types and resource groups, as well as the upper and lower limits of the health indicators of each resource group and the initial health. By searching the business isolation list, the message center can quickly locate the resource group that matches the priority of the new business. For example, if the new business is marked as high priority, the message center will search the business isolation list for a resource group with sufficient processing power and resource capacity to ensure that the new business can obtain the required resources.

[0179] The query process can be implemented in a variety of ways. One possible implementation is to use a database query, where the business priority is used as a query condition to retrieve the corresponding resource group. Another implementation is to use a hash table or dictionary structure and directly access the corresponding resource group information using the business priority as a key value.

[0180] After querying the corresponding resource group, the message center will verify whether the resource group has sufficient health and resource capacity to accommodate the new business. If the resource group meets the requirements, the message center will proceed to the next step; if not, it may be necessary to trigger the resource expansion process or select other appropriate resource groups.

[0181] Step S7300: Randomly select a message node whose health is higher than the upper limit of the health index in the node queue of the queried resource group to create a partition of the topic of the new business;

[0182] In the node queue of the queried resource group, each message node of the resource group has an associated health, which reflects the real-time operation status and load of the node. Nodes with a health higher than the upper limit of the health index indicate that they are in good condition and have not reached the upper limit of their processing capacity, so these nodes are ideal for creating new business topic partitions.

[0183] The message center will randomly select multiple message nodes from the resource group whose health is higher than the upper limit of the health index. Obviously, this is not limited to the case of selecting the message node with the highest health. This random selection method can ensure that the load is evenly distributed among all nodes in the resource group, avoiding excessive concentration on one or some nodes, thereby improving the processing capacity and stability of the entire resource group.

[0184] After selecting the nodes, the message center will create topic partitions for the new business on these nodes, including primary partitions and replica partitions. This process includes configuring the node's storage resources, setting message routing rules, and updating the system's status information. After the partitions are created, the new business can start sending and receiving messages to these partitions, thereby starting its normal business process.

[0185] Step S7400: associate the service type and service priority of the new service with its corresponding resource group and add them to the service isolation list.

[0186] When the topic partition of the new business is created on the selected resource group node, the message center needs to record this configuration for resource scheduling and management. This includes adding the business type, business priority and selected resource group information of the new business to the business isolation list.

[0187] In the specific implementation, the message center will update the business isolation list and insert the identifier, business type, business priority and associated resource group information of the new business as a new record. For example, if the new business is a high-priority financial transaction system and has been assigned to a resource group with sufficient processing capacity, the message center will record this information, including the business name, business type, business priority and resource group identifier. In this way, in the future resource scheduling and monitoring process, the resource configuration related to the business can be quickly identified and accessed.

[0188] In this embodiment, partitions for new business topics are created by randomly selecting message nodes whose health is higher than the upper limit of the health index. The unique technical effect of this embodiment compared to other embodiments is that it enhances the load balancing capability of the message center and the fairness of resource allocation. By randomly allocating to message nodes whose health is higher than the upper limit of the health index, instead of relying on the message nodes with the highest health, the message center can ensure that the nodes in the resource group are evenly utilized, avoiding performance bottlenecks caused by excessive loads on some nodes, and preventing other node resources from being idle. This method not only improves resource utilization, but also reduces the risk of service interruption caused by node overload, thereby improving the stability and reliability of the entire message center. In addition, the random selection mechanism simplifies the resource allocation process, reduces the need for manual intervention, and enables the system to respond to the access needs of new businesses more automatically and intelligently.

[0189] See also Figure 2According to one aspect of the present application, a message middle station resource scheduling device is provided, including a list acquisition module 3100, a sorting and scheduling module 3200, a topic creation module 3300, and a resource expansion module 3400, wherein the list acquisition module 3100 is configured to obtain a business isolation list, the business isolation list contains business types, business priorities and resource groups corresponding to multiple businesses, each resource group includes multiple message nodes for distributed storage of message streams of each topic of its corresponding business, and each resource group is set with a matching health indicator upper limit, health indicator lower limit and initialized health degree corresponding to its business priority; the sorting and scheduling module 3200 is configured to dynamically adjust the health degree of each message node according to multiple real-time load indicators of the message node in each resource group, The order of each message node in the node queue corresponding to each resource group is adjusted according to the adjusted health; the topic creation module 3300 is configured to respond to a business topic creation request, query the business isolation list according to the business type in the request to determine the corresponding target resource group, determine multiple message nodes with high health ranking and health higher than the corresponding health indicator upper limit in the node queue of the target resource group as target nodes, and create partitions of the topic specified by the request between the target nodes; the resource expansion module 3400 is configured to determine the overall health of the resource group according to the health of all message nodes in each resource group, and when the overall health is lower than the corresponding health indicator lower limit, according to the business priority of the resource group, prioritize the resource group with higher business priority to configure new message nodes from the resource pool.

[0190] Based on any embodiment of the device of the present application, the sorting and scheduling module 3200 includes: a real-time collection module, which is configured to automatically collect multiple pre-specified real-time load indicators of each message node in each resource group according to a preset time slot; a formula application module, which is configured to obtain the current health of each message node by weighted summation of each real-time load indicator according to a preset health calculation formula; and a queue adjustment module, which is configured to adjust the sorting of each message node in the node queue of the resource group according to the current health of each message node.

[0191] On the basis of any embodiment of the device of the present application, prior to the formula application module, it includes: an anomaly detection module, which is configured to perform anomaly detection based on the historical data of the load indicators recorded in the system operation log of each message node in each resource group, and determine the health probability corresponding to each load indicator being in a normal state; a weight optimization module, which is configured to adjust the preset weight of each load indicator according to its health probability, and obtain the adjusted weight of each load indicator; a weight normalization module, which is configured to normalize the adjusted weight of each load indicator, and obtain the actual weight of each load indicator; a formula correction module, which is configured to substitute the actual weight of each load indicator into a preset health calculation formula, so as to calculate the current health by weighted summing of each of the real-time load indicators.

[0192] On the basis of any embodiment of the device of the present application, after the sorting and scheduling module 3200, it includes: a deviation solving module, which is configured to solve the average deviation of the current cycle relative to the historical cycle according to the health change data of the message nodes in the current resource group between the current cycle and the historical cycle; a prediction and correction module, which is configured to fit the health curve of the current cycle based on the health change data generated in the historical cycle and the current cycle, and correct the health change data that has not been generated in the health curve according to the average deviation; a node marking module, which is configured to calculate the average of the corrected health change data corresponding to the future time period, and when the average is lower than the lower limit of the health indicator of the current resource group, mark the current message node as being in an expected busy state, otherwise clear the expected busy state; a whole group marking module, which is configured to count the proportion of the number of message nodes in the current resource group that are expected to be busy to the total number of message nodes in the current resource group, and when the proportion is higher than a preset threshold, mark the current resource group as being in a resource emergency state, lock the message nodes in the resource pool to achieve preemption, otherwise clear the resource emergency state of the current resource group, and release the current resource group from locking the message nodes in the resource pool.

[0193] Based on any embodiment of the device of the present application, the resource expansion module 3400 includes: an emergency response module, which is configured to detect whether the current resource group is in a resource emergency state. When in a resource emergency state, it determines the message node locked by the current resource group in the resource pool, and configures the message node as a new message node for the current resource group; a non-emergency response module, which is configured to determine an idle message node that is not locked by other resource groups from the resource pool when the resource is not in the emergency state, first lock the message node, and then configure it as a new message node for the current resource group; a failure processing module, which is configured to send an alarm notification message to a preset communication interface when the configuration of the new message node fails.

[0194] The resource expansion module 3400, based on any embodiment of the device of the present application, includes: an idle anchoring module, configured to identify multiple unlocked message nodes in the resource pool as potential new nodes; a stress testing and optimization module, configured to perform stress testing on each potential new node, and obtain the potential new node with the best stress test result as the target new node of the resource group with the highest business priority that needs to be expanded; an expansion and update module, configured to redistribute the load of each message node in the resource group after the target new node completes the expansion of the resource group to which it belongs.

[0195] Based on any embodiment of the device of the present application, the device also includes: a configuration determination module, configured to receive configuration information of a new business, and determine the business type and business priority of the new business from the configuration information; a list query module, configured to query the resource group corresponding to the new business from the business isolation list according to the determined business priority; a random allocation module, configured to randomly select a message node with a health higher than the upper limit of the health index in the node queue of the queried resource group to create a partition of the topic of the new business; a list update module, configured to associate the business type and business priority of the new business with its corresponding resource group and add them to the business isolation list.

[0196] Another embodiment of the present application also provides a message middle station resource scheduling device. Figure 3 As shown, a schematic diagram of the internal structure of a message middle station resource scheduling device. The message middle station resource scheduling device includes a processor, a computer-readable storage medium, a memory, and a network interface connected via a system bus. Among them, the computer-readable non-volatile readable storage medium of the message middle station resource scheduling device stores an operating system, a database, and computer-readable instructions. The database may store an information sequence. When the computer-readable instructions are executed by the processor, the processor can implement a message middle station resource scheduling method.

[0197] The processor of the message middle station resource scheduling device is used to provide computing and control capabilities to support the operation of the entire message middle station resource scheduling device. The memory of the message middle station resource scheduling device may store computer-readable instructions, which, when executed by the processor, enable the processor to execute the message middle station resource scheduling method of the present application. The network interface of the message middle station resource scheduling device is used to connect and communicate with the terminal.

[0198] Those skilled in the art will understand that Figure 3 The structure shown in the figure is only a block diagram of a partial structure related to the scheme of the present application, and does not constitute a limitation on the message middle-station resource scheduling device to which the scheme of the present application is applied. The specific message middle-station resource scheduling device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0199] In this embodiment, the processor is used to execute Figure 2 The memory stores the program code and various data required to execute the above modules or submodules. The network interface is used to realize data transmission between user terminals or servers. The non-volatile readable storage medium in this embodiment stores the program code and data required to execute all modules in the message middle station resource scheduling device of this application, and the server can call the program code and data of the server to execute the functions of all modules.

[0200] The present application also provides a non-volatile readable storage medium storing computer-readable instructions. When the computer-readable instructions are executed by one or more processors, the one or more processors execute the steps of the message middle station resource scheduling method of any embodiment of the present application.

[0201] The present application also provides a computer program product, including a computer program / instruction, which implements the steps of the method described in any embodiment of the present application when executed by one or more processors.

[0202] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments of the present application can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the aforementioned storage medium can be a computer-readable storage medium such as a disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0203] In summary, this application provides an efficient, flexible and scalable solution through refined resource management and intelligent scheduling to meet the message processing needs of different business systems. Its application advantages are mainly reflected in the following aspects: First, through business isolation and dynamic resource allocation, the performance and stability of high-priority businesses are ensured, while optimizing resource utilization; secondly, the real-time monitoring and health assessment mechanism enables the system to respond quickly to load changes, achieve load balancing, and effectively avoid the occurrence of overload; thirdly, the automated resource expansion and reduction strategy simplifies operation and maintenance work, reduces costs, and improves the adaptability of the message middle station; finally, this method supports rapid response to business growth and changes, ensures the high availability of the message middle station and service continuity, and provides users with a more reliable and efficient service experience.

Claims

1. A message middle station resource scheduling method, characterized in that: include: Obtain a business isolation list, the business isolation list including business types, business priorities and resource groups corresponding to multiple businesses, each resource group including multiple message nodes for distributed storage of message flows of each topic of its corresponding business, and each resource group having a matching health indicator upper limit, health indicator lower limit and initialized health degree corresponding to its business priority; According to multiple real-time load indicators of the message nodes in each resource group, the health of each message node is dynamically adjusted, and the order of each message node in the node queue corresponding to each resource group is adjusted according to the adjusted health; In response to a business topic creation request, query the business isolation list according to the business type in the request to determine the corresponding target resource group, determine multiple message nodes with a high health ranking and a health level higher than the corresponding health indicator upper limit as target nodes in the node queue of the target resource group, and create a partition of the topic specified by the request between the target nodes; The overall health of the resource group is determined based on the health of all message nodes in each resource group. When the overall health is lower than the corresponding lower limit of the health indicator, new message nodes are configured from the resource pool for resource groups with higher business priorities according to the business priorities of the resource groups.

2. The message middle station resource scheduling method according to claim 1 is characterized in that: According to multiple real-time load indicators of the message nodes in each resource group, the health of each message node is dynamically adjusted, and the order of each message node in the node queue corresponding to each resource group is adjusted according to the adjusted health, including: For each resource group, multiple pre-specified real-time load indicators of each message node therein are automatically collected according to preset time slots; According to the preset health calculation formula, the weighted sum of each of the real-time load indicators is obtained to obtain the current health of each message node; The order of each message node in the node queue of the resource group is adjusted according to the current health of each message node.

3. The message middle station resource scheduling method according to claim 2 is characterized in that: According to the preset health calculation formula, the weighted sum of each of the real-time load indicators is used to obtain the current health of each message node, including: Perform anomaly detection based on the historical data of load indicators recorded in the system operation log of each message node in each resource group, and determine the health probability corresponding to each load indicator being in a normal state; The preset weight of each load indicator is adjusted according to its health probability to obtain the adjusted weight of each load indicator; Normalize the adjusted weight of each load indicator to obtain the actual weight of each load indicator; The actual weight of each load indicator is substituted into a preset health calculation formula to calculate the current health by weighted summing of each of the real-time load indicators.

4. The message middle station resource scheduling method according to claim 1 is characterized in that: According to multiple real-time load indicators of the message nodes in each resource group, the health of each message node is dynamically adjusted, and the order of each message node in the node queue corresponding to each resource group is adjusted according to the adjusted health, including: According to the health change data of the message nodes in the current resource group between the current cycle and the historical cycle, the average deviation of the current cycle relative to the historical cycle is solved; Fitting a health curve of the current cycle based on the health change data generated in the historical cycle and the current cycle, and correcting the health change data not generated in the health curve according to the average deviation; The average of the corrected health change data corresponding to the future time period is calculated. When the average is lower than the lower limit of the health indicator of the current resource group, the current message node is marked as being in the expected busy state, otherwise the expected busy state is cleared; The proportion of the number of expected busy message nodes in the current resource group to the total number of message nodes in the current resource group is counted. When the proportion is higher than the preset threshold, the current resource group is marked as in a resource emergency state, and the message nodes in the resource pool are locked to achieve preemption. Otherwise, the resource emergency state of the current resource group is cleared, and the current resource group unlocks the message nodes in the resource pool.

5. The message middle station resource scheduling method according to claim 4 is characterized in that: New message nodes are configured from the resource pool for resource groups with higher business priorities, including: Detect whether the current resource group is in a resource emergency state. When in the resource emergency state, determine the message node locked by the current resource group in the resource pool, and configure the message node as a new message node of the current resource group; When the resource is not in an emergency state, an idle message node that is not locked by other resource groups is determined from the resource pool, the message node is first locked, and then configured as a new message node of the current resource group; When the configuration of the newly added message node fails, an alarm notification message is sent to the preset communication interface.

6. The message middle station resource scheduling method according to claim 1, characterized in that: New message nodes are configured from the resource pool for resource groups with higher business priorities, including: Identify multiple message nodes that are not locked in the resource pool as potential new nodes; Perform stress tests on each potential new node, and select the potential new node with the best stress test results as the target new node for the resource group with the highest business priority that needs to be expanded; After the target newly added node completes the expansion of the resource group to which it belongs, the load of each message node in the resource group is redistributed.

7. The message middle station resource scheduling method according to any one of claims 1 to 6, characterized in that: The method further comprises: receiving configuration information of a new service, and determining a service type and a service priority of the new service from the configuration information; According to the determined service priority, query the resource group corresponding to the new service from the service isolation list; In the node queue of the queried resource group, randomly select a message node whose health is higher than the upper limit of the health index to create a partition of the topic of the new business; The service type and service priority of the new service are associated with the corresponding resource group and added to the service isolation list.

8. A message middle station resource scheduling device, characterized in that: include: A list acquisition module is configured to acquire a business isolation list, wherein the business isolation list includes business types, business priorities, and resource groups corresponding to multiple businesses, each resource group includes multiple message nodes for distributed storage of message flows of each topic of its corresponding business, and each resource group is provided with a matching health indicator upper limit, health indicator lower limit, and initialized health degree corresponding to its business priority; A sorting and scheduling module is configured to dynamically adjust the health of each message node according to multiple real-time load indicators of the message nodes in each resource group, and adjust the sorting of each message node in the node queue corresponding to each resource group according to the adjusted health; A topic creation module is configured to respond to a business topic creation request, query the business isolation list according to the business type in the request to determine the corresponding target resource group, determine multiple message nodes with a high health ranking and a health level higher than the corresponding health indicator upper limit as target nodes in the node queue of the target resource group, and create a partition of the topic specified by the request between the target nodes; The resource expansion module is configured to determine the overall health of the resource group based on the health of all message nodes in each resource group. When the overall health is lower than the corresponding lower limit of the health indicator, new message nodes are configured from the resource pool for resource groups with higher business priorities according to the business priorities of the resource groups.

9. A message middle station resource scheduling device, comprising a central processing unit and a memory, characterized in that: The central processing unit is used to call and run the computer program stored in the memory to execute the steps of the method according to any one of claims 1 to 7.

10. A non-volatile readable storage medium, characterized in that: It stores a computer program implemented according to the method described in any one of claims 1 to 7 in the form of computer-readable instructions, and when the computer program is called and executed by a computer, the steps included in the corresponding method are executed.

Images