Theme allocation method and device applied to Rocket MQ cluster, server and medium

Abstract

The embodiment of the invention discloses a theme distribution method and device applied to a Rocket MQ cluster, a server side and a medium, and the method comprises the steps: obtaining to-be-processed data of all Rocket MQ clusters under different evaluation dimensions when a theme distribution request is received; for different assessment dimensions, determining assessment attributes under the assessment dimensions by analyzing and processing the to-be-processed data under the assessment dimensions; performing weighting processing on the evaluation attribute of each Rocket MQ cluster to obtain a target use attribute of each Rocket MQ cluster; and according to all the target use attributes, determining a target Rocket MQ cluster corresponding to the theme allocation request, and allocating a target theme corresponding to the theme allocation request to the target Rocket MQ cluster for processing. According to the technical scheme provided by the embodiment of the invention, the target Rocket MQ cluster corresponding to the to-be-allocated theme is determined according to the target use attribute of each Rocket MQ cluster, and the technical effect of utilizing the Rocket MQ cluster to the maximum extent is achieved.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of data processing technology, and in particular to a topic allocation method, device, server and medium applied to a RocketMQ cluster. Background Art

[0002] Currently, in the management of large-scale RocketMQ clusters, cross-cluster topic allocation and creation are relatively important. Unreasonable topic allocation will cause problems such as large differences in loads of different RocketMQ clusters and uneven overall server resource utilization.

[0003] Currently, for the management of large-scale RocketMQ clusters, the allocation and creation of topics are specified by manually configured routing rules. In other words, when allocating and creating topics, the selection of RocketMQ clusters is entirely up to the user to analyze and judge the operating status of all RocketMQ clusters and finally select the appropriate RocketMQ cluster.

[0004] All of the above methods require human participation, resulting in high labor costs and possible errors in manual allocation, which greatly reduces the efficiency of subject allocation and causes excessive differences in resource utilization. Summary of the invention

[0005] The embodiments of the present invention provide a method, device, server and medium for allocating topics applied to a RocketMQ cluster, which achieve the effect of effectively and conveniently determining a target RocketMQ cluster corresponding to a topic.

[0006] In a first aspect, an embodiment of the present invention provides a topic allocation method applied to a RocketMQ cluster, the method comprising:

[0007] When receiving a topic assignment request, obtain the pending data of all RocketMQ clusters under different evaluation dimensions;

[0008] For different evaluation dimensions, by analyzing and processing the data to be processed under the evaluation dimensions, the evaluation attributes under the evaluation dimensions are determined;

[0009] By weighting the evaluation attributes of each RocketMQ cluster, the target usage attributes of each RocketMQ cluster are obtained;

[0010] According to all target usage attributes, a target RocketMQ cluster corresponding to the topic allocation request is determined, and the target topic corresponding to the topic allocation request is allocated to the target RocketMQ cluster for processing.

[0011] Further, the evaluation dimensions include CPU usage dimension, memory usage dimension, disk usage dimension, traffic usage dimension, and assigned topic attribute dimension;

[0012] Correspondingly, the data to be processed includes occupied data corresponding to the central processing unit usage dimension, occupied memory data corresponding to the memory usage dimension, disk usage data corresponding to the disk usage dimension, the traffic usage dimension includes the traffic data occupied by the processed data, and the allocated topic attribute dimension includes the number of allocated topics.

[0013] Furthermore, the step of analyzing and processing the data to be processed under the evaluation dimension to determine the evaluation attribute under the evaluation dimension includes:

[0014] Determine the evaluation attribute under the CPU usage dimension according to the occupied amount data under the CPU usage dimension and the total usage data allocated to the RocketMQ cluster;

[0015] Determine the evaluation attribute corresponding to the memory usage dimension according to the occupied memory data corresponding to the memory usage dimension and the total memory data allocated to the RocketMQ cluster;

[0016] Determine the evaluation attribute of the disk usage dimension according to the disk usage data corresponding to the disk usage dimension and the available disk usage allocated to the RocketMQ cluster;

[0017] Determine the evaluation attribute of the traffic usage dimension according to the traffic data corresponding to the traffic usage dimension and the total traffic data allocated to the RocketMQ cluster;

[0018] Determine the evaluation attribute of the allocated topic attribute dimension according to the number of allocated topics and the upper limit of the total number of topics of the allocated topic attribute dimension.

[0019] Furthermore, before obtaining the target usage attribute of each RocketMQ cluster by weighted processing of the evaluation attributes of each RocketMQ cluster, the method further includes:

[0020] Corresponding usage weights are assigned to the multiple evaluation dimensions to determine the target usage attribute of the RocketMQ cluster based on the usage weight corresponding to each evaluation dimension.

[0021] Furthermore, by weighting the evaluation attributes of each RocketMQ cluster, the target usage attributes of each RocketMQ cluster are obtained, including:

[0022] Obtain the usage weight corresponding to each evaluation dimension;

[0023] For each RocketMQ cluster, according to the evaluation attributes of the RocketMQ cluster in each evaluation dimension and the corresponding usage weight, determine the attributes to be superimposed of the RocketMQ cluster in each evaluation dimension;

[0024] The target usage attribute of the RocketMQ cluster is obtained by superimposing a plurality of attributes to be superimposed of the RocketMQ cluster.

[0025] Further, determining the target RocketMQ cluster corresponding to the topic allocation request based on all target usage attributes includes:

[0026] The RocketMQ cluster corresponding to the minimum target usage attribute is used as the target RocketMQ cluster, and the target topic corresponding to the topic assignment request is assigned to the target RocketMQ cluster, so that the RocketMQ cluster processes the message corresponding to the target topic.

[0027] In a second aspect, an embodiment of the present invention further provides a topic allocation device applied to a RocketMQ cluster, the device comprising:

[0028] The pending data acquisition module is used to obtain the pending data of all RocketMQ clusters under different evaluation dimensions when receiving a topic allocation request;

[0029] An evaluation attribute determination module is used to determine the evaluation attributes under different evaluation dimensions by analyzing and processing the data to be processed under the evaluation dimensions;

[0030] The target usage attribute determination module is used to obtain the target usage attribute of each RocketMQ cluster by weighting the evaluation attributes of each RocketMQ cluster;

[0031] The topic allocation module is used to determine the target RocketMQ cluster corresponding to the topic allocation request according to all target usage attributes, and allocate the target topic corresponding to the topic allocation request to the target RocketMQ cluster for processing.

[0032] In a third aspect, an embodiment of the present invention provides a server, which includes:

[0033] one or more processors;

[0034] A memory for storing one or more programs;

[0035] When the one or more programs are executed by the one or more processors, the one or more processors implement the topic allocation method applied to the RocketMQ cluster as provided in any embodiment of the present invention.

[0036] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements a topic allocation method applied to a RocketMQ cluster as provided in any embodiment of the present invention.

[0037] In a fifth aspect, an embodiment of the present invention further provides a computer program product, including a computer program, characterized in that when the computer program is executed by a processor, it implements the topic allocation method applied to the RocketMQ cluster as described in any one of the embodiments of the present invention.

[0038] The technical solution provided by the embodiment of the present invention can obtain the data to be processed of all RocketMQ clusters under different evaluation dimensions after receiving the topic allocation request. Next, for the data to be processed under different evaluation dimensions, the evaluation attributes under the evaluation dimensions are determined by analyzing and processing the data to be processed under the evaluation dimensions. Again, the evaluation attributes of each RocketMQ cluster are weighted to obtain the target usage attributes of each RocketMQ cluster. According to all the target usage attributes, the target RocketMQ cluster corresponding to the topic allocation request is determined, which solves the problem of high cost of manual participation when determining the target RocketMQ cluster corresponding to the topic to be allocated in the prior art based on manual methods, and the problem that certain errors may be generated when allocating topics based on manual methods, resulting in low topic allocation efficiency and excessive resource utilization differences. It is achieved that the target RocketMQ cluster that is most suitable for the topic to be allocated can be determined based on the actual application data of each RocketMQ cluster, thereby improving the resource utilization of the RocketMQ cluster and the effect of automatic allocation. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the technical solutions of the exemplary embodiments of the present invention, the following briefly introduces the drawings required to describe the embodiments. Obviously, the drawings introduced are only drawings of a part of the embodiments to be described in the present invention, rather than all the drawings. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0040] Figure 1 A flowchart of a topic allocation method applied to a RocketMQ cluster provided by an embodiment of the present invention;

[0041] Figure 2A flowchart of a topic allocation method applied to a RocketMQ cluster provided by an embodiment of the present invention;

[0042] Figure 3 A schematic diagram of the structure of a topic allocation device applied to a RocketMQ cluster provided by an embodiment of the present invention;

[0043] Figure 4 A schematic diagram of the structure of a server provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0044] The present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the present invention, rather than to limit the present invention. It should also be noted that, for ease of description, only parts related to the present invention, rather than all structures, are shown in the accompanying drawings.

[0045] The present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the present invention, rather than to limit the present invention. It should also be noted that, for ease of description, only parts related to the present invention, rather than all structures, are shown in the accompanying drawings.

[0046] Before introducing the technical solution provided by the embodiment of the present invention, an application scenario may be exemplified first.

[0047] For a large-scale RocketMQ cluster, if it is necessary to allocate a corresponding target RocketMQ cluster for a topic to be allocated, the solution provided in this embodiment can be used to achieve this.

[0048] For each RocketMQ cluster, the number of topics that can be supported does not exceed a preset value. Based on this, the method provided in this embodiment can be used to determine a target RocketMQ cluster that is compatible with the topic to be allocated.

[0049] At present, the process of allocating and creating topics in large-scale RocketMQ clusters relies on manual analysis and judgment of the running status of RocketMQ clusters to configure the routing rules for creating topics, and the human factor has a relatively large impact. At the same time, in the process of use, when facing hundreds of topic accesses, not only does it increase the labor cost, but manual allocation may also cause errors, greatly reducing the efficiency of topic allocation and causing the problem of excessive differences in resource utilization.

[0050] Before introducing the detailed scheme of this embodiment, we can first explain the RocketMQ cluster. Multiple RocketMQ clusters can be deployed on the same server, or one RocketMQ cluster can be deployed on different servers. For each RocketMQ cluster, its target usage attribute can be determined, and then the target RocketMQ cluster corresponding to the target topic can be determined.

[0051] Figure 1 A flowchart of a topic allocation method applied to a RocketMQ cluster provided in an embodiment of the present invention. This embodiment can be applied to a scenario of determining a target RocketMQ cluster corresponding to a topic to be allocated. The topic allocation method applied to a RocketMQ cluster provided in this embodiment can be executed by a topic allocation device applied to a RocketMQ cluster integrated in a client and / or a server.

[0052] like Figure 1 As shown, the method specifically comprises the following steps:

[0053] S110. When a topic allocation request is received, the to-be-processed data of all RocketMQ clusters under different evaluation dimensions are obtained.

[0054] The topic allocation request can be understood as a request to allocate a RocketMQ cluster to the topic to be allocated. Optionally, when it is detected that a corresponding RocketMQ cluster needs to be allocated to the topic to be allocated, a corresponding control can be triggered to generate a topic allocation request.

[0055] Different RocketMQ clusters have different usage data, and the data to be processed under different evaluation dimensions of the RocketMQ cluster can be obtained. In other words, for the RocketMQ cluster, the data to be processed under multiple evaluation dimensions can be obtained. The data to be processed at this time can be the usage data when the RocketMQ cluster processes different topics.

[0056] Specifically, when it is detected that a RocketMQ cluster needs to be allocated for a topic to be allocated, a display interface can be triggered or a corresponding topic allocation request can be generated based on a certain program code. After receiving the topic allocation request, the server can obtain the pending data of all RocketMQ clusters under different evaluation dimensions.

[0057] In this embodiment, the reason and benefit of obtaining the data to be processed under different evaluation dimensions is that it is possible to determine whether the RocketMQ cluster is suitable for re-assigning topics to it from different evaluation dimensions, thereby achieving the effect of reasonably allocating topics to the RocketMQ cluster.

[0058] S120: For different evaluation dimensions, determine the evaluation attributes under the evaluation dimensions by analyzing and processing the data to be processed under the evaluation dimensions.

[0059] It should be noted that the number of evaluation dimensions may include multiple ones, and the specific number of evaluation dimensions is adapted to the resources occupied by the RocketMQ cluster when it is running.

[0060] For each RocketMQ cluster, the data to be processed under different evaluation dimensions can be obtained. Then, for the RocketMQ cluster, the data to be processed under different evaluation dimensions of the RocketMQ cluster can be analyzed and processed to obtain the evaluation attributes of the RocketMQ cluster under each evaluation dimension.

[0061] The evaluation attribute is used to characterize whether the RocketMQ cluster can be used under the evaluation dimension. Optionally, a higher evaluation attribute indicates that the RocketMQ cluster can be used under the evaluation dimension, and vice versa, indicates that the RocketMQ cluster cannot be used under the evaluation dimension.

[0062] It should be noted that the method for determining the evaluation attributes of each RocketMQ cluster is the same. In this embodiment, determining the evaluation attributes of one RocketMQ cluster is taken as an example for description.

[0063] Specifically, after obtaining the data to be processed under different evaluation dimensions of the RocketMQ cluster, the data to be processed under different evaluation dimensions can be analyzed and processed to obtain evaluation data corresponding to the evaluation dimensions.

[0064] S130 . Obtain target usage attributes of each RocketMQ cluster by weighted processing of the evaluation attributes of each RocketMQ cluster.

[0065] Among them, for different evaluation dimensions, the weights they occupy when used in the RocketMQ cluster are different. Therefore, there are certain differences in the weight values ​​corresponding to different evaluation dimensions. Of course, if the importance of the RocketMQ cluster in different evaluation dimensions is similar, then the weight values ​​corresponding to different evaluation dimensions are the same. The target usage attribute is used to evaluate the usage attributes of the RocketMQ cluster. Optionally, the higher the target usage attribute, the higher the availability of the RocketMQ cluster. Conversely, the lower the availability of the RocketMQ cluster. In other words, the target usage attribute is used to evaluate the evaluation value of whether the RocketMQ cluster can be used.

[0066] Specifically, after obtaining the evaluation attributes corresponding to the RocketMQ cluster under each evaluation dimension, the weight value corresponding to each evaluation dimension can be retrieved. According to the evaluation attributes corresponding to different evaluation dimensions and the corresponding weight values, the target usage attributes of the RocketMQ cluster can be obtained.

[0067] S140: Determine a target RocketMQ cluster corresponding to the topic assignment request according to all target usage attributes, and assign the target topic corresponding to the topic assignment request to the target RocketMQ cluster for processing.

[0068] The target RocketMQ cluster is a cluster selected from multiple RocketMQ clusters. The target topic is a topic to be assigned corresponding to the topic assignment request. The target RocketMQ cluster is a RocketMQ cluster that carries the target topic.

[0069] Specifically, after obtaining the target usage attributes of all RocketMQ clusters, the target RocketMQ cluster that meets the conditions can be screened out from all RocketMQ clusters, and the target topic corresponding to the topic allocation request is allocated to the target RocketMQ cluster, so as to execute the data corresponding to the target topic based on the target RocketMQ cluster.

[0070] The technical solution provided by the embodiment of the present invention can obtain the data to be processed of all RocketMQ clusters under different evaluation dimensions after receiving the topic allocation request. Next, for the data to be processed under different evaluation dimensions, the evaluation attributes under the evaluation dimensions are determined by analyzing and processing the data to be processed under the evaluation dimensions. Again, the evaluation attributes of each RocketMQ cluster are weighted to obtain the target usage attributes of each RocketMQ cluster. According to all the target usage attributes, the target RocketMQ cluster corresponding to the topic allocation request is determined, which solves the problem of high cost of manual participation when determining the target RocketMQ cluster corresponding to the topic to be allocated in the prior art based on manual methods, and the problem that certain errors may be generated when allocating topics based on manual methods, resulting in low topic allocation efficiency and excessive resource utilization differences. It is achieved that the target RocketMQ cluster that is most suitable for the topic to be allocated can be determined based on the actual application data of each RocketMQ cluster, thereby improving the resource utilization of the RocketMQ cluster and the effect of automatic allocation.

[0071] Figure 2A flowchart of a topic allocation method applied to a RocketMQ cluster is provided in an embodiment of the present invention. Based on the aforementioned embodiment, the above steps can be further refined. The specific implementation method can refer to the detailed description of this embodiment, wherein the technical terms that are the same as or corresponding to the above embodiment are not repeated in this embodiment.

[0072] like Figure 2 As shown, the method includes:

[0073] S210: When a topic allocation request is received, the to-be-processed data of all RocketMQ clusters under different evaluation dimensions are obtained.

[0074] In this embodiment, different evaluation dimensions may include at least five evaluation dimensions. Optionally, the at least five evaluation dimensions include a CPU usage dimension, a memory usage dimension, a disk usage dimension, a traffic usage dimension, and an allocated topic attribute dimension. Accordingly, the data to be processed corresponding to different evaluation dimensions may be: occupied data corresponding to the CPU usage dimension, occupied memory data corresponding to the memory usage dimension, disk usage data corresponding to the disk usage dimension, the traffic usage dimension including traffic data occupied by the processed data, and the allocated topic attribute dimension including the number of allocated topics.

[0075] It should be noted that: since the RocketMQ cluster is deployed on the server side, the target usage attributes of the RocketMQ cluster can be evaluated from the perspective of the server side. Furthermore, when the RocketMQ cluster is deployed on the server side, corresponding usage parameters can be configured for it. Optionally, the usage parameters can be CPU usage parameters, memory usage parameters, disk usage parameters, traffic usage parameters, and topic quantity parameters. Based on this, the target usage attributes of the RocketMQ cluster can be determined from these dimensions.

[0076] Among them, the central processing unit dimension can be the CPU usage dimension of the CPU usage space allocated to it, the memory usage dimension can be the dimension corresponding to the memory allocated to it, the disk usage dimension can be the dimension corresponding to the disk allocated to it, the traffic usage dimension can be the dimension of the traffic allocated to it, and the allocated topic dimension is the dimension of the topic allocated to it.

[0077] You can obtain the data to be processed corresponding to different evaluation dimensions. The CPU usage dimension is the occupied data corresponding to the CPU allocated to the RocketMQ cluster. The data to be processed in the memory usage dimension is the occupied memory data corresponding to the memory allocated to the RocketMQ cluster. The data to be processed in the disk usage dimension is the disk usage data corresponding to the disk data allocated to the RocketMQ cluster. The traffic usage dimension is the traffic data occupied by the traffic allocated to the RocketMQ cluster when processing the processed data. The allocated topic attribute dimension is the number of allocated topics allocated to the RocketMQ cluster.

[0078] After obtaining the data to be processed corresponding to the above evaluation dimensions, the data to be processed may be analyzed and processed to obtain evaluation attributes corresponding to different evaluation dimensions.

[0079] S220: Determine the evaluation attribute under the CPU usage dimension according to the occupied amount data under the CPU usage dimension and the total usage data allocated to the RocketMQ cluster.

[0080] It can be understood as: the occupied amount under the CPU usage dimension and the total usage data pre-allocated for the RocketMQ cluster can be obtained. The ratio between the occupied amount data and the total usage data is calculated to obtain the evaluation attribute under the CPU usage dimension.

[0081] S230. Determine an evaluation attribute corresponding to the memory usage dimension according to the occupied memory data corresponding to the memory usage dimension and the total memory data allocated to the RocketMQ cluster.

[0082] It can be understood as: the current memory usage data and the total memory data allocated to the RocketMQ cluster can be obtained. The ratio between the memory usage data and the total memory data is calculated to obtain the evaluation attribute of the memory usage dimension.

[0083] S240: Determine an evaluation attribute of the disk usage dimension according to the disk usage data corresponding to the disk usage dimension and the available disk capacity allocated to the RocketMQ cluster.

[0084] It can be understood that: the disk usage data of the disk usage dimension and the available disk data allocated to the RocketMQ cluster can be obtained. By calculating the ratio between the disk usage data and the available disk data, the evaluation attribute of the disk usage dimension is obtained.

[0085] S250: Determine an evaluation attribute of the traffic usage dimension according to the traffic data corresponding to the traffic usage dimension and the total traffic data allocated to the RocketMQ cluster.

[0086] It can be understood as follows: the traffic usage data of the traffic usage dimension, that is, the traffic data, and the total traffic data allocated to the RocketMQ cluster can be obtained. The ratio between the traffic usage data and the total traffic data is calculated to obtain the evaluation attribute of the traffic usage dimension.

[0087] S260. Determine the evaluation attribute of the allocated topic attribute dimension according to the number of allocated topics and the upper limit of the total number of topics of the allocated topic attribute dimension.

[0088] It should be noted that for each RocketMQ cluster, there is an upper limit on the number of topics it can carry, which can be used as the upper limit of the total number of topics.

[0089] It can be understood as: the number of topics currently allocated to the RocketMQ cluster and the upper limit of the total number of topics that the RocketMQ cluster can carry can be obtained. The ratio of the number of allocated topics to the upper limit of the total number of topics is calculated, and this ratio is used as the evaluation attribute of the allocated topic attribute dimension.

[0090] S270. Assign corresponding usage weights to the multiple evaluation dimensions to determine the target usage attribute of the RocketMQ cluster based on the usage weight corresponding to each evaluation dimension.

[0091] The weight value pre-assigned to each evaluation dimension can be obtained, and this weight value is the usage weight corresponding to each evaluation dimension.

[0092] Optionally, the target usage attributes of the RocketMQ cluster are determined based on the usage weight corresponding to each evaluation dimension, including: obtaining the usage weight corresponding to each evaluation dimension; for each RocketMQ cluster, determining the attributes to be superimposed of the RocketMQ cluster under each evaluation dimension according to the evaluation attributes of the RocketMQ cluster under each evaluation dimension and the corresponding usage weight; and obtaining the target usage attributes of the RocketMQ cluster by superimposing multiple attributes to be superimposed of the RocketMQ cluster.

[0093] For each evaluation dimension, the product of the evaluation attribute and the usage weight under the evaluation dimension can be calculated, and the value obtained at this time is used as the attribute to be superimposed. After superimposing the attributes to be superimposed of all evaluation dimensions under the RocketMQ cluster, the target usage attribute of the RocketMQ cluster can be obtained.

[0094] The target usage attributes of each RocketMQ cluster can be determined based on the above method.

[0095] S280. The RocketMQ cluster corresponding to the minimum target usage attribute is used as the target RocketMQ cluster, and the target topic corresponding to the topic assignment request is assigned to the target RocketMQ cluster, so that the RocketMQ cluster processes the message corresponding to the target topic.

[0096] It can be understood that: for each RocketMQ cluster, there is a corresponding target usage attribute. The smaller the target usage attribute, the more topics it can carry. Based on this, the RocketMQ cluster with the smallest target usage attribute can be used as the final target RocketMQ cluster.

[0097] The technical solution provided by the embodiment of the present invention can obtain the data to be processed of all RocketMQ clusters under different evaluation dimensions after receiving the topic allocation request. Next, for the data to be processed under different evaluation dimensions, the evaluation attributes under the evaluation dimensions are determined by analyzing and processing the data to be processed under the evaluation dimensions. Again, the evaluation attributes of each RocketMQ cluster are weighted to obtain the target usage attributes of each RocketMQ cluster. According to all the target usage attributes, the target RocketMQ cluster corresponding to the topic allocation request is determined, which solves the problem of high cost of manual participation when determining the target RocketMQ cluster corresponding to the topic to be allocated in the prior art based on manual methods, and the problem that certain errors may be generated when allocating topics based on manual methods, resulting in low topic allocation efficiency and excessive resource utilization differences. It is achieved that the target RocketMQ cluster that is most suitable for the topic to be allocated can be determined based on the actual application data of each RocketMQ cluster, thereby improving the resource utilization of the RocketMQ cluster and the effect of automatic allocation.

[0098] As an optional embodiment of the above embodiment, a specific example can be used to introduce how to allocate a corresponding target RocketMQ cluster to a target topic. The specific implementation method can refer to the detailed description of the present technical solution, wherein the technical terms that are the same as or corresponding to the above embodiment are not repeated in this embodiment.

[0099] First, obtain the data to be processed under the monitoring indicators. That is, collect the data to be processed under different monitoring indicators of all RocketMQ clusters. Optionally, the monitoring indicators correspond to evaluation dimensions. The evaluation dimensions (monitoring indicators) include the central processing unit usage dimension (CPU usage dimension), the memory usage dimension, the disk usage dimension, and the allocated topic attribute dimension. After obtaining the data to be processed under the above multiple dimensions, the data to be processed can be analyzed and processed to obtain the evaluation attributes under different evaluation dimensions.

[0100] In this embodiment, the evaluation attributes corresponding to different evaluation dimensions can be expressed as percentages. Optionally, the evaluation attribute of the CPU usage dimension can be: the CPU usage data is divided by the total usage data allocated by the RocketMQ cluster to obtain the CPU usage data. usage , whose value range is between [0, 1]; the evaluation attribute of the memory usage attribute can be: the occupied memory data is divided by the total memory data allocated by the RocketMQ cluster to get Memory usage , whose value range is between [0, 1]; the evaluation attribute of the disk usage dimension can be: the disk usage data is divided by the available disk allocated by the RocketMQ cluster to obtain Disk usage , and its value range is also between [0, 1]. The evaluation attribute of the traffic usage dimension can be: the used power data of the RocketMQ cluster is divided by the total traffic data allocated to the RocketMQ cluster, and TX is obtained. usage , the value range is between [0, 1]; the evaluation attribute of the allocated topic attribute dimension can be: the number of allocated topics that have been allocated to the RocketMQ cluster divided by the upper limit of the total number of topics that the RocketMQ cluster can carry, to obtain the topic load factor TC usage , the value range is between [0, 1].

[0101] Based on the above method, the evaluation attributes under different monitoring indicators can be obtained.

[0102] Next, configure the corresponding weight values ​​for different monitoring indicators. That is, set the weight of each monitoring indicator.

[0103] In this embodiment, the usage weight corresponding to the CPU usage dimension is W cpu , the usage weight corresponding to the memory usage dimension is W memory , the usage weight corresponding to the disk usage dimension is W disk , the usage weight corresponding to the traffic usage dimension is W tx , the usage weight of the assigned topic attribute dimension is W tc The above usage weights are all within the range of [0,1], and the sum of all usage weights is the preset value 1.

[0104] After obtaining the evaluation attributes and usage weights corresponding to each evaluation dimension, the target usage attributes of each RocketMQ cluster can be determined in the following manner.

[0105] Optionally, the formula can be: EV = CPU usage W cpu +Memory usage W memory +Disk usageW disk +TX usage W tx +TC usage W tc；

[0106] Among them, EV is the target usage attribute of the RocketMQ cluster.

[0107] Finally, after obtaining the target usage attributes of all RocketMQ clusters, the RocketMQ cluster corresponding to the minimum target usage attribute can be used as the target RocketMQ cluster. The target topic corresponding to the topic allocation request is allocated to the target RocketMQ cluster with the minimum target usage attribute, and the topic is created.

[0108] The technical solution provided by the embodiment of the present invention is a topic allocation method based on comprehensive weighted evaluation of multiple monitoring indicator dimensions, which can automatically and efficiently select the optimal RocketMQ cluster, effectively avoid the mistakes that may occur in the manual evaluation process, and reduce the cost of manual evaluation, thereby improving the topic access efficiency of large-scale RocketMQ clusters and keeping the overall server resource utilization balanced.

[0109] The following is an embodiment of a topic allocation device applied to a RocketMQ cluster provided by an embodiment of the present invention. The device and the topic allocation method applied to a RocketMQ cluster in the above-mentioned embodiments belong to the same inventive concept. For details not described in detail in the embodiment of the topic allocation device applied to a RocketMQ cluster, please refer to the above-mentioned embodiment of the topic allocation method applied to a RocketMQ cluster.

[0110] Figure 3 A structural diagram of a topic allocation device applied to a RocketMQ cluster provided in an embodiment of the present invention, the device specifically includes: a to-be-processed data acquisition module 310, an evaluation attribute determination module 320, a target usage attribute determination module 330 and a topic allocation module 340.

[0111] Among them, the to-be-processed data acquisition module 310 is used to obtain the to-be-processed data of all RocketMQ clusters under different evaluation dimensions when a topic allocation request is received; the evaluation attribute determination module 320 is used to determine the evaluation attributes under the evaluation dimensions by analyzing and processing the to-be-processed data under the evaluation dimensions for different evaluation dimensions; the target usage attribute determination module 330 is used to obtain the target usage attributes of each RocketMQ cluster by weighted processing of the evaluation attributes of each RocketMQ cluster; the topic allocation module 340 is used to determine the target RocketMQ cluster corresponding to the topic allocation request based on all target usage attributes, and allocate the target topic corresponding to the topic allocation request to the target RocketMQ cluster for processing.

[0112] Based on the above technical solution, the evaluation dimensions include CPU usage dimension, memory usage dimension, disk usage dimension, traffic usage dimension and assigned subject attribute dimension;

[0113] Correspondingly, the data to be processed includes occupied data corresponding to the central processing unit usage dimension, occupied memory data corresponding to the memory usage dimension, disk usage data corresponding to the disk usage dimension, the traffic usage dimension includes the traffic data occupied by the processed data, and the allocated topic attribute dimension includes the number of allocated topics.

[0114] Based on the above technical solutions, the evaluation attribute determination module includes:

[0115] A first determining unit, configured to determine an evaluation attribute under the CPU usage dimension according to the occupied amount data under the CPU usage dimension and the total usage data allocated to the RocketMQ cluster;

[0116] A second determining unit, configured to determine an evaluation attribute corresponding to the memory usage dimension according to the occupied memory data corresponding to the memory usage dimension and the total memory data allocated to the RocketMQ cluster;

[0117] A third determining unit is used to determine the evaluation attribute of the disk usage dimension according to the disk usage data corresponding to the disk usage dimension and the available disk usage allocated to the RocketMQ cluster;

[0118] A fourth determining unit, configured to determine an evaluation attribute of the traffic usage dimension according to the traffic data corresponding to the traffic usage dimension and the total traffic data allocated to the RocketMQ cluster;

[0119] The fifth determining unit is used to determine the evaluation attribute of the allocated topic attribute dimension according to the number of allocated topics and the upper limit of the total number of topics of the allocated topic attribute dimension.

[0120] Based on the above technical solutions, the device also includes a usage weight acquisition module, which is used to assign corresponding usage weights to the multiple evaluation dimensions, so as to determine the target usage attribute of the RocketMQ cluster based on the usage weight corresponding to each evaluation dimension.

[0121] Based on the above technical solutions, the target usage attribute determination module includes:

[0122] A usage weight acquisition unit is used to acquire the usage weight corresponding to each evaluation dimension;

[0123] A superimposed attribute determination unit, configured to determine, for each RocketMQ cluster, the attributes to be superimposed of the RocketMQ cluster under each evaluation dimension according to the evaluation attributes of the RocketMQ cluster under each evaluation dimension and the corresponding usage weights;

[0124] The target usage attribute determination unit is used to obtain the target usage attribute of the RocketMQ cluster by superimposing multiple attributes to be superimposed of the RocketMQ cluster.

[0125] On the basis of the above technical solutions, the topic allocation module is also used to use the RocketMQ cluster corresponding to the minimum target usage attribute as the target RocketMQ cluster, so as to allocate the target topic corresponding to the topic allocation request to the target RocketMQ cluster, so that the RocketMQ cluster processes the message corresponding to the target topic. The technical solution provided by the embodiment of the present invention can obtain the data to be processed of all RocketMQ clusters under different evaluation dimensions after receiving the topic allocation request. Next, for the data to be processed under different evaluation dimensions, the evaluation attributes under the evaluation dimensions are determined by analyzing and processing the data to be processed under the evaluation dimensions. Again, the evaluation attributes of each RocketMQ cluster are weighted to obtain the target usage attributes of each RocketMQ cluster. According to all the target usage attributes, the target RocketMQ cluster corresponding to the topic allocation request is determined, which solves the problem of high cost of manual participation when determining the target RocketMQ cluster corresponding to the topic to be allocated in the prior art based on manual methods, and the problem that certain errors may be generated when allocating topics based on manual methods, resulting in low topic allocation efficiency and excessive resource utilization differences. It is achieved that the target RocketMQ cluster that is most suitable for the topic to be allocated can be determined based on the actual application data of each RocketMQ cluster, thereby improving the resource utilization of the RocketMQ cluster and the effect of automatic allocation.

[0126] The topic allocation device applied to the RocketMQ cluster provided in the embodiment of the present invention can execute the topic allocation method applied to the RocketMQ cluster provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of executing the topic allocation method applied to the RocketMQ cluster.

[0127] It is worth noting that in the above-mentioned embodiment of the topic allocation device applied to the RocketMQ cluster, the various units and modules included are divided only according to functional logic, but are not limited to the above-mentioned division, as long as the corresponding functions can be realized; in addition, the specific names of the functional units are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of the present invention.

[0128] Figure 4 A schematic diagram of the structure of a server provided in an embodiment of the present invention. Figure 4 A block diagram of an exemplary server 12 suitable for use in implementing embodiments of the present invention is shown. Figure 4 The server 12 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0129] like Figure 4As shown, the server 12 is in the form of a general purpose computing device. The components of the server 12 may include, but are not limited to: one or more processors or processing units 16, a system memory 28, and a bus 18 that connects various system components (including the system memory 28 and the processing unit 16).

[0130] Bus 18 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processor or a local bus using any of a variety of bus architectures. By way of example, these architectures include, but are not limited to, an Industry Standard Architecture (ISA) bus, a Micro Channel Architecture (MAC) bus, an Enhanced ISA bus, a Video Electronics Standards Association (VESA) local bus, and a Peripheral Component Interconnect (PCI) bus.

[0131] The server 12 typically includes a variety of computer system readable media. These media can be any available media that can be accessed by the server 12, including volatile and non-volatile media, removable and non-removable media.

[0132] The system memory 28 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 30 and / or cache memory 32. The server 12 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, the storage system 34 may be used to read and write non-removable, non-volatile magnetic media ( Figure 4 not shown, usually called a "hard drive"). Although Figure 4 Not shown in the figure, a disk drive for reading and writing to a removable non-volatile disk (e.g., a "floppy disk"), and an optical disk drive for reading and writing to a removable non-volatile optical disk (e.g., a CD-ROM, a DVD-ROM, or other optical media) may be provided. In these cases, each drive may be connected to the bus 18 via one or more data medium interfaces. The system memory 28 may include at least one program product having a set (e.g., at least one) of program modules that are configured to perform the functions of the various embodiments of the present invention.

[0133] A program / utility 40 having a set (at least one) of program modules 42 may be stored, for example, in system memory 28, such program modules 42 including, but not limited to, an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment. Program modules 42 generally perform the functions and / or methods of the embodiments described herein.

[0134] The server 12 may also communicate with one or more external devices 14 (e.g., keyboards, pointing devices, displays 24, etc.), may communicate with one or more devices that enable a user to interact with the server 12, and / or may communicate with any device that enables the server 12 to communicate with one or more other computing devices (e.g., network cards, modems, etc.). Such communication may be performed via an input / output (I / O) interface 22. In addition, the server 12 may also communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) via a network adapter 20. As shown, the network adapter 20 communicates with other modules of the server 12 via a bus 18. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the server 12, including, but not limited to, microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0135] The processing unit 16 executes various functional applications and data processing by running the program stored in the system memory 28, for example, implementing the topic allocation method steps applied to the RocketMQ cluster provided in the first embodiment of the present invention, the method comprising:

[0136] When receiving a topic assignment request, obtain the pending data of all RocketMQ clusters under different evaluation dimensions;

[0137] For different evaluation dimensions, by analyzing and processing the data to be processed under the evaluation dimensions, the evaluation attributes under the evaluation dimensions are determined;

[0138] By weighting the evaluation attributes of each RocketMQ cluster, the target usage attributes of each RocketMQ cluster are obtained;

[0139] According to all target usage attributes, a target RocketMQ cluster corresponding to the topic allocation request is determined, and the target topic corresponding to the topic allocation request is allocated to the target RocketMQ cluster for processing.

[0140] Of course, those skilled in the art can understand that the processor can also implement the technical solution of the topic allocation method applied to the RocketMQ cluster provided by any embodiment of the present invention.

[0141] This embodiment provides a computer-readable storage medium, on which a computer program is stored. When the program is executed by a processor, the method steps of the topic allocation method applied to the RocketMQ cluster provided in the above embodiment of the present invention are implemented. The method includes:

[0142] When receiving a topic assignment request, obtain the pending data of all RocketMQ clusters under different evaluation dimensions;

[0143] For different evaluation dimensions, by analyzing and processing the data to be processed under the evaluation dimensions, the evaluation attributes under the evaluation dimensions are determined;

[0144] By weighting the evaluation attributes of each RocketMQ cluster, the target usage attributes of each RocketMQ cluster are obtained;

[0145] According to all target usage attributes, a target RocketMQ cluster corresponding to the topic allocation request is determined, and the target topic corresponding to the topic allocation request is allocated to the target RocketMQ cluster for processing.

[0146] The computer storage medium of the embodiment of the present invention can adopt any combination of one or more computer-readable media. The computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples (non-exhaustive list) of computer-readable storage media include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this document, a computer-readable storage medium can be any tangible medium containing or storing a program, which can be used by an instruction execution system, device or device or used in combination with it.

[0147] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, which carry computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0148] Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0149] Computer program code for performing the operation of embodiments of the present invention may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as an independent software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0150] Note that the above are only preferred embodiments of the present invention and the technical principles used. Those skilled in the art will understand that the present invention is not limited to the specific embodiments herein, and that various obvious changes, readjustments and substitutions can be made by those skilled in the art without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in more detail through the above embodiments, the present invention is not limited to the above embodiments, and may include more other equivalent embodiments without departing from the concept of the present invention, and the scope of the present invention is determined by the scope of the appended claims.

Claims

1. A topic allocation method applied to a RocketMQ cluster, characterized in that: include: When receiving a topic assignment request, obtain the pending data of all RocketMQ clusters under different evaluation dimensions; For different evaluation dimensions, by analyzing and processing the data to be processed under the evaluation dimensions, the evaluation attributes under the evaluation dimensions are determined; By weighting the evaluation attributes of each RocketMQ cluster, the target usage attributes of each RocketMQ cluster are obtained; According to all target usage attributes, a target RocketMQ cluster corresponding to the topic allocation request is determined, and the target topic corresponding to the topic allocation request is allocated to the target RocketMQ cluster for processing.

2. The method according to claim 1, characterized in that The evaluation dimensions include CPU usage, memory usage, disk usage, traffic usage, and assigned topic attribute dimensions; Correspondingly, the data to be processed includes occupied data corresponding to the central processing unit usage dimension, occupied memory data corresponding to the memory usage dimension, disk usage data corresponding to the disk usage dimension, the traffic usage dimension includes the traffic data occupied by the processed data, and the allocated topic attribute dimension includes the number of allocated topics.

3. The method according to claim 2, characterized in that The step of analyzing and processing the data to be processed under the evaluation dimension to determine the evaluation attribute under the evaluation dimension includes: Determine the evaluation attribute under the CPU usage dimension according to the occupied amount data under the CPU usage dimension and the total usage data allocated to the RocketMQ cluster; Determine the evaluation attribute corresponding to the memory usage dimension according to the occupied memory data corresponding to the memory usage dimension and the total memory data allocated to the RocketMQ cluster; Determine the evaluation attribute of the disk usage dimension according to the disk usage data corresponding to the disk usage dimension and the available disk usage allocated to the RocketMQ cluster; Determine the evaluation attribute of the traffic usage dimension according to the traffic data corresponding to the traffic usage dimension and the total traffic data allocated to the RocketMQ cluster; Determine the evaluation attribute of the allocated topic attribute dimension according to the number of allocated topics and the upper limit of the total number of topics of the allocated topic attribute dimension.

4. The method according to claim 1, characterized in that Before obtaining the target usage attribute of each RocketMQ cluster by weighted processing of the evaluation attributes of each RocketMQ cluster, the method further includes: Corresponding usage weights are assigned to the plurality of evaluation dimensions, so as to determine the target usage attribute of the RocketMQ cluster based on the usage weight corresponding to each evaluation dimension.

5. The method according to claim 1, characterized in that The target usage attributes of each RocketMQ cluster are obtained by weighting the evaluation attributes of each RocketMQ cluster, including: Obtain the usage weight corresponding to each evaluation dimension; For each RocketMQ cluster, according to the evaluation attributes of the RocketMQ cluster in each evaluation dimension and the corresponding usage weight, determine the attributes to be superimposed of the RocketMQ cluster in each evaluation dimension; The target usage attribute of the RocketMQ cluster is obtained by superimposing a plurality of attributes to be superimposed of the RocketMQ cluster.

6. The method according to claim 1, characterized in that Determining the target RocketMQ cluster corresponding to the topic allocation request according to all target usage attributes includes: The RocketMQ cluster corresponding to the minimum target usage attribute is used as the target RocketMQ cluster, and the target topic corresponding to the topic assignment request is assigned to the target RocketMQ cluster, so that the RocketMQ cluster processes the message corresponding to the target topic.

7. A topic allocation device applied to a RocketMQ cluster, characterized in that: include: The pending data acquisition module is used to obtain the pending data of all RocketMQ clusters under different evaluation dimensions when receiving a topic allocation request; An evaluation attribute determination module is used to determine the evaluation attributes under different evaluation dimensions by analyzing and processing the data to be processed under the evaluation dimensions; The target usage attribute determination module is used to obtain the target usage attribute of each RocketMQ cluster by weighting the evaluation attributes of each RocketMQ cluster; The topic allocation module is used to determine the target RocketMQ cluster corresponding to the topic allocation request according to all target usage attributes, and allocate the target topic corresponding to the topic allocation request to the target RocketMQ cluster for processing.

8. A server, characterized in that: The server includes: one or more processors; A memory for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the topic allocation method applied to the RocketMQ cluster as described in any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, a topic allocation method applied to a RocketMQ cluster as described in any one of claims 1 to 6 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by the processor, it implements the topic allocation method applied to the RocketMQ cluster as described in any one of claims 1 to 6.

Images