Message middleware layout method, device, server and storage medium

Abstract

The embodiments of the present invention relate to the field of information and communication technology, and disclose a layout method, device, server and storage medium for a message middleware. In the present invention, a coupling set of topics is obtained; wherein, the topics in each coupling set have a coupling relationship; all topics are decoupled and grouped according to the coupling set of topics to obtain multiple topic sets; the multiple topic sets are respectively deployed in each message middleware container cluster, wherein one topic set is deployed in one message middleware container cluster. Through the above technical means, the topics of the message middleware are decoupled and grouped, and the message middleware is distributedly deployed in a cluster group composed of a message middleware cluster, so that the service pressure of multiple services accessing the message middleware is dispersed, the coupling between multiple services accessing the message middleware is reduced, and the working stability of the message middleware is improved, while also simplifying the deployment process of the message middleware.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of information and communication technology, and in particular to a layout method, device, server, and storage medium for message middleware. Background Art

[0002] Message-based middleware, also known as message queue middleware, is an intermediary component that uses efficient and reliable messaging mechanisms for platform-independent data exchange. It comprises a message server (Broker), producers (Producers), consumers (Consumers), and consumption topics (Topics). The Broker provides core messaging services for the server, the Producer is responsible for producing messages, the Consumer is responsible for consuming and processing messages, and Topics aggregate messages in a publish-subscribe model. In multi-business system scenarios, using message-based middleware to deliver messages can effectively address business coupling. When interacting with message-based middleware, message-based middleware services used by multiple services or in service-based architectures typically utilize a shared single node or multi-node cluster to consume and produce messages. As the importance of service data increases, the number of message replicas increases, increasing the throughput of writes to the message-based middleware and the corresponding I / O pressure.

[0003] However, in the process of using message middleware, there may be interactions between production and consumption of services. However, as business needs increase, the number of services also increases. There is an inevitable coupling between services, which makes the working stability of message middleware unable to meet user needs. Taking the business scenario of message middleware Kafka transmitting messages between multiple services as an example, the following problems often exist in the process of using message middleware: (1) The first scenario is as follows Figure 1 As shown in the figure, there are Service A and Service B. In actual business, A is both the producer and consumer of B; B is both the producer and consumer of A. A and B share the same Kafka service, which will cause coupling between (A, B). (2) The second scenario is as follows Figure 2 As shown in the figure, there are Service A, Service B, and Service C. B is a consumer of A and a producer of C. When the two groups of services access the same Kafka service, when the version is upgraded or some Kafka failure occurs, the two groups of services will not be able to use the Kafka service. This is the coupling between (A, B) and (B, C). (3) The third scenario is as follows Figure 3As shown, there are Service A, Service B, Service C, and Service D. Because the business data of (A, B) and (C, D) differ in importance, the number of replicas for Topic 1 and Topic 2 also differs. The number of partition replicas for each topic is positively correlated with the importance of the business data. The more important the business data, the more partition replicas are needed. Due to the characteristics of Kafka itself, the number of partition replicas is also the number of copies. Excessive partition replicas can increase throughput but also increase write pressure. Furthermore, the cumbersome environment configuration and deployment operations also pose technical challenges for developers. Summary of the Invention

[0004] The main purpose of the embodiments of the present application is to propose a layout method for message middleware, so that the service pressure of multiple services accessing the message middleware is dispersed, the coupling between multiple services accessing the message middleware is reduced, and the working stability of the message middleware is improved.

[0005] To achieve the above-mentioned purpose, an embodiment of the present application provides a layout method for a message middleware, including: obtaining a coupled set of topics; wherein the topics in each coupled set have a coupling relationship; decoupling and grouping all topics according to the coupled set of topics to obtain multiple topic sets; and deploying the multiple topic sets in each message middleware container cluster respectively, wherein one topic set is deployed in one message middleware container cluster.

[0006] To achieve the above-mentioned purpose, an embodiment of the present application also provides a layout device for a message middleware, including: an acquisition module for obtaining a coupled set of topics; wherein the topics in each coupled set have a coupling relationship; a grouping module for decoupling and grouping all topics according to the coupled set of topics to obtain multiple topic sets; a cluster deployment module for deploying multiple topic sets in each message middleware container cluster respectively, wherein one topic set is deployed in one message middleware container cluster.

[0007] To achieve the above-mentioned purpose, an embodiment of the present application also provides a server, including: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the layout method of the message middleware as described above.

[0008] To achieve the above-mentioned purpose, an embodiment of the present application further provides a computer-readable storage medium storing a computer program, which implements the above-mentioned message middleware layout method when executed by a processor.

[0009] The message middleware layout method proposed in this application decouples and groups the topics of the message middleware, and distributes the message middleware in a cluster group composed of message middleware clusters, so that the service pressure of multiple services accessing the message middleware is dispersed. Moreover, since the topics deployed in a message middleware cluster are not coupled topics, the coupling between multiple services accessing the message middleware can be reduced, thereby improving the working stability of the message middleware and simplifying the deployment process of the message middleware. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 This is a schematic diagram of a message transmission path of a message middleware in the first scenario according to the background technology of the present invention;

[0011] Figure 2 This is a schematic diagram of a message delivery path of a message middleware in the second scenario according to the background technology of the present invention;

[0012] Figure 3 This is a schematic diagram of a message delivery path of a message middleware in the third scenario according to the background technology of the present invention;

[0013] Figure 4 is a flow chart of a method for laying out a message middleware according to a first embodiment of the present invention;

[0014] Figure 5 is a flow chart of a method for laying out a message middleware according to a second embodiment of the present invention;

[0015] Figure 6 This is a flowchart of decoupling and grouping all topics using a greedy algorithm in a message middleware layout method according to the second embodiment of the present invention;

[0016] Figure 7 2 is a schematic diagram of the structure of a layout device for a message middleware according to a third embodiment of the present invention;

[0017] Figure 8 2 is a schematic diagram of the structure of a container multi-cluster distributed system of a message middleware according to a fourth embodiment of the present invention;

[0018] Figure 9 FIG. 5 is a schematic structural diagram of a server according to a fifth embodiment of the present invention. DETAILED DESCRIPTION

[0019] In order to make the purpose, technical solutions and advantages of the embodiments of the present application clearer, each embodiment of the present application will be described in detail below with reference to the accompanying drawings. However, it will be understood by those skilled in the art that in each embodiment of the present application, many technical details are proposed to enable the reader to better understand the present application. However, even without these technical details and various changes and modifications based on the following embodiments, the technical solutions claimed in the present application can be implemented. The division of the following embodiments is for convenience of description and should not constitute any limitation on the specific implementation of the present application. The various embodiments can be combined and referenced with each other under the premise of no contradiction.

[0020] The first embodiment of the present invention relates to a method for laying out a message middleware, comprising: obtaining metadata of each consumption topic; generating a coupled set of topics based on the metadata of the topics; wherein the topics in each coupled set have a coupling relationship; decoupling and grouping all topics based on the coupled sets of topics to obtain multiple topic sets; and deploying the multiple topic sets in each message middleware container cluster, wherein one topic set is deployed in one message middleware container cluster. In this embodiment, the method for laying out the message middleware is applied to a multi-cluster distributed system, and the execution subject is a server in the system that serves as a message middleware container hosting node. The message middleware is illustrated by taking Kafka, a high-throughput distributed publish-subscribe messaging system, as an example.

[0021] The following further describes this embodiment with reference to the accompanying drawings. The layout method of the message middleware in this embodiment is as follows: Figure 4 As shown, specifically including:

[0022] Step 401: Obtain metadata of each consumption topic.

[0023] Specifically, topic metadata describes topic attributes. These attributes include: the topic's capacity, which measures the importance of business data; the set of producers for the topic, which represents the set of services that produce the topic; and the set of services that consume the topic. The topic's capacity refers to the number of backups or copies of the topic's messages.

[0024] In Kafka, topic capacity is typically measured by the number of replicas. This metric also serves as a proxy for the importance of business data. To ensure disaster recovery, the more critical the business data, the greater the number of replicas. Alternatively, other types of message-based middleware can employ pre-defined importance scores, with operations personnel assigning them to each topic. In this case, topic importance is also described in the topic's metadata as attribute information.

[0025] Step 402: Generate a topic coupling set based on the topic metadata.

[0026] Specifically, a coupling set describes the coupling relationships between services in the service topic metadata. To obtain the specific coupling relationships between all topics, you need to first establish multiple topic coupling sets. In each topic coupling set, all topics have coupling relationships.

[0027] Furthermore, in one example, a coupling set is typically established by describing the relationship between services using topics. The intersection of the producer and consumer sets under all topic metadata is then calculated. If the intersection is not empty, the services of the two topics are considered coupled and added to the coupling set. In other words, topics with the same service in the producer or consumer service set are combined into a coupling set. After traversing the metadata of all topics, a complete topic coupling set is generated. This operation is then continued for the remaining topics until all topics are included in the topic coupling set.

[0028] In one example, service metadata can also be used to describe service attributes. The attributes described by the service metadata include: basic information about the service, a Topic set of which the service is a producer, and a Topic set of which the service is a consumer. It can be found that both service metadata and Topic metadata can describe the relationship between the service and the message middleware, the difference being the direction of description. Therefore, in this embodiment, a Topic coupling set can also be generated based on service metadata, or a Topic coupling set can be generated based on service metadata and Topic metadata. It is worth noting that in this embodiment, whether service metadata or Topic metadata is used to generate a Topic coupling set, the coupling relationship between services is described with Topic as the center.

[0029] In one example, topic metadata and service metadata can be represented by service message tuples and topic message tuples, respectively.

[0030] Among them, Topic message tuple Ti=(Rc i , PrSet i , CoSet i ), where TC (Topic Capacity) represents the capacity of the Topic; ProSet (Producer Set) represents the producer set under the Topic, which includes all services that produce the Topic; ConSet (Consumer Set) represents the consumer set under the Topic, which includes all services that consume the Topic.

[0031] Service message tuple S = {Info, ProTopic, ConTopic}, where Info represents basic service information, ProTopic represents the Topic set as a producer, and ConTopic represents the Topic set as a consumer.

[0032] In one example, the process of generating a Topic coupling set by using a service message tuple and a Topic message tuple includes: establishing an association relationship between the service and the Topic according to the Topic message tuple and the service message tuple: Relation = (T i , P.S. j , CS k ), generate the association relationship set {(T i , P.S. j , CS k )}, according to the coupling relationship definition, PS j and CS j Belong to the same service, so the association relationship can be processed to generate the coupling set {(T i , T j )}.

[0033] Step 403: Decouple and group all topics according to their coupled sets to obtain multiple topic sets.

[0034] Specifically, to reduce the coupling between services after deployment, topics need to be decoupled and grouped. The primary approach is to group all coupled topics into a single set, known as a topic group. During deployment, this topic group is then deployed as a cluster. This eliminates the coupling between topic clusters and improves the stability of the message middleware within the distributed system. This grouping approach also minimizes the number of topic sets after grouping, improving the efficiency of message middleware deployment.

[0035] Step 404: deploy multiple Topic sets in each message middleware container cluster, wherein one Topic set is deployed in one message middleware container cluster.

[0036] Specifically, decoupling groups to create multiple topic sets can be understood as defining the specific layout for deploying these topics. Furthermore, each topic set can deploy a varying number of message middleware containers, forming message middleware clusters of varying sizes. The number of containers in a message middleware cluster can be determined based on the number of topics or the expected load of the topic cluster.

[0037] In one example, based on the characteristic that larger topic capacity indicates higher importance, to ensure disaster recovery capabilities for business data and the reliability of the message middleware service, the average capacity of a topic set can be used as an indicator to measure the importance of the entire topic set. This, in turn, determines the number of containers required to deploy the message middleware container cluster for that topic set based on the importance of the topic set. Average capacity is the ratio of the topic set capacity to the number of topics. A cluster with a larger average capacity indicates that the message middleware messages in that cluster are more important and require more containers, meaning the cluster is larger. Ultimately, the number of message backups in the cluster is the product of the number of middleware containers and the number of topic replicas. A large number of data backups ensures the reliability of important topics.

[0038] In actual applications, assume that Topic set A = {T1, T2, T3, T4, T5} and Topic set B = {T6, T7, T8, T9}. Therefore, there is a message middleware cluster S corresponding to set A and set B. A and S B Assuming that the number of replicas of T1, T2, T3, T4, and T5 are 9, 2, 3, 7, and 9 respectively, the cluster S A Average capacity The number of replicas of T6, T7, T8, and T9 are 4, 1, 8, and 3 respectively, so cluster S B Average capacity The ratio of the preset average capacity to the number of message middleware containers is 2:1, so the final message middleware cluster S A Allocate 3 containers for S B Allocating two containers achieves even load distribution among the containers and reduces the read and write pressure on the container carrying the more important topic.

[0039] It should be noted that the above examples in this embodiment are only illustrative for ease of understanding and do not limit the technical solutions of the present invention.

[0040] Compared to existing technologies, the messaging middleware layout method in this embodiment decouples and groups all topics, placing coupled topics into a messaging middleware container cluster. This reduces the coupling between the messaging middleware container clusters that deploy each topic, thereby improving the overall reliability of the messaging middleware clusters and preventing the possibility of widespread service outages caused by the downtime of a single container cluster. Furthermore, more containers are allocated to topic clusters with high average capacity, i.e., container clusters with high importance, further ensuring the reliability of important services.

[0041] The second embodiment of the present invention relates to a layout method for a message middleware, which is substantially the same as the first embodiment of the present invention, with the main difference being that in this embodiment, all topics are decoupled and grouped according to the coupling set of the topic to obtain multiple topic sets, including: using an improved greedy algorithm to decouple and group the topics. In this embodiment, the selection condition of the improved greedy algorithm is that the currently traversed topic and the topics selected in this traversal do not belong to the same coupling set. The termination conditions include: the sum of the capacities of the topics selected in this traversal is greater than or equal to the preset capacity and the initial topic set is empty.

[0042] The following further describes this embodiment with reference to the accompanying drawings. The layout method of the message middleware in this embodiment is as follows: Figure 5 As shown, specifically including:

[0043] Step 501: Obtain metadata of each consumption topic.

[0044] Step 502: Generate a topic coupling set based on the topic metadata.

[0045] Step 501 and step 502 are the same as step 401 and step 402 in the first embodiment of the present invention. The relevant implementation details have been specifically described in the first embodiment of the present invention and will not be repeated in this embodiment.

[0046] Step 503: Using an improved greedy algorithm, all topics are decoupled and grouped according to their coupled sets to obtain multiple topic sets.

[0047] Specifically, the greedy algorithm in this embodiment is characterized by preprocessing the traversal objects. Specifically, all topics are sorted by importance to generate an ordered topic set, referred to as the initial set in this embodiment. The greedy strategy of the greedy algorithm executes the traversal process according to the order of the elements in the set. In addition, the selection conditions in the greedy strategy are: the currently traversed topic and the topics selected in this traversal do not belong to the same coupling set; the termination conditions are: the total capacity of the topics selected in this traversal is greater than or equal to the preset capacity, and the initial set is empty.

[0048] In an example, a greedy algorithm is used to decouple and group all topics according to their coupled sets, and the process of obtaining multiple topic sets is as follows: Figure 6 As shown, specifically including:

[0049] Step 5031: sort all topics according to their importance to obtain an initial set.

[0050] Specifically, using Kafka as a messaging middleware, the importance of a topic is positively correlated with the number of replicas it has. That is, the importance of a topic is represented by the number of replicas. Topics in the initial set are sorted by importance. Subsequent greedy algorithms will access topics in the initial set in the same order.

[0051] Step 5032: Create a new Topic collection.

[0052] Specifically, the newly created Topic set is an empty set. When a Topic that meets the selection criteria is accessed according to the subsequent greedy algorithm, the Topic is selected and added to the Topic set.

[0053] Step 5033: traverse the initial set and access the topics in the initial set in sequence.

[0054] Step 5034: Determine whether the currently accessed Topic belongs to the same coupling set as the selected Topic in the Topic set. If so, continue with step 5033 and traverse the initial set. If not, proceed to step 5035.

[0055] Step 5035: Select the currently accessed Topic and add it to the Topic set.

[0056] Specifically, each time a topic is accessed, all selected topics in the newly created topic set are searched for topics that are coupled with the currently accessed topic, based on the coupling set to which the topic belongs. If a topic with a coupling relationship already exists, the currently accessed topic is not added to the topic set, and the next topic in the initial set is accessed sequentially. If the currently accessed topic does not have a coupling relationship with any topic already in the newly created topic set, the currently accessed topic is added to the newly created topic set.

[0057] Step 5036, determine whether the traversal is completed; if the traversal is completed, execute step 5037; if the traversal is not completed, continue to execute step 5033 and traverse the initial set.

[0058] Specifically, after the traversal has accessed all topics in the initial set, the Topic set is saved, that is, a Topic set has been obtained through the traversal.

[0059] In one example, before creating a new Topic collection, you can set an upper capacity limit for the Topic collection. Before the traversal ends, determine whether the capacity of the Topic collection after the current Topic is added to the Topic collection is greater than or equal to the collection's upper capacity limit. If the capacity of the Topic collection is greater than or equal to the collection's upper capacity limit, then the traversal ends and the current Topic collection is directly saved. This technical approach allows you to control the capacity of a single Topic collection, thereby controlling the number of containers required to deploy a single Topic's message middleware container cluster while minimizing the number of Topic collections. This avoids having too many important Topics in a cluster and reduces the probability of large-scale service failures for important Topics.

[0060] Step 5037, update the initial set and determine whether the updated initial set is empty; if the updated initial set is empty, execute step 5038; if the updated initial set is not empty, execute step 5032 again to create a new empty Topic set.

[0061] Step 5038: Output all Topic sets.

[0062] Specifically, the initial set is updated by taking the difference between the topics in the initial set and all topics in the acquired topic set as the updated initial set. When the updated initial set is empty, it indicates that all topics have been decoupled and grouped. At this point, the greedy algorithm terminates and no longer executes. All existing topic sets are output and used as the layout basis for deploying the message middleware container cluster.

[0063] Furthermore, because the topics in the initial set are sorted by importance, the topics in the first set generated will be more important than those in subsequent sets. This greedy algorithm is highly efficient and prioritizes assigning more important topics to the same set, reducing the number of generated sets. Furthermore, when deploying message middleware containers corresponding to important topic sets, allocating more containers improves the reliability of these more important topics, simplifying the deployment and maintenance strategies for message middleware clusters.

[0064] Step 504: deploy multiple Topic sets in each message middleware container cluster respectively.

[0065] Step 504 is the same as step 404 in the first embodiment of the present invention. The relevant implementation details have been specifically described in the first embodiment of the present invention and will not be repeated in this embodiment.

[0066] The steps of the various methods above are divided only for the purpose of clear description. During implementation, they can be combined into one step or some steps can be split and decomposed into multiple steps. As long as they include the same logical relationship, they are all within the scope of protection of this patent. Adding insignificant modifications or introducing insignificant designs to the algorithm or process without changing the core design of the algorithm and process are all within the scope of protection of this patent.

[0067] Compared to existing technologies, the message middleware layout method in this embodiment employs an improved greedy algorithm when decoupling and grouping all topics. Topics in the initial set traversed by the greedy algorithm are sorted, prioritizing high-importance topics to be grouped into the same topic set. When deploying the corresponding message middleware containers for this topic set, more containers are allocated, reducing the pressure on important topic access service messages and further improving the reliability of the important message middleware. This also simplifies the deployment and maintenance strategies for message middleware clusters by operations and maintenance personnel.

[0068] The third embodiment of the present invention relates to a layout device of a message middleware, such as Figure 7 Shown, including:

[0069] The acquisition module 701 is used to acquire a coupled set of topics; wherein the topics in each coupled set have a coupling relationship.

[0070] In an example, the generating module 701 may also be configured to generate a Topic coupling set based on a service message tuple, or based on both a service message tuple and a Topic message tuple.

[0071] In another example, the generating module 701 may also be used to generate topics with the same service in the producer service set or the consumer service set into a coupling set.

[0072] The grouping module 702 is used to decouple and group all topics according to their coupled sets to obtain multiple topic sets.

[0073] In one example, the grouping module 702 can also be used to decouple and group topics using an improved greedy algorithm. The specific process of decoupling and grouping includes: sorting all topics according to their importance to obtain an initial set; traversing the initial set using a greedy strategy and placing the selected topics in the same topic set; wherein, the selection condition of the greedy strategy is: the currently traversed topic and the topics selected in this traversal do not belong to the same coupling set; the difference between the topics included in the initial set and all topics in the acquired topic set is used as the updated initial set; repeatedly traversing the updated initial set using the greedy strategy to obtain a new topic set and an update of the initial set, until the updated initial set is empty.

[0074] The cluster deployment module 703 is used to deploy multiple Topic sets in each message middleware container cluster respectively, wherein one Topic set is deployed in one message middleware container cluster.

[0075] In an example, the cluster deployment module 703 may also be used to determine the number N of message middleware containers required for the deployed message middleware container cluster based on the number of topics in the Topic set and the number of copies of each Topic; and deploy the Topic set in the N message middleware containers.

[0076] In another example, the cluster deployment module 704 can also be used to calculate the average capacity of the topic set based on the number of topics and the number of copies of each topic backup; determine N based on the average capacity of the topic set, where N is positively correlated with the average capacity of the topic set.

[0077] It is not difficult to find that this embodiment is a virtual device embodiment corresponding to the first embodiment or the second embodiment, and this embodiment can be implemented in conjunction with the first embodiment or the second embodiment. The relevant technical details mentioned in the first embodiment or the second embodiment are still valid in this embodiment, and to reduce repetition, they are not repeated here. Accordingly, the relevant technical details mentioned in this embodiment can also be applied to the first embodiment or the second embodiment.

[0078] It is worth noting that all modules involved in this embodiment are logical modules. In actual applications, a logical unit can be a physical unit, a part of a physical unit, or a combination of multiple physical units. Furthermore, to highlight the innovations of the present invention, this embodiment does not include units that are not closely related to solving the technical problems proposed by the present invention. However, this does not mean that other units do not exist in this embodiment.

[0079] A fourth embodiment of the present invention relates to a multi-cluster distributed system of a message middleware container. This embodiment may include a message middleware layout device described in the third embodiment of the present invention.

[0080] The following further describes this embodiment with reference to the accompanying drawings. Figure 8 Shown, including:

[0081] The basic environment module 801 is used to automatically build and maintain the automatically compiled operating environment, build tools, installation environment, and image operating environment through a one-click script.

[0082] Specifically, the construction of the basic environment includes both the construction of the compilation environment and the construction and installation of the container environment. This module mainly builds the compilation environment according to the requirements of the Kafka source code, including the detection and installation of the Java environment, the adaptation and installation of the Scala environment, and the adaptation and installation of the Gradle environment. Since the Kafka source code has a recommended installation environment version, it is necessary to adapt the environment installation package according to the Gradle configuration file to build the compilation environment; in addition, it is necessary to build a container environment, which is mainly the docker and docker-compose environment.

[0083] The source code compiling module 802 is used to compile and package the customized open source message middleware source code into a compressed package through a series of standardized operations.

[0084] Specifically, based on the completion of the basic environment module, the source code compilation module imports the personalized customized Kafka source code developed and tested by the user, uses a one-click compilation script, starts a series of standardized compilation actions, and uses the Gradle compilation tool to package Kafka into an installation package.

[0085] Customized message middleware source code refers to compilable source code that has passed all tests and is developed based on the user's business scenario requirements or requires recompilation of open-source message middleware source code. A series of table conversion operations involves encapsulating the construction of the basic environment and compilation environment build tools through automated scripts. This automated script includes the installation of the basic environment based on source code requirements, as well as updates to the message middleware source code, the extraction of compilation dependencies, and the packaging of the installation package.

[0086] The image generation module 803 is used to generate the message middleware image supporting the system, the basic image required by the system, and the upper-layer application image supporting the message middleware.

[0087] Specifically, the container images required for topic collection and collection monitoring are packaged. The installation package used by the Kafka image packaged by this module comes from the installation package generated by the source code compilation module. The installation packages of the other two images, Zookeeper and Kafka-Eagle, need to be pulled from the official website; building images is mainly executed through Shell scripts and Dockerfile files, and the images are updated to the image repository for use in the production environment.

[0088] The service management module 804 is used to reallocate services to different message middleware clusters based on the importance of multiple service business data and the mutual coupling relationship of services.

[0089] Specifically, the service management module mainly implements the rearrangement of the Kafka cluster assigned to each microservice. The module first abstracts the microservice information data into the Topic microservice metadata, extracts the coupling relationship between microservices through the production Topic set and the consumption Topic set, and rearranges the position of the microservice Topic in the cluster by using an improved greedy algorithm centered on the Topic.

[0090] The cluster deployment module 805 is used to deploy the laid out topics on the message middleware multi-cluster through the container cluster metadata to realize the container cluster group.

[0091] Furthermore, the cluster deployment module is also used to build a microservice layout cluster of the image generated by the image generation module through cluster management tools (docker-compose, docker-manage) to form a container cluster group. The cluster group is composed of multiple container cluster elements. According to the concept of container cluster element, each container cluster corresponds to a Topic set.

[0092] The security authentication module 806 is used to implement secure authentication access of the client through dynamic SSL certificate update.

[0093] Specifically, the security authentication module in this embodiment is mainly used to manage the dynamic update of user client SSL authentication, Zookeeper ACL authorization control, and the security authentication required by Kafka itself.

[0094] The unified interface module 807 is used to provide interface access for external clients through encapsulated interface configuration.

[0095] Specifically, the unified interface module in this embodiment provides a unified interface for the client and server for the cloud management platform. The server interface is mainly used in the actual deployment environment, and the Topic is created in each cluster environment. The interface includes a series of parameters for Topic creation and can be created through user customization; the client interface includes the configuration parameter interface for producer and consumer clients to connect to the Kafka server.

[0096] The operation monitoring module 808 is used to implement message middleware cluster monitoring using an existing open source message middleware monitoring tool.

[0097] Specifically, the operation monitoring module mainly uses the Kafka monitoring container built by the mirrored Kafka-Eagle image to monitor the Kafka topic status, the throughput of production and consumption messages, the status of the container cluster, and event alarms.

[0098] It is worth noting that all modules involved in this embodiment are logical modules. In actual applications, a logical unit can be a physical unit, a part of a physical unit, or a combination of multiple physical units. Furthermore, to highlight the innovations of the present invention, this embodiment does not include units that are not closely related to solving the technical problems proposed by the present invention. However, this does not mean that other units do not exist in this embodiment.

[0099] A fifth embodiment of the present invention relates to a server, such as Figure 9As shown, it includes: at least one processor 901; and a memory 902 communicatively connected to the at least one processor 901; wherein the memory 902 stores instructions that can be executed by the at least one processor 901, and the instructions are executed by the at least one processor 901 to enable the at least one processor 901 to execute the layout method of the message middleware in the first or second embodiment.

[0100] Memory 902 and processor 901 are connected via a bus. The bus can include any number of interconnected buses and bridges, connecting various circuits within one or more processors 901 and memory 902. The bus can also connect various other circuits, such as peripheral devices, voltage regulators, and power management circuits. These are well known in the art and are not described further herein. A bus interface provides an interface between the bus and a transceiver. A transceiver can be a single component or multiple components, such as multiple receivers and transmitters, providing a means for communicating with various other devices over a transmission medium. Data processed by processor 901 is transmitted over a wireless medium via an antenna. Furthermore, the antenna receives data and transmits it to processor 901. Processor 901 is responsible for bus management and general processing, and can also provide various functions, including timing, peripheral interfacing, voltage regulation, power management, and other control functions. Memory 902 can be used to store data used by processor 901 when performing operations.

[0101] A sixth embodiment of the present invention relates to a computer-readable storage medium storing a computer program, which implements the above method embodiment when executed by a processor.

[0102] That is, those skilled in the art will understand that all or part of the steps in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a program, which is stored in a storage medium and includes a number of instructions for causing a device (which may be a single-chip microcomputer, chip, etc.) or a processor to execute all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage medium includes: a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, etc., various media that can store program code.

[0103] Those skilled in the art will appreciate that the above embodiments are specific embodiments for implementing the present invention, and that in actual applications, various changes may be made thereto in form and detail without departing from the spirit and scope of the present invention.

Claims

1. A layout method for message middleware, applied to a multi-cluster distributed system, characterized in that: Obtain a coupled set of topics; wherein each topic in the coupled set has the same services, including a service that produces the topic and a service that consumes the topic; Decoupling and grouping all the topics according to the topic coupling sets to obtain multiple topic sets, where topics in the topic sets belong to different coupling sets; The plurality of Topic sets are respectively deployed in each message middleware container cluster, wherein one Topic set is deployed in one message middleware container cluster.

2. The layout method of message middleware according to claim 1, characterized in that: Decoupling and grouping all the topics according to the topic coupling set to obtain multiple topic sets, including: Sort all the topics according to their importance to obtain an initial set; The initial set is traversed using a greedy strategy, and the selected topics are placed in the same topic set. The selection condition of the greedy strategy is that the currently traversed topic and the topics selected in this traversal do not belong to the same coupling set. The difference between the topics included in the initial set and all topics in the acquired topic set is used as the updated initial set; Repeat the traversal of the greedy strategy on the updated initial set to obtain a new Topic set and an update of the initial set until the updated initial set is empty.

3. The message middleware layout method according to claim 2, characterized in that: The adopting a greedy strategy to traverse the initial set includes: If the total capacity of the selected topics in this traversal is greater than or equal to the preset capacity, the traversal ends.

4. The layout method of the message middleware according to claim 2 or 3, characterized in that: The importance of the topic is positively correlated with the number of copies of the topic backup.

5. The message middleware layout method according to claim 1, characterized in that: Before obtaining the coupled set of topics, the following steps are also included: Determine the producer service set and consumer service set of the Topic based on the metadata of the Topic; Topics with the same service in the producer service set or the consumer service set are generated into a coupling set.

6. The message middleware layout method according to claim 1, characterized in that: Deploying the plurality of Topic sets in respective message middleware container clusters includes: Determine the number N of message middleware containers required for the deployed message middleware container cluster based on the number of topics in the topic set and the number of replicas of each topic; Deploy the Topic set in N message middleware containers.

7. The layout method of message middleware according to claim 6, characterized in that: Determining the number N of message middleware containers required for the deployed message middleware container cluster based on the number of topics in the topic set and the number of replicas of each topic includes: Calculate the average capacity of the topic set based on the number of topics and the number of copies of each topic backup; The N is determined according to the average capacity of the Topic set, wherein the N is positively correlated with the average capacity of the Topic set.

8. A layout device for message middleware, characterized in that: include: An acquisition module is used to acquire a coupled set of topics; wherein each topic in the coupled set has the same services, including a service for producing the topic and a service for consuming the topic; A grouping module, configured to decouple and group all the topics according to the coupling sets of the topics to obtain multiple topic sets, where topics in the topic sets belong to different coupling sets; The cluster deployment module is used to deploy the plurality of Topic sets in each message middleware container cluster respectively, wherein one Topic set is deployed in one message middleware container cluster.

9. A server, characterized in that: include: at least one processor; as well as, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the message middleware layout method according to any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the layout method of the message middleware according to any one of claims 1 to 7 is implemented.

Images