Message theme management method and data bus system thereof

Abstract

The invention provides a message theme management method and a data bus system thereof, and the method comprises the steps: constructing a Kafka message mechanism which comprises a data source, a message theme, a user, a business group and authorization, and maintaining the metadata information of the message theme; user information and service group information are set, and centralized authorization management of operation authority of a message theme is carried out on a service group; performing permission judgment on the user according to user information input by the user, and allowing the authorized user to perform message operation corresponding to the permission; and collecting state information of the message theme, and carrying out early warning pushing according to a preset early warning rule. According to the method, the complexity of message management can be reduced, the operation is simple and visual, and the method is suitable for operation and maintenance management of Internet enterprises on message themes.

Description

Technical Field

[0001] The invention belongs to the field of information data technology, and in particular relates to a message subject management method and a data bus system thereof. Background Art

[0002] With the rapid development of the Internet in the information age, computer hardware is developing rapidly, network and storage space capabilities are becoming stronger and stronger, and data exchange between users and enterprises, as well as data exchange within enterprises, is becoming more frequent and important. Data exchange has various protocols such as http, https, tcp and other synchronous methods, as well as asynchronous data exchange methods based on message queues.

[0003] The development of the Internet is reflected in the rapid changes and iterations of business. In order to ensure the agility of business, it is often forced to continuously accelerate the progress of R&D in technology and compress the life cycle of technology R&D. Enterprises continue to propose new technical methodologies based on the optimization of development processes. At present, microservice solutions based on front-end and back-end separation and domain splitting are the most popular. This leads to the continuous extension of the link of a complex business, and even derives many branch links. However, users do need a fast and effective system response experience. This part of the contradiction prompts enterprises to prioritize the data consistency of the main link, and the branch links are often processed in an asynchronous way. In the information age, a large amount of valuable data such as customer behavior, normal logical data of the system, system log data, and binlog logs of data need to be collected and processed to mine valuable information for business operators as a reference for business development direction. The processing method of these two parts is usually completed by producers producing data to a topic in the message queue, and then processing it through consumers subscribing to the topic. As the enterprise grows and develops, there are more and more message queue clusters, topics, producers, and consumers. As time goes by, more and more metadata information is lost, which makes it difficult to govern message topics. On the one hand, it is difficult to divide the permissions of message topics according to business service areas. On the other hand, when a message topic encounters a business scenario that requires changing the topic attributes, it is impossible to estimate the impact of the change. At the same time, as the number of message topics continues to increase, the workload of operation and maintenance managers for regular inspections is also increasing. Summary of the invention

[0004] The present invention provides a basic message topic management method and a data bus system thereof to solve the problem of difficulty in existing message topic management.

[0005] Based on the above purpose, an embodiment of the present invention provides a message topic governance method, including: constructing a Kafka message mechanism including a data source, a message topic, a user, a business group, and authorization, and maintaining the metadata information of the message topic; setting user information and business group information, and centrally authorizing the business group to manage the operation permissions of the message topic; judging the permissions of the user based on the user information input by the user, and allowing the authorized user to perform message operations corresponding to the permissions; collecting the status information of the message topic, and pushing warnings according to preset warning rules.

[0006] Optionally, the method further includes: uniformly managing the message middleware that carries the message topics; and authorizing the user to use the message middleware based on a preset verification method.

[0007] Optionally, the setting of user information and business group information, through centralized authorization management of operation permissions of message subjects for business groups, includes: setting user information, dividing users into business groups based on business dimensions, users of different business dimensions have different data permissions, the user information includes: user name, password, and administrator; setting business group information, and centrally authorizing management of operation permissions of message subjects based on business groups.

[0008] Optionally, one user corresponds to multiple business groups.

[0009] Optionally, the method also includes: obtaining new user information and new message subject information; obtaining business group information associated with the new user information based on the new user information, and selecting a message subject based on the new message subject information; determining whether the business group corresponding to the business group information has the authority to select the message subject; if so, authorizing the new user to have the authority to select the message subject.

[0010] Optionally, the user's authority is judged based on the user information input by the user, and the authorized user is allowed to perform message operations corresponding to the authority, including: judging the user's authority based on the user name, password, and selected business group in the user information input by the user; if it is judged that the user has read authority over the message subject, the data of the message subject is automatically consumed and pushed to the user in real time in the form of a chat dialog box; if it is judged that the user has write authority over the message subject, the message input by the user to be sent to the message subject is received, and the sending result is automatically fed back to the user.

[0011] Optionally, the status information of the collected message topics is used to push warnings according to preset warning rules, including: collecting the status information of any message topic through a customized kafka_exporter, and setting warning rules for any message topic through an Alert Manager; determining whether the message topic satisfies the warning rules; if so, pushing the warning content to the operation and maintenance personnel of the message topic.

[0012] Based on the same inventive concept, an embodiment of the present invention also proposes a data bus system for message topic governance, including: a message topic information maintenance module, used to construct a Kafka message mechanism including a data source, a message topic, a user, a business group, and authorization, and maintain the metadata information of the message topic; a business group permission maintenance module, connected to the message topic information maintenance module, used to set user information and business group information, and to centrally authorize the business group to manage the operation permissions of the message topic; a white screen operation module, connected to the business group permission maintenance module, used to judge the permissions of the user according to the user information input by the user, and allow the authorized user to perform message operations corresponding to the permissions; an early warning push module, connected to the business group permission maintenance module, used to collect the status information of the message topic, and push early warnings according to preset early warning rules.

[0013] Based on the same inventive concept, an embodiment of the present invention further proposes an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the aforementioned method when executing the program.

[0014] Based on the same inventive concept, an embodiment of the present invention further proposes a computer storage medium, in which at least one executable instruction is stored, and the executable instruction enables a processor to execute the aforementioned method.

[0015] The beneficial effects of the present invention are as follows: as can be seen from the above description, an embodiment of the present invention provides a message topic governance method and a data bus system thereof, the method comprising: constructing a Kafka message mechanism including a data source, a message topic, a user, a business group, and authorization, and maintaining metadata information of the message topic; setting user information and business group information, and centrally authorizing the management of the operation permissions of the message topic for the business group; performing permission judgment on the user based on the user information input by the user, and allowing the authorized user to perform message operations corresponding to the permissions; collecting status information of the message topic, and pushing warnings based on preset warning rules, which can reduce the complexity of message management, is simple to operate, and is visualized, and is suitable for operation and maintenance management of message topics by Internet companies. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0017] Figure 1 A schematic diagram of a flow chart of a message topic management method in an embodiment of the present invention;

[0018] Figure 2 This is a schematic diagram of authorization logic in an embodiment of the present invention;

[0019] Figure 3 A schematic diagram of the structure of a data bus system for message topic management in an embodiment of the present invention;

[0020] Figure 4 Schematic diagram of an electronic device in an embodiment of the present invention. DETAILED DESCRIPTION

[0021] In order to make the objectives, technical solutions and advantages of the present disclosure more clearly understood, the present disclosure is further described in detail below in combination with specific embodiments and with reference to the accompanying drawings.

[0022] It should be noted that, unless otherwise defined, the technical terms or scientific terms used in the embodiments of the present invention should be understood by people with ordinary skills in the field to which the present disclosure belongs. The "first", "second" and similar words used in the embodiments of the present invention do not represent any order, quantity or importance, but are only used to distinguish different components. "Including" or "comprising" and similar words mean that the elements or objects appearing before the word cover the elements or objects listed after the word and their equivalents, without excluding other elements or objects. "Connect" or "connected" and similar words are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative positional relationships. When the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0023] The embodiment of the present invention provides a message topic management method. The message topic management method of the embodiment of the present invention is applied to a server. Figure 1 As shown, the message topic governance method includes:

[0024] Step S11: Build a Kafka message mechanism including data source, message topic, user, business group, and authorization, and maintain the metadata information of the message topic.

[0025] In an embodiment of the present invention, a Kafka message mechanism including data source, message topic, user, business group, and authorization is constructed to maintain metadata information of the message topic, such as the server cluster to which the message topic belongs, the business application service to which it belongs, the number of topic partitions, the number of replicas, etc.

[0026] The embodiment of the present invention also manages the message middleware that carries the message topic in a unified manner; and authorizes the user to use the message middleware based on a preset verification method. The verification method includes but is not limited to user name and password, token, and other verification methods. By controlling each middleware, all the message queues of the enterprise can be managed.

[0027] The embodiment of the present invention designs a unified message topic management system based on a unified domain model based on data source->message topic (topic)->user->business group->authorization, shields the differences in the underlying message queues, and uniformly abstracts the middle layer to connect to each message queue.

[0028] Step S12: Setting user information and business group information, and centrally authorizing and managing the operation rights of message topics for the business group.

[0029] In an embodiment of the present invention, user information is set, and users are divided into business groups based on business dimensions. Users of different business dimensions have different data permissions, and the user information includes: user name, password, and affiliated administrator. Business group (Group) information is set, and centralized authorization management of the operation permissions of message topics is performed on a business group basis. Business group (Group) information includes at least Group id, business group name, and application service to which the business group belongs. Users associated with the business group (Group) no longer need separate management permissions, and at the same time, there is a one-to-many relationship between users and business groups, that is, one user corresponds to multiple business groups, which can avoid multiple users consuming message topics through a Group group at the same time, resulting in resource competition. The embodiment of the present invention controls the users of each system and controls the business domain dimension of enterprise users. It can be split based on the organizational structure of the enterprise, and unified permission control is performed on the Group, which is more suitable for the needs of enterprise business scenarios and reduces complexity.

[0030] Step S13: judging the authority of the user according to the user information input by the user, and allowing the authorized user to perform message operations corresponding to the authority.

[0031] In the embodiment of the present invention, by centrally authorizing and managing Topic operation permissions for business groups, users associated with business groups no longer need to manage permissions separately. Traditional Kafka message queue permissions are generally divided into Group write / read and User write / read permissions. The embodiment of the present invention is based on the user->business group->topic model to uniformly authorize business groups and uniformly manage the entire domain model. Figure 2 As shown, it includes: first obtaining newly created user information and newly created message subject information; then obtaining business group information associated with the newly created user information according to the newly created user information, selecting a message subject according to the newly created message subject information; then judging whether the business group corresponding to the business group information has the authority to select the message subject; if so, authorizing the newly created user to have the authority to select the message subject. If not, reselecting the message subject information.

[0032] The existing common ways of operating message topics are all based on the Kafka command line or the Kafka Client API call. The embodiment of the present invention performs a white screen operation on the message topic, that is, the Kafka message topic can be directly operated on the data bus system governed by the message topic. The user does not need to understand the Kafka command line and API interface. He only needs to enter the user name, password, and select the user group Group to which he belongs. The system will automatically determine the current user authority. Optionally, the user's authority is determined according to the user name, password, and business group selected in the user information entered by the user; if it is determined that the user has the read authority of the message topic, the data of the message topic is automatically consumed and pushed to the user in real time in the form of a chat dialog box. By default, consumption starts automatically from the current start time, and the start time set by the user to view the message can also be received, and consumption starts from the start time set by the user. At the same time, the keywords set by the user can also be received, and the pushed information can be filtered according to the keywords set by the user. If it is determined that the user has the write authority of the message topic, the message entered by the user to be sent to the message topic is received, and the sending result is automatically fed back to the user. That is, the user can directly enter the message to be sent to the Topic in the dialog box, and the system will automatically feedback the sending result.

[0033] The embodiment of the present invention uniformly manages the model of production and consumption based on message topics by configuring different users for different system teams and then authorizing users with topic permissions, thereby solving the problem of confusion in the ownership of message topics and preventing unauthorized users of the topic from arbitrarily reading and writing the topic, which causes business and system abnormalities for consumers of the message topic. In addition, the white screen operation of the message topic is realized by combining and encapsulating the Kafka API, which facilitates the inspection and maintenance of the message topic by relevant operation and maintenance personnel.

[0034] Step S14: collect the status information of the message subject and push an early warning according to the preset early warning rules.

[0035] Message topics are usually split based on business logic. For example, in e-commerce, topics can be set based on products, orders, shopping carts, messages, etc. A module can correspond to multiple topics, and users can query the status information of the business group, message backlog, partition activity, etc. associated with the topic in real time.

[0036] Optionally, first collect the status information of any message topic through a custom kafka_exporter, and set the alert rules for any message topic through the Alert Manager. That is, collect the Metrics information of the Kafka message topic status through a custom kafka_exporter, and expose it to Prometheus for monitoring, and then set the alert rules, push method, and push message template through the Alert Manager. Set different alert rules for different consumer topics.

[0037] Then determine whether the message topic meets the warning rules; if so, push the warning content to the operation and maintenance personnel of the message topic. That is, when the consumption topic meets the warning rules, the warning content is actively pushed to the relevant operation and maintenance personnel of the message topic, such as: message topic administrator, Group administrator, and the administrator can trace the cause of the warning by querying the various partitions under the message topic and the different types of log files under the partitions and the corresponding copies of the partitions, including: log files, offset index files, and timestamp index files.

[0038] The embodiment of the present invention obtains Metrics information related to the Kafka message topic status by customizing kafka_exporter, and can set different warning rules for single or multiple Metrics information, monitor the message topic status in real time, and realize real-time warning of the message topic.

[0039] To summarize, the message topic governance method of the embodiment of the present invention constructs a Kafka message mechanism including a data source, a message topic, a user, a business group, and authorization, and maintains the metadata information of the message topic; sets user information and business group information, and centrally authorizes the business group to manage the operation permissions of the message topic; makes permission judgments on the user based on the user information input by the user, and allows the authorized user to perform message operations corresponding to the permissions; collects the status information of the message topic, and pushes warnings based on preset warning rules, which can reduce the complexity of message management, is simple to operate, and is visualized, and is suitable for the operation and maintenance management of message topics by Internet companies.

[0040] The above specific embodiments of the present invention are described. In some cases, the actions or steps recorded in the embodiments of the present invention can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the process depicted in the accompanying drawings does not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0041] Based on the same concept, the embodiment of the present invention also provides a data bus system for message topic management. Applicable to a server. Figure 3 As shown, the data bus system for message subject management includes: message subject information maintenance module, business group authority maintenance module, white screen operation module and warning push module.

[0042] The message topic information maintenance module is used to build a Kafka message mechanism including data source, message topic, user, business group, and authorization, and maintain the metadata information of the message topic;

[0043] A business group authority maintenance module, connected to the message subject information maintenance module, is used to set user information and business group information, and to centrally authorize and manage the operation authority of the message subject for the business group;

[0044] A white screen operation module, connected to the business group authority maintenance module, for determining the authority of the user according to the user information input by the user, and allowing the authorized user to perform message operations corresponding to the authority;

[0045] The early warning push module is connected to the business group authority maintenance module, and is used to collect status information of message topics and push early warnings according to preset early warning rules.

[0046] For the convenience of description, the above device is described as various modules according to their functions. Of course, when implementing the embodiment of the present invention, the functions of each module can be implemented in the same or multiple software and / or hardware.

[0047] The device of the above embodiment is applied to the corresponding method in the above embodiment, and has the beneficial effects of the corresponding method embodiment, which will not be described in detail here.

[0048] Based on the same inventive concept, an embodiment of the present invention further provides an electronic device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the method described in any one of the above embodiments is implemented.

[0049] An embodiment of the present invention provides a non-volatile computer storage medium, wherein the computer storage medium stores at least one executable instruction, and the computer executable instruction can execute the method described in any of the above embodiments.

[0050] Figure 4 A more specific schematic diagram of the hardware structure of an electronic device provided in this embodiment is shown, and the device may include: a processor 401, a memory 402, an input / output interface 403, a communication interface 404, and a bus 405. The processor 401, the memory 402, the input / output interface 403, and the communication interface 404 are connected to each other in communication within the device through the bus 405.

[0051] The processor 401 can be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solution provided by the method embodiment of the present invention.

[0052] The memory 402 can be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage device, dynamic storage device, etc. The memory 402 can store an operating system and other application programs. When the technical solution provided by the method embodiment of the present invention is implemented by software or firmware, the relevant program code is stored in the memory 402 and called and executed by the processor 401.

[0053] The input / output interface 403 is used to connect the input / output module to realize information input and output. The input / output module can be configured in the device as a component (not shown in the figure), or it can be externally connected to the device to provide corresponding functions. The input device may include a keyboard, a mouse, a touch screen, a microphone, various sensors, etc., and the output device may include a display, a speaker, a vibrator, an indicator light, etc.

[0054] The communication interface 404 is used to connect a communication module (not shown) to realize communication interaction between the device and other devices. The communication module can realize communication through a wired mode (such as USB, network cable, etc.) or a wireless mode (such as mobile network, WIFI, Bluetooth, etc.).

[0055] The bus 405 comprises a pathway for transmitting information between the various components of the device (eg, the processor 401 , the memory 402 , the input / output interface 403 , and the communication interface 404 ).

[0056] It should be noted that, although the above device only shows the processor 401, the memory 402, the input / output interface 403, the communication interface 404 and the bus 405, in the specific implementation process, the device may also include other components necessary for normal operation. In addition, it can be understood by those skilled in the art that the above device may also only include the components necessary for implementing the embodiment of the present invention, and does not necessarily include all the components shown in the figure.

[0057] Those skilled in the art should understand that the discussion of any of the above embodiments is merely illustrative and is not intended to imply that the scope of the present application is limited to these examples. Based on the concept of the present application, the technical features in the above embodiments or different embodiments may be combined, the steps may be implemented in any order, and there are many other variations of the different aspects of the present application as described above, which are not provided in detail for the sake of simplicity.

[0058] This application is intended to cover all such substitutions, modifications and variations that fall within the broad scope of the embodiments of the present invention. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the embodiments of the present invention should be included in the scope of protection of this application.

Claims

1. A message topic management method, characterized in that: The method comprises: Build a Kafka message mechanism including data source, message topic, user, business group, and authorization, and maintain metadata information of message topics; Set user information and business group information, and centrally authorize the business group to manage the operation rights of message topics; Determine the authority of the user according to the user information input by the user, and allow the authorized user to perform message operations corresponding to the authority; Collect status information of message topics and push warnings according to preset warning rules.

2. The method according to claim 1, characterized in that: The method further comprises: Centralized management of the message middleware that carries message topics; Authorize users to access the message middleware based on the preset authentication method.

3. The method according to claim 1, characterized in that: The setting of user information and business group information, through centralized authorization management of the operation authority of the message subject for the business group, includes: Set user information, divide users into business groups based on business dimensions, and users in different business dimensions have different data permissions. The user information includes: user name, password, and belonging administrator; Set business group information to centrally authorize and manage the operation permissions of message topics based on business groups.

4. The method according to claim 3, characterized in that: One user corresponds to multiple business groups.

5. The method according to claim 1, characterized in that: The method further comprises: Get new user information and new message subject information; Acquire the service group information associated with the newly created user information according to the newly created user information, and select the message subject according to the newly created message subject information; Determine whether the business group corresponding to the business group information has authority over the selected message topic; If yes, the newly created user is granted permission to select the message topic.

6. The method according to claim 1, characterized in that: The step of determining the authority of the user according to the user information input by the user, and allowing the authorized user to perform message operations corresponding to the authority, includes: Determine the user's authority based on the user name, password, and selected business group in the user information entered by the user; If it is determined that the user has the read permission for the message topic, the data of the message topic is automatically consumed and pushed to the user in real time in the form of a chat dialog box; If it is determined that the user has the write permission for the message subject, the message input by the user to be sent to the message subject is received, and the sending result is automatically fed back to the user.

7. The method according to claim 1, characterized in that: The state information of the collected message subject is pushed according to the preset warning rules, including: Collect the status information of any message topic through custom kafka_exporter, and set the warning rules of any message topic through Alert Manager; Determining whether the message subject satisfies the warning rule; If yes, the warning content is pushed to the operation and maintenance personnel of the message topic.

8. A data bus system for message topic management, characterized in that: The system comprises: The message topic information maintenance module is used to build a Kafka message mechanism including data source, message topic, user, business group, and authorization, and maintain the metadata information of the message topic; A business group authority maintenance module, connected to the message subject information maintenance module, is used to set user information and business group information, and to centrally authorize and manage the operation authority of the message subject for the business group; A white screen operation module, connected to the business group authority maintenance module, for determining the authority of the user according to the user information input by the user, and allowing the authorized user to perform message operations corresponding to the authority; The early warning push module is connected to the business group authority maintenance module, and is used to collect status information of message topics and push early warnings according to preset early warning rules.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the method according to any one of claims 1 to 7 is implemented.

10. A computer storage medium, characterized in that: The storage medium stores at least one executable instruction, and the executable instruction enables the processor to execute the method as described in any one of claims 1-7.

Images