Subscription relationship query method and device based on distributed message queue, equipment and medium

Abstract

The application relates to a subscription relationship query method and device based on a distributed message queue, equipment and a medium in the computer technology field, and the method comprises the following steps: in response to a subscription query request, a queue access address and a subscription relationship member set in subscription query information carried by the request are parsed; all consumption groups and state values thereof in a distributed message queue corresponding to the queue access address are queried; detail description information of each consumption group whose state value represents an active state is queried, and each consumption group and a topic group subscribed by the consumption group in the detail description information are extracted to form a subscription relationship pair; for each subscription relationship member in the subscription relationship member set, all subscription relationship pairs containing the subscription relationship member are returned to answer the subscription query request. The application can efficiently and accurately query the subscription relationship in the distributed message queue.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a subscription relationship query method based on a distributed message queue and the corresponding apparatus, computer equipment, and computer-readable storage medium. Background Technology

[0002] In distributed message queues like Apache Kafka, messages are organized into different topics. Producers publish messages to topics, and consumers subscribe to and consume these topics collaboratively through consumer groups. As business grows, queues typically contain a large number of topics and consumer groups, forming complex many-to-many subscription relationships. During daily operations, troubleshooting, or resource planning, it is often necessary to quickly determine which consumer groups are subscribed to a specific topic, or to understand the range of topics currently consumed by a particular consumer group.

[0003] To address this need, the current standard practice is to rely on manual queries using the command-line tools built into the distributed message queue. The typical workflow is as follows: Operations personnel first obtain the names of all consumer groups in the cluster; then, for each consumer group in the list, they query its details to obtain a detailed description. This detailed description is usually output in text format, including the topic name subscribed to by the consumer group, the consumption progress of each partition, and the member status. Next, operations personnel need to manually review this output, reading it line by line, to identify the topic field and filter out consumer groups subscribed to a specific topic. When it's necessary to filter out inactive consumer groups, it's also necessary to additionally check the consumer group status field or member information to determine if it is currently active.

[0004] This traditional method relies entirely on manual operation and visual analysis by maintenance personnel. Its technical shortcomings are very obvious: the query efficiency is very low, the operation is cumbersome and prone to errors, that is, it is easy to produce omissions or false alarms, making it difficult to meet the actual needs of fast and accurate query of subscription relationships in a large-scale distributed message queue environment. Summary of the Invention

[0005] The primary objective of this application is to solve at least one of the above-mentioned problems by providing a subscription relationship query method based on a distributed message queue, and the corresponding apparatus, computer device, and computer-readable storage medium thereof.

[0006] To achieve the various objectives of this application, the following technical solution is adopted: A subscription relationship query method based on a distributed message queue, provided to suit one of the purposes of this application, includes the following steps: In response to a subscription query request, parse out the queue access address and subscription relationship member set from the subscription query information carried in the request; Query all consumer groups and their status values ​​in the distributed message queue corresponding to the queue access address; Query the detailed description information of each consumer group whose status value represents its active status, and extract the corresponding consumer group and its subscribed topics from each detailed description information to form a subscription relationship pair; For each subscription member in the corresponding subscription member set, return all subscription relationship pairs containing that member to respond to the subscription query request.

[0007] On the other hand, a subscription relationship query device based on a distributed message queue, provided to meet one of the purposes of this application, includes a request response module, a data query module, a subscription mapping module, and a request answering module. The request response module is used to respond to a subscription query request and parse out the queue access address and subscription relationship member set from the subscription query information carried in the request. The data query module is used to query all consumer groups and their status values ​​in the distributed message queue corresponding to the queue access address. The subscription mapping module is used to query the detailed description information of each consumer group whose status value represents its active state, and extract the corresponding consumer group and its subscribed topics from each detailed description information to form subscription relationship pairs. The request answering module is used to return all subscription relationship pairs containing the subscription relationship member for each subscription relationship member in the subscription relationship member set, in response to the subscription query request.

[0008] On another front, a computer device provided for one of the purposes of this application includes a central processing unit and a memory, wherein the central processing unit is used to invoke and run a computer program stored in the memory to perform the steps of the subscription relationship query method based on a distributed message queue of this application.

[0009] In another aspect, a computer-readable storage medium is provided to suit another purpose of this application, which stores, in the form of computer-readable instructions, a computer program implementing a distributed message queue-based subscription relationship query method, which, when invoked by a computer, executes the steps included in the method. Attached Figure Description

[0010] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein: Figure 1 This is a flowchart illustrating a typical embodiment of the subscription relationship query method based on a distributed message queue according to this application; Figure 2 This is a schematic diagram of the subscription relationship query device based on a distributed message queue according to this application; Figure 3 This is a schematic diagram of the structure of a computer device used in this application. Detailed Implementation

[0011] The embodiments of this application are described in detail below. Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain this application, and should not be construed as limiting this application.

[0012] Those skilled in the art will understand that, unless explicitly stated otherwise, the singular forms “a,” “an,” “the,” and “the” used herein may also include the plural forms. It should be further understood that the term “comprising” as used in the specification of this application means the presence of features, integers, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. It should be understood that when we say an element is “connected” or “coupled” to another element, it can be directly connected or coupled to the other element, or there may be intermediate elements. Furthermore, “connected” or “coupled” as used herein can include wireless connections or wireless coupling. The term “and / or” as used herein includes all or any units and all combinations of one or more associated listed items.

[0013] Those skilled in the art will understand that, unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains. It should also be understood that terms such as those defined in general dictionaries should be understood to have the same meaning as in the context of the prior art, and should not be interpreted in an idealized or overly formal sense unless specifically defined as herein.

[0014] Those skilled in the art will understand that the terms "client," "terminal," and "terminal device" as used herein include both devices that receive wireless signals, devices that only possess wireless signal receiver capabilities without transmission capabilities, and devices with receiving and transmitting hardware, devices that have receiving and transmitting hardware capable of bidirectional communication over a bidirectional communication link. Such devices may include: cellular or other communication devices such as personal computers or tablets, having single-line displays, multi-line displays, or cellular or other communication devices without multi-line displays; PCS (Personal Communications Service) that can combine voice, data processing, fax, and / or data communication capabilities; PDAs (Personal Digital Assistants) that may include radio frequency receivers, pagers, internet / intranet access, web browsers, notebooks, calendars, and / or GPS (Global Positioning System) receivers; and conventional laptops and / or handheld computers or other devices that have and / or include radio frequency receivers. As used herein, "client," "terminal," and "terminal device" can be portable, transportable, installed in a means of transportation (air, sea, and / or land), or suitable and / or configured to operate locally and / or in a distributed manner, operating in any other location on Earth and / or in space. "Client," "terminal," and "terminal device" as used herein can also be a communication terminal, an internet access terminal, or a music / video playback terminal, such as a PDA, a MID (Mobile Internet Device), and / or a mobile phone with music / video playback capabilities, or a smart TV, set-top box, etc.

[0015] The hardware referred to by the names "server," "client," and "service node" in this application is essentially an electronic device with the equivalent capabilities of a personal computer. It is a hardware device with the necessary components revealed by the von Neumann architecture, such as a central processing unit (including an arithmetic logic unit and a control unit), memory, input devices, and output devices. The computer program is stored in its memory, and the central processing unit loads the program stored in the secondary storage into the main memory to run it, execute the instructions in the program, and interact with the input and output devices to complete specific functions.

[0016] It should be noted that the concept of "server" used in this application can also be extended to apply to server clusters. Based on network deployment principles as understood by those skilled in the art, each server should be a logical division. Physically, these servers can be independent of each other but accessible through interfaces, or they can be integrated into a single physical computer or a computer cluster. Those skilled in the art should understand this flexibility and should not use it to constrain the implementation of the network deployment method described in this application.

[0017] One or more of the technical features of this application, unless explicitly specified herein, can be deployed on a server and accessed by a client remotely calling the online service interface provided by the server, or can be directly deployed and run on a client for access.

[0018] Unless otherwise specified, all data involved in this application may be stored remotely on a server or on a local terminal device, as long as it is suitable for use by the technical solution of this application.

[0019] Those skilled in the art will understand that although the various methods in this application are described based on the same concept and thus present commonality among them, they can be performed independently unless otherwise specified. Similarly, the various embodiments disclosed in this application are all based on the same inventive concept; therefore, concepts expressed in the same way, as well as concepts that are appropriately changed for convenience but are expressed differently, should be understood equivalently.

[0020] Unless otherwise expressly stated, the various embodiments disclosed in this application can be combined in a cross-cutting manner to flexibly construct new embodiments, as long as such combination does not depart from the inventive spirit of this application and can meet the needs of the prior art or solve a certain deficiency in the prior art. Those skilled in the art should be aware of such modifications.

[0021] The subscription relationship query method based on distributed message queue of this application can be programmed into a computer program product and deployed on a client or server. For example, in the exemplary application scenario of this application, it can be deployed on servers in the fields of content delivery network (CDN), video platform, live streaming platform, e-commerce platform, etc., so that the method can be executed by human-computer interaction with the process of the computer program product through a graphical user interface by accessing the interface opened after the computer program product is running.

[0022] Please see Figure 1 The subscription relationship query method based on distributed message queues in this application includes the following steps in some embodiments: Step S1100: Respond to the subscription query request and parse out the queue access address and subscription relationship member set from the subscription query information carried in the request.

[0023] Distributed message queues are middleware systems that handle high-throughput message delivery. A consumer group is a logical grouping within a distributed message queue used for collaborative message consumption; multiple consumers within the same group share the responsibility of consuming messages from the topics subscribed to by that group.

[0024] The queue access address includes the IP address and port number required to access the corresponding distributed message queue. For ease of understanding, as an example, in a specific implementation of a distributed message queue like Kafka, this address is the Bootstrap Server address, typically in the format of "IP address:port number", such as "11.70.16.98:9092". This address is the entry point into the entire Kafka cluster; all subsequent operations in the query process, such as retrieving cluster metadata and consumer group lists, must be completed through a connection established with this address.

[0025] A subscription relationship member set includes at least one subscription relationship member. The subscription relationship member set by the querier can be any consumer group or topic within the distributed message queue that it knows. A subscription relationship refers to the mapping relationship between a consumer group and the topic it consumes in the distributed message queue. The querier can be a user or an automated query script.

[0026] In one embodiment, the user, acting as the query party, visually edits the queue access address corresponding to the distributed message queue to be queried, as well as the subscription relationship member set consisting of at least one consumer group or topic in the queue known to the user as a subscription relationship member, on the front-end query interface. After editing, the user's client constructs subscription query information based on the edited queue access address and subscription relationship member set, encapsulates the subscription query request, and submits it to the server. The design of this interface can be flexibly modified by those skilled in the art.

[0027] After receiving a subscription query request, the server responds immediately. Typically, it first parses the subscription query information, which is a form parameter in the request body, identifies the fields pointing to the IP address and port number, extracts these fields and assigns them to the queue access address, and identifies the fields pointing to the subscription relationship member set, extracts these fields and assigns them to the subscription relationship member set.

[0028] Step S1200: Query all consumer groups and their status values ​​in the distributed message queue corresponding to the queue access address.

[0029] Status values ​​are used to characterize the current running status of the corresponding consumer group. Taking Kafka as an example, the status values ​​of a consumer group can include "Stable" (active state) and several inactive states: "Empty" (no active members in the consumer group), "PreparingRebalance" (the consumer group is undergoing rebalancing, reallocating partitions), "Dead" (the consumer group has no members and its metadata may have been cleaned up), "Preparing Rebalance" (the consumer group is preparing for rebalancing, and consumers that are about to join or leave have not yet completed the joining or leaving operation), and "Dead and Empty" (the consumer group has expired or been deleted, and there are no consumers in the consumer group).

[0030] To query all consumer groups in the distributed message queue corresponding to the queue access address, in one embodiment, the server instantiates an administrator client object based on the official client library provided by the distributed message queue, passes the queue access address as a connection configuration parameter, calls the interface for listing consumer groups, and directly extracts all consumer groups from the returned metadata object set to construct a consumer group list.

[0031] To query the status value of each consumer group in the aforementioned distributed message queue, one embodiment involves invoking a status query script to sequentially traverse the queue and retrieve the status value of each consumer group. The script uses a loop to read each consumer group in the list, constructs a single status description query instruction for the current consumer group, and executes it. In a Kafka environment, this instruction is a consumer group management script with description parameters. The script parses the status field in the returned result to obtain the status value, then continues processing the next consumer group until the traversal is complete. In another embodiment, the status query script utilizes a thread pool or asynchronous coroutine mechanism for concurrent batch queries. The script divides the consumer group list into multiple batches, assigns an independent execution thread to each consumer group, and simultaneously sends multiple status query requests to the distributed message queue server using the administrator client interface. By aggregating the response results of all concurrent requests, it extracts and maps the status value corresponding to each consumer group, thereby efficiently obtaining the real-time running status of the global consumer groups.

[0032] Step S1300: Query the detailed description information of each consumer group whose status value represents the active status, and extract the corresponding consumer group and its subscribed topics from each detailed description information to form a subscription relationship pair.

[0033] The server iterates through the state values ​​of all consumer groups, extracts the consumer groups whose state values ​​match the active state identifier (e.g., state: stable when the distributed message queue is Kafka), and obtains the corresponding active consumer groups. This filters out inactive consumer groups and reduces the overhead of invalid queries.

[0034] For each active consumer group, its detailed description information is queried. This detailed description information is full metadata and status data about the consumer group's runtime maintained by the distributed message queue server. Specifically, it includes the consumer group, coordinator node identifier, identifiers of each consumer member within the group, the topic partitions assigned to each member, the current consumption offset of each partition, and the lag offset. In one embodiment, the server constructs and executes a detailed query command for the consumer group. Taking Kafka as an example, executing the command "kafka-consumer-groups.sh --bootstrap-server <queue access address> --describe --group <consumer group name>" will return multiple lines of text data, each corresponding to a topic partition. The "TOPIC" column records the topic name to which the partition belongs. For the same consumer group, all the deduplicated topic names appearing in the "TOPIC" column in all lines of the command's output represent all the topics subscribed to by that consumer group. In practice, this is achieved by reading the standard output stream, traversing line by line, extracting the value of the "TOPIC" column, and performing deduplication to obtain all the topics subscribed to by the consumer group.

[0035] In another embodiment, using the client library provided by the official distributed message queue, such as Kafka's JavaAdminClient, a batch of ConsumerGroupDescription objects can be directly obtained by calling the describeConsumerGroups method and passing in a collection containing the names of active consumer groups. Each ConsumerGroupDescription object contains a list of members for that consumer group, and the data type of this list is Collection. <memberdescription>In other words, it's a collection of MemberDescription objects. Each MemberDescription object corresponds to a specific consumer within the consumer group. Furthermore, each MemberDescription object encapsulates an assignment property, which is a collection of TopicPartitions. By iterating through these TopicPartition objects and retrieving their topic fields, all topics subscribed to by that consumer group can be extracted. This approach eliminates the need for text parsing, directly retrieving structured data based on a strongly typed object model, resulting in more robust code.

[0036] After obtaining all the topics subscribed to by each active consumer group, an extraction operation is performed to construct subscription pairs. A subscription pair is a tuple consisting of a consumer group and a topic it subscribes to, representing the atomic subscription fact that "a consumer group subscribes to a certain topic." Since a consumer group may subscribe to multiple topics simultaneously, it needs to be combined with each of its subscribed topics individually. For example, if the consumer group "order-service-group" subscribes to the topics "order-created" and "order-paid," two subscription pairs will be extracted: ("order-service-group", "order-created") and ("order-service-group", "order-paid"). This is the set of subscription pairs formed by all active consumer groups after this processing. In a concrete implementation, the extraction process can use a nested loop: the outer loop iterates through each active consumer group, and the inner loop iterates through the list of topics subscribed to by that consumer group, thus generating a subscription pair for each topic iterates through.

[0037] Step S1400: For each subscription relationship member in the corresponding subscription relationship member set, return all subscription relationship pairs containing the subscription relationship member to respond to the subscription query request.

[0038] After obtaining all subscription pairs in this step, a further nested loop approach can be used: the outer loop iterates through each subscription member in the subscription member set, while the inner loop iterates through all the resulting subscription pairs to see if the given subscription member is present. This process, for each subscription member traversed, yields all subscription pairs containing that member, forming the response result. In the specific response result, subscription pairs containing the same subscription member can be tightly grouped together, and the font style of the subscription member in each pair can be highlighted; alternatively, the subscription member can be used as the key, with the corresponding value including another element in each subscription pair containing that subscription member.

[0039] As can be seen from the above embodiment of this application, the technical solution of this application has many advantages, including but not limited to the following aspects: First, this application achieves a complete query loop from target queue location to specific member filtering by responding to subscription query requests and accurately parsing the queue access address and subscription relationship member set carried in them.

[0040] Secondly, after obtaining all consumer groups and their status values ​​in the distributed message queue, detailed description information is queried and extracted specifically for consumer groups whose status values ​​represent active status. This mechanism cleverly filters out redundant consumer group data in inactive status, avoids reading and processing invalid data, not only greatly saves system memory and network bandwidth resources, but also significantly accelerates the aggregation speed of subscription relationship data, thereby effectively improving the execution efficiency and response time of the overall query method.

[0041] Furthermore, by structuring active consumer groups and their subscribed topics into subscription relationship pairs, and strictly corresponding to the subscription relationship member set to return all subscription relationship pairs containing a specific member, the complete subscription relationship required for the query is output in a one-time, full, and accurate manner. This eliminates the need for operations and maintenance personnel or developers to manually compare and piece together data from multiple query results. They can intuitively and comprehensively grasp the complete subscription chain of a specific member in the global message queue based on the response result of a single request. This greatly facilitates message routing troubleshooting and subscription relationship sorting, and comprehensively improves the operation and maintenance management efficiency and fault location capabilities of distributed systems.

[0042] In a further embodiment, step S1200, querying all consumer groups and their status values ​​in the distributed message queue corresponding to the queue access address, includes the following steps: Step S1210: Construct and execute a consumer group query instruction that operates on the corresponding distributed message queue based on the queue access address to obtain a consumer group list containing all consumer groups in the queue.

[0043] In the process of querying all consumer groups and their status values ​​in a distributed message queue, the server first needs to construct and execute a consumer group query command based on the resolved queue access address to obtain a list of consumer groups containing all consumer groups in the queue. The server uses the queue access address as input, embedding it into a pre-defined consumer group query command to generate a complete consumer group query command. When the distributed message queue is specifically implemented as Apache Kafka, the constructed command is "... / kafka-consumer-groups.sh --bootstrap-server 11.70.16.98:9092--list", where "11.70.16.98:9092" is the queue access address. The server executes this command through a child process call and captures its standard output stream. Each line of text in the standard output stream is the name identifier of a consumer group. Collecting all lines forms a consumer group list containing all current consumer groups in the Kafka cluster.

[0044] Step S1220: Construct and execute a status query script that applies to all consumer groups in the consumer group list to obtain the status value of each consumer group.

[0045] After obtaining the consumer group list, the server needs to construct and execute a status query script to iterate through the consumer groups in the list and query their status values. In one embodiment, a serial traversal logic is used: the script uses a loop structure to sequentially read each consumer group name in the consumer group list. For the currently traversed consumer group, a status description query instruction for that specific consumer group is dynamically constructed. Taking Kafka as an example, this instruction is also constructed based on the consumer group management script, in the form of "kafka-consumer-groups.sh --bootstrap-server 11.70.16.98:9092--describe --group current consumer group name". After executing this instruction, the returned text description information includes a "STATE" field, the value of which represents the status value of the consumer group. For example, "Stable" indicates an active state, and others indicate an inactive state.

[0046] In this embodiment, by directly constructing and executing native consumer group query instructions and status query scripts based on queue access addresses, the specific technical problem of difficulty in reliably obtaining the full range of consumer groups and their fine-grained operational status in distributed message queue operations and maintenance due to the lack of a unified cross-language status query interface or third-party client dependency conflicts is effectively solved. This mechanism bypasses complex underlying application programming interface encapsulation, utilizes official native management scripts to ensure the absolute accuracy and cross-environment compatibility of status data acquisition, and ensures the orderliness of the status query process through serial traversal and dynamic instruction construction, laying a solid and highly reliable data foundation for subsequent accurate filtering of active consumer groups and reduction of invalid query overhead.

[0047] In a further embodiment, step S1400, for each subscription relationship member in the corresponding subscription relationship member set, returns all subscription relationship pairs containing the subscription relationship member, including the following steps: Step S1410: For each subscription member in the subscription relationship member set, construct the corresponding query regular expression for the subscription relationship member.

[0048] The server first passes the subscription relationship members as the input string (String) to the `re.escape(String)` method. This method escapes all special characters in the input string (such as ., *, ?, etc.), treating them as ordinary characters. The resulting processed string is then embedded into a predefined query regular expression template. The template is: `r"\bembedded string\b"`, where `\b` serves as a word boundary in the regular expression, effectively separating word characters from non-word characters and preventing substring matching. Thus, precise matching is achieved through this query regular expression.

[0049] Step S1420: Apply a query regular expression to query all extracted subscription relationship pairs to obtain all subscription relationship pairs containing subscription relationship members.

[0050] To clearly illustrate the precise matching mechanism of query regular expressions, let's take a subscription relationship member as the string "order" (which has already been escaped using the `re.escape()` function) as an example. The constructed query regular expression is "r'\border\b'". In regular expression syntax, word characters include letters, numbers, and underscores, while non-word characters include hyphens, spaces, and punctuation marks. When checking the subscription relationship pair "('user_group', 'order_service')", the underscore immediately following "order" in the text to be matched "order_service" is a word character and does not meet the word boundary condition. Therefore, the query regular expression fails to match, and the subscription relationship pair is excluded, effectively avoiding false positives for "order_service". When checking the subscription relationship pair "('payment_group', 'order')", the text to be matched "order" starts at the beginning of the string and ends at the end of the string, both satisfying the word boundary condition. The query regular expression matches successfully, and the subscription relationship pair is successfully retrieved. This word boundary-based matching mechanism can strictly distinguish consumer groups or topics with similar names but different actual characteristics, ensuring the absolute accuracy of query results.

[0051] This document describes specific implementations for querying using regular expressions, supporting multiple execution paths to adapt to data processing needs of varying scales. One embodiment employs a memory-based list traversal matching method. The server loads all extracted subscription pairs into a list data structure in memory, and sequentially reads each subscription pair in the list using a loop. For the currently read subscription pair, a regular expression matching function is called, applying the query regular expression to the consumer group and topic within that subscription pair. If either matches successfully, the subscription pair is appended to the result list in memory. This embodiment is suitable for scenarios with a small number of subscription pairs and sufficient memory resources, featuring simple execution logic and fast response speed.

[0052] In another embodiment, a line-by-line matching method based on file streaming is employed to handle the massive number of subscription pairs generated by a large-scale distributed message queue cluster. In the preceding steps, the server has written all subscription pairs line by line to a temporary log file, with each line recording a complete subscription pair. During the query phase, the temporary log file is opened in read-only mode, and the text content is read line by line using a file pointer. For each line read, the corresponding consumer group and topic are immediately parsed in memory, and a query regular expression is applied for matching and validation. If a match is successful, the line of text is directly written to another intermediate file used to store the query results, and then the memory space occupied by the current line is immediately released to continue reading the next line. This embodiment, through its streaming read and immediate memory release mechanism, ensures that memory usage remains at a consistently low level when processing millions of subscription pairs, avoiding memory overflow issues.

[0053] In another embodiment, a regular expression query method based on distributed caching or a database is employed. If all subscription pairs extracted in the preceding steps have been persistently stored in a relational database or a NoSQL database that supports regular expression matching, the server converts the query regular expression into a regular expression query statement supported by the corresponding database. In the MySQL database, an SQL query statement containing the "REGEXP" keyword is constructed, and the query regular expression is directly sent to the database engine for execution as a matching condition. This embodiment shifts the computational burden to the storage layer, significantly reducing the amount of network data transmission between the server and the database, and significantly improving the overall query throughput.

[0054] After retrieving all subscription pairs containing members with subscription relationships, the result set is structured and encapsulated. Iterating through all the final filtered subscription pairs, and following a pre-defined JSON data format template, the consumer group name and topic name in each subscription pair are mapped to their corresponding key-value pairs, along with metadata information such as the query status code, execution time, and total number of results. If the query result is empty, a standard JSON response is generated, clearly indicating that no matching subscription pair was found. This provides a unified, standardized, and easily parsed data interface for the front-end display platform or automated operation and maintenance scripts.

[0055] In this embodiment, by introducing a query regular expression mechanism that combines special character escaping with word boundaries, and supplemented by multi-path adaptive matching and structured encapsulation, the specific technical problems of traditional string inclusion matching easily leading to mismatched substrings and resulting in a large amount of noisy data in the query results, and the single data processing method easily causing memory overflow or performance bottlenecks when querying ultra-large-scale subscription relationships, are effectively solved. This mechanism completely eliminates the risk of false positives caused by name similarity, achieves absolute and accurate positioning of subscription relationship members, and ensures the purity and accuracy of query results. At the same time, by flexibly supporting multiple execution paths such as in-memory list traversal, file streaming processing, and database regular expression sinking, it enables dynamic optimization of computing and storage resource allocation according to the data scale, maximizing query throughput while ensuring memory safety for massive data queries. Finally, the standardized data interface output comprehensively improves the data parsing efficiency and fault diagnosis reliability of the automated operation and maintenance platform.

[0056] In a further embodiment, before step S1100, responding to the subscription query request, the following steps are included: Step S1000: Respond to the query parsing event, obtain the subscription query statement expressed in natural language corresponding to the event, and obtain the preset parsing prompt template. The parsing prompt template includes the preset queue access address corresponding to at least one distributed message queue and its matching subscription relationship list. The subscription relationship list includes all subscription relationship pairs composed of each consumer group whose active status is represented by a status value and the topic they subscribe to in the distributed message queue corresponding to the queue access address, which are pre-cached.

[0057] In the daily operation and maintenance and development troubleshooting scenarios of distributed message queues, operation and maintenance personnel or developers of newly integrated services sometimes do not accurately remember or obtain the specific queue access address and subscription relationship members of the distributed message queue, but they can determine whether it is what they want to query by seeing these specific subscription relationship members and queue access address. Furthermore, they can usually use conversational natural language to express their query intent, such as entering "check the consumer groups related to the order service in the store queue" or "see which groups have subscribed to the payment topic in the store queue". Traditional query methods force users to enter precise IP addresses, port numbers, and exactly matching consumer group or topic names. This rigid interaction method leads to a high usage threshold and causes query failures due to incorrect parameter input. To solve the interaction bottleneck in this specific technical scenario, this application introduces a natural language parsing mechanism. The server first responds to the query parsing event, which is generated by the user submitting natural language text in the front-end interactive interface or inputting voice commands through the speech recognition module. After responding to this event, the server obtains the subscription query statement submitted by the event or expressed in natural language by the voice input. This statement is an unstructured text expression of the user's query intent.

[0058] It is understood that the parsing prompt template can be prepared and stored in advance for use in this step, and the preset queue access addresses and their matching subscription relationship lists for each distributed message queue in the template can be updated and maintained. Furthermore, the template also includes a parsing task description text for the subscription query intent, used to constrain / guide the large language model to match the queue access address of the required distributed message queue from the provided queue access addresses and their matching subscription relationship lists of all distributed message queues, and the subscription relationship member set composed of all the subscription relationship members to be queried. Those skilled in the art can further configure this parsing task description text as needed based on the disclosure herein. An exemplary parsing task description text is as follows: "Based on the following distributed message queue information: 'The preset queue access addresses and their matching subscription relationship lists of at least one distributed message queue to be embedded', match the IP port of the corresponding distributed message queue for the user's query: 'The [original text to be embedded]', and the specific component names of the subscription relationship members to be queried in that queue, outputting in JSON format: {Queue access address: "extracted single IP port"; Subscription relationship member set: "extracted individual subscription relationship members"}.

[0059] Step S1010: Construct parsing prompt text based on subscription query statement and parsing prompt template, and use it to guide the large language model to determine the queue access address and subscription relationship member set that match the subscription query statement, which is then used to construct the subscription query request.

[0060] Large language models are suitable for text processing in the field of NLP. They are pre-trained to convergence using an extremely large corpus, acquiring the ability to generate human language and possessing a certain degree of accurate text semantic understanding and logical reasoning capabilities. Model selection includes OpenAI, Falcon, Chinchilla, PaLM, LLaMA 2, text-embedding-ada-002, etc., which can be selected and implemented by those skilled in the art as needed.

[0061] The large language model, based on its ability to understand natural language, parses the intent objects in the user query (such as "store queue" and "order service") and performs semantic similarity matching and logical reasoning with the candidate queue addresses and subscription relationship list provided in the parsed prompt text. For example, the model can associate "store" with "queue A" in the queue address description (if there is a corresponding alias mapping in the preset information, or through semantic inference from the topic name in the subscription relationship), and associate "order service" with consumer groups or topics such as "order-service-group" in the subscription relationship list, ultimately determining a single queue access address (such as "11.70.16.98:9092") that matches the query, as well as one or more precise subscription relationship members (such as "order-service-group"). The model generates a JSON string as a response according to the output format constraints.

[0062] After receiving the JSON response from the large language model, the server parses it and extracts the values ​​of the "queue access address" and "subscription relationship member set" fields. After format validation, the queue access address and subscription relationship member set are used as parameters to encapsulate and construct the subscription query request to be responded to in the subsequent step S1100. This achieves automatic conversion from natural language interaction to precise query commands, significantly reducing the usage threshold for operation and maintenance personnel.

[0063] In this embodiment, by introducing a large language model and a preset parsing prompt template, the technical problems of high usage threshold and easy parameter input errors caused by the requirement of users to manually enter precise network addresses and strict component names in traditional query methods are effectively solved. It realizes the automated and intelligent mapping from fuzzy natural language to the relevant specific parameters of the distributed message queue, significantly reduces the complexity of operation, improves the user-friendly interactive experience, greatly shortens the parameter preparation time for a single query, and greatly improves the ease of use, intelligence level and overall fault diagnosis efficiency of distributed message queue operation and maintenance management.

[0064] In a further embodiment, after step S1200, which involves querying all consumer groups and their status values ​​in the distributed message queue corresponding to the queue access address, the following steps are included: Step S2200: For each consumer group whose state value represents an inactive state, obtain the update timestamp corresponding to the most recent consumption update partition displacement information of each consumer in the consumer group, and the update interval time from the current timestamp.

[0065] In actual operation and maintenance scenarios of distributed message queues, there are many business tasks with periodic or specific time window execution characteristics, such as offline T+1 reporting tasks in data warehouses. These tasks are usually in a downtime state during weekends or public holidays. Alternatively, there are data analysis tasks that require accumulating a certain amount of messages to be consumed before being processed by the data warehouse. During the waiting period for these tasks, the corresponding consumer groups are usually in a downtime state. When the downtime of such consumer groups exceeds the maximum downtime set by the distributed message queue itself, such as 24 hours, the default operation and maintenance mechanism at the queue's underlying level will automatically mark the consumer group as inactive and forcibly clean up its consumption offset. This can lead to serious production incidents such as duplicate data consumption or data loss when business tasks restart after holidays, as the consumer group cannot continue consuming from its previous position. Clearly, such accidental deletions are caused by the maximum downtime set in the queue. While it's possible to customize the configuration to allow a longer downtime for the consumer group performing the aforementioned task, this requires specific configuration and clearing the configuration when the consumer group is no longer needed. Alternatively, the maximum downtime set in the queue can be increased, but this could lead to prolonged periods without cleanup. Therefore, this application explores alternative solutions that do not change the original maximum downtime of the queue or require further customization, as follows: To address the business interruption issues caused by the default queue cleanup mechanism in the aforementioned specific technical scenarios, a precise monitoring mechanism for the update interval of inactive consumer groups is introduced. For each consumer group whose status value represents an inactive state, the update timestamp corresponding to the most recent consumption update of partition displacement information for each consumer in that group is obtained, along with the update interval from the current timestamp. An inactive state refers to a consumer group's operational state where there are currently no active members or it is undergoing rebalancing, preventing normal message consumption. A consumer is a client instance within the consumer group that actually executes message retrieval and processing logic. Partition displacement information refers to the message position identifier that a consumer has consumed in a specific partition of a specific topic, used to record consumption progress. The update timestamp is the absolute time recorded when the consumer submits the displacement to the coordinator node of the distributed message queue. The update interval is the time difference between the current system time and the aforementioned update timestamp.

[0066] The process of obtaining the update interval duration includes several specific implementation methods. In one implementation, the server directly reads the timestamp field attached to the most recent offset committed by each consumer by calling the administrator client interface provided by the distributed message queue, and then calculates the update interval duration for each consumer by subtracting the timestamp from the current system time.

[0067] In another embodiment, the server directly parses the system topic used to store displacement information inside the distributed message queue. The server subscribes to the system topic as a regular consumer, reads the displacement commit message of the corresponding target consumer group, extracts the timestamp field recorded in the message header or message body, compares it with the current time, and calculates the accurate update interval.

[0068] Step S2210: If the sum of all update interval durations exceeds the preset cumulative duration, then the consumer group is confirmed as a long-term inactive consumer group.

[0069] The sum of all update interval durations refers to the total time value obtained by adding up the update interval durations calculated by each consumer within the same consumer group. The preset cumulative duration is a time threshold set based on business tolerance and the queue cleanup mechanism. A long-term inactive consumer group refers to a consumer group whose total inactivity time for all consumers has exceeded the system's allowed safety threshold, and which faces the risk of having its metadata and displacement data forcibly cleaned up by the underlying queue mechanism.

[0070] When determining whether the sum exceeds the preset cumulative duration, the server iterates through the update interval duration of all consumers in the consumer group, adds up all durations using a numerical accumulator to obtain the sum, and compares the sum with the preset cumulative duration. If the sum is greater than the preset cumulative duration, the consumer group is determined to be a long-term inactive consumer group.

[0071] Step S2220: Determine whether each long-term inactive consumer group exists in the preset long-term mounting whitelist. If it exists, enable external storage of the partition offset information of the most recent consumption update of each consumer in the corresponding consumer group, so that the corresponding partition can be used when the consumer group that was deleted and rebuilt continues to process consumption.

[0072] The default long-term whitelist is a pre-configured and persistent list that records the names of specific consumer groups whose names are required to retain offset information for extended periods of downtime. External storage refers to persistent storage media independent of the distributed message queue's internal storage system, used for secure backup of critical data.

[0073] The deleted and rebuilt consumer group refers to a new consumer group instance with the same name that was automatically cleaned up by the underlying queue mechanism after a long period of inactivity and then recreated when the business restarts. The process of determining the whitelist and enabling external storage includes several specific implementations. In one implementation, whitelist verification and storage are based on a relational database. The server stores a preset long-term whitelist in a specific configuration table of the relational database and determines whether the name of the currently long-term inactive consumer group exists in this table by executing a query. If it exists, the server serializes the partition offset information of each consumer in the corresponding consumer group and writes it to an offset backup table in the same database, using the consumer group name and partition identifier as a composite primary key. When the consumer group is deleted and rebuilt, the server reads the offset backup table, calls the administrator interface to resubmit the backed-up offset information to the queue, and restores the consumption progress.

[0074] In another embodiment, whitelist verification and storage are performed using a memory key-value storage system. The server uses the set data structure of the memory key-value storage system to store a preset long-term whitelist, and quickly determines whether a long-term inactive consumer group is in the whitelist by executing a member judgment command. If the verification passes, the server uses a hash structure to store partition offset information, with the consumer group name as the primary key, the partition identifier as the field, and the specific offset value as the value, and writes it. When a consumer group is rebuilt, the server retrieves all partition offset information of the consumer group at once and restores it to the queue.

[0075] In another embodiment, this is implemented using a combination of a distributed configuration center and a document database. A pre-configured, dynamically configured whitelist is permanently mounted in the distributed configuration center, and the server monitors configuration changes in real time to obtain the latest whitelist. Upon successful verification, the server writes the partition offset information into the document database in document form. Utilizing the database's indexing capabilities, the corresponding offset data can be quickly located and restored during consumer group reconstruction through conditional queries.

[0076] In this embodiment, by accurately monitoring the update interval of inactive consumer groups and combining it with a preset long-term whitelist mechanism, the specific technical problem of data duplication or loss during business restarts caused by the default operation and maintenance mechanism of distributed message queues mistakenly clearing displacement data of periodically long-term offline consumer groups is effectively solved. This mechanism significantly ensures the breakpoint resumption capability of periodic tasks such as offline reports in data warehouses, avoids data consistency damage caused by displacement loss, and greatly improves the robustness of distributed message queues in complex business scenarios. At the same time, through precise filtering of the whitelist, it avoids the waste of storage resources caused by full displacement backup of all inactive consumer groups, achieving a perfect balance between efficient use of system resources and business continuity.

[0077] In a further embodiment, after step S1300, which queries the detailed description information of each consumer group whose status value represents its active status, the following steps are included: Step S2300: For each active consumer group, parse the details of each consumer and their subscribed topics in their description information.

[0078] In microservice architectures using distributed message queues, all consumer instances within the same consumer group must subscribe to the exact same list of topics. This is a fundamental prerequisite for ensuring the consumer group can perform partitioning and message consumption correctly. If different consumers within the same consumer group subscribe to different topics, it can lead to serious errors when the consumer group is rebalanced, potentially causing the entire consumer group to crash or completely halt message consumption. However, in actual business iterations, due to inconsistent code version releases, delays in configuration center deployments, or developer errors, a hidden error can occur where some instances within the same consumer group update their subscribed topics while others retain the old ones. This issue of inconsistent subscription topics is often difficult to detect in the early stages of the system, only becoming apparent when a rebalance triggers an online failure, greatly complicating operational troubleshooting.

[0079] To address the potential risks in the aforementioned specific technical scenarios, a deep validation mechanism for subscription consistency within active consumer groups is introduced. For each active consumer group, its detailed description information is parsed to identify each consumer and their subscribed topics. An active consumer group refers to one that is currently running normally and consuming messages. The detailed description information is full metadata about the consumer group's runtime maintained by the distributed message queue server. A consumer is a client instance within the consumer group that actually executes the message retrieval logic. A subscribed topic refers to the message channel that the consumer is currently listening to and consuming. The specific implementation of parsing the detailed description information includes several concrete embodiments. In one embodiment, the server obtains the detailed description object of the consumer group by calling the administrator client interface provided by the distributed message queue, iterates through the list of consumer members within this object, reads the topic partition set in the assigned attributes of each consumer member, and extracts all topic names from it, thereby constructing a mapping relationship between each consumer and its subscribed topics.

[0080] In another embodiment, the server captures text data in its standard output stream by executing the consumer group description command-line tool that comes with the distributed message queue, and uses regular expressions to parse the consumer identifier column and topic column in the text line by line, aggregating topics belonging to the same consumer identifier, and thus obtaining each consumer and its subscribed topics.

[0081] Step S2310: Compare the topics subscribed to by all consumers in the same consumer group to determine whether there are conflicting consumers with inconsistent subscribed topics.

[0082] A conflicting consumer is a consumer instance within the same consumer group whose subscribed topic set is not entirely identical to the topic sets subscribed to by the majority of other consumers in the same group or by the standard configuration. The specific implementation of comparing subscribed topics includes various embodiments. In one embodiment, a strict comparison mechanism based on set equality is employed. The server converts the topic list subscribed to by each consumer into an unordered and element-unique hash set, selects the topic set of the first consumer in the consumer group as the baseline set, and then iterates through the topic sets of all other consumers, checking whether each set is completely equal to the baseline set. If any discrepancies exist, the consumer is determined to be a conflicting consumer.

[0083] In another embodiment, a fault-tolerant comparison mechanism based on the majority principle is adopted. The server counts the frequency of occurrence of the topic sets subscribed to by all consumers in the consumer group, and determines the topic set with the highest frequency as the standard subscription set for the consumer group. Then, each consumer's topic set is compared with the standard subscription set, and consumers who do not match are marked as conflicting consumers. This method can effectively avoid misjudgments caused by incorrect selection of the benchmark set itself.

[0084] In another embodiment, a fast comparison mechanism based on hash digests is employed. The server sorts the list of topics subscribed to by each consumer in lexicographical order, concatenates the sorted list of topics into a single string, and calculates the hash digest value of this string. By comparing whether the hash digest values ​​of all consumers are consistent, it quickly determines whether there are conflicting consumers, thereby significantly reducing the computational overhead when comparing large-scale list topics.

[0085] Step S2320: If there are conflicting consumers, extract the conflicting consumers and their corresponding differing topics, and generate subscription relationship conflict alarm information.

[0086] A discrepancy topic refers to the specific topic name that a conflicting consumer has subscribed to more or less than the standard subscription set. Subscription conflict alert information is structured data used to accurately describe the location of the conflict and the specific discrepancy to operations and maintenance personnel.

[0087] The specific implementation of extracting discrepancy topics and generating alarm information includes several specific embodiments. In one embodiment, an extraction method based on set difference operations is adopted. The server calculates the symmetric difference between the topic set of conflicting consumers and the standard subscription set, directly identifying the elements in the difference set as discrepancy topics. Subsequently, the consumer group name, conflicting consumer identifier, standard subscription set, and list of discrepancy topics are encapsulated into subscription relationship conflict alarm information in JSON format.

[0088] In another embodiment, a classification-based extraction method is used. The server calculates the first difference between the set of conflicting consumer topics and the standard subscription set, and the second difference between the standard subscription set and the set of conflicting consumer topics. Topics in the first difference are marked as redundant subscription topics, and topics in the second difference are marked as missing subscription topics. These difference topics with classification labels, along with contextual information such as consumer network addresses and instance startup times, are used to generate detailed subscription relationship conflict alarm information, so that operation and maintenance personnel can more intuitively understand the specific manifestations of the conflict.

[0089] Step S2330: Attach the subscription relationship conflict alarm information and its corresponding subscription relationship repair suggestions to the response result of the subscription query request.

[0090] Subscription relationship repair suggestions provide specific troubleshooting directions or operational instructions for current subscription conflict issues. The implementation of attaching alarm information and repair suggestions includes various specific embodiments. In one embodiment, a static rule template-based generation and attachment method is used. The server matches a preset repair rule template based on the classification tags of the differing topics. For example, when a missing subscription topic exists, a repair suggestion is generated: "Please check the configuration file or code version of this consumer instance to ensure that it has added the subscription logic for the missing topic." The alarm information and repair suggestion are then appended as independent warning fields to the JSON response result of the subscription query request. Those skilled in the art can flexibly adapt the aforementioned static rule template.

[0091] In another embodiment, an intelligent matching and appending method based on an operations and maintenance knowledge base is employed. The server uses the instance identifier of the conflicting consumer, its node IP address, and the differing topic as query conditions to search a pre-built distributed message queue fault operations and maintenance knowledge base. It then matches the standard operating procedure for resolving similar conflicts in the past, such as "It is recommended to forcibly distribute the latest topic list through the configuration center and continuously restart the container instance where the consumer resides." This standard operating procedure is then appended to the response result as a subscription relationship repair suggestion. Those skilled in the art can flexibly adapt the aforementioned operations and maintenance knowledge base.

[0092] In this embodiment, by proactively performing deep verification of subscription consistency within active consumer groups during the query phase, the specific technical problem of difficulty in early detection and location of rebalancing failures or consumption stagnation caused by inconsistent subscription topics among consumers within the same consumer group is effectively solved. This mechanism, while returning to the normal subscription relationship, accurately intercepts and alerts to potential subscription inconsistencies, providing clear conflict nodes and targeted remediation suggestions. This significantly reduces fault diagnosis and recovery time from hours to minutes, substantially ensuring the stability and high availability of the distributed message queue consumption chain and improving the overall system's operational intelligence.

[0093] In a further embodiment, after parsing the queue access address and subscription relationship membership set in the subscription query information carried by the request in step S1100, the following steps are included: Step S2100: Query whether there is a subscription relationship list in the local cache that matches the queue access address. The subscription relationship list includes all subscription relationship pairs in the distributed message queue corresponding to the queue access address, which are pre-cached and consist of each consumer group whose active status is represented by a status value and the topic they subscribe to.

[0094] In high-concurrency query scenarios of distributed message queues, front-end monitoring dashboards or automated operation and maintenance scripts will frequently and periodically initiate subscription relationship query requests. If each query directly calls the server-side interface of the distributed message queue via the network to pull the full amount of consumer group and topic metadata, it will not only consume a lot of network bandwidth, but also put a huge computational and input / output pressure on the message queue controller node, and may even cause query timeouts or affect the normal message sending and receiving of the message queue itself.

[0095] To address the performance bottlenecks and resource consumption issues caused by high-frequency queries, a subscription relationship query optimization mechanism based on local caching and event-driven incremental updates is introduced. The server first checks its local cache for a list of subscription relationships matching the queue access address. The local cache is a high-speed data storage area deployed in server memory or a distributed cache cluster, used to temporarily store frequently accessed data to avoid redundant calculations and remote network requests. The subscription relationship list is a pre-cached set of all subscription relationship pairs in the distributed message queue corresponding to the queue access address, consisting of each consumer group whose active status is represented by a status value and the topics they subscribe to.

[0096] The implementation of querying the local cache includes several specific embodiments. In one embodiment, a cache query mechanism based on local memory is employed. The server utilizes a high-performance local memory caching component, calculates the queue access address using a hash algorithm, and uses it as the cache key. By calling the retrieval interface of the cache component, it determines whether the corresponding subscription relationship list exists in memory within nanoseconds.

[0097] In another embodiment, a query mechanism based on a distributed cache cluster is employed. The server uses the queue access address as the key and executes an existence check command from the distributed cache system to query whether the subscription relationship list for that queue is stored in the remote cache cluster, thereby enabling cache data sharing among multiple application servers.

[0098] Step S2200: If a subscription relationship list exists, then directly execute the steps for each subscription relationship member in the corresponding subscription relationship member set based on the subscription relationship list, and return all subscription relationship pairs containing the subscription relationship members.

[0099] Execution based directly on the subscription relationship list means skipping the actual network query to the distributed message queue server and relying entirely on existing data in the cache for filtering and response. In practice, this includes several specific implementations. One implementation uses a filtering mechanism based on in-memory streaming computation. The server loads the subscription relationship list obtained from the cache into an in-memory set, uses the streaming processing interface provided by the programming language to traverse each member in the subscription relationship member set, filters out subscription relationship pairs whose consumer group name or topic name exactly matches that member, and collects the filtering results into a list and returns it directly.

[0100] In another embodiment, a fast retrieval mechanism based on an inverted index is employed. When the server writes the subscription relationship list to the cache, it simultaneously constructs an inverted index structure with subscription relationship members as keys and a list of subscription relationship pairs containing those members as values. When results are needed, the server directly queries the inverted index using members from the subscription relationship member set as keys, retrieving and returning the corresponding subscription relationship pairs in constant time complexity, completely eliminating the time overhead of traversal filtering.

[0101] Step S2300: If no subscription relationship list exists, continue to execute the steps of querying all consumer groups and their status values ​​in the distributed message queue corresponding to the queue access address and subsequent steps. After obtaining all subscription relationship pairs, write them as a new subscription relationship list to the local cache.

[0102] A cache miss occurs when the subscription relationship list does not exist. In this case, it is necessary to retrieve the full set of real data from the distributed message queue server and then populate the cache for subsequent requests. The specific implementation of retrieving data and writing it to the local cache includes several implementation methods. One implementation uses a distributed cache population mechanism based on batch pipeline writing. After obtaining the full set of subscription relationship pairs through the administrator client interface, the server serializes them into a byte array. Using the pipeline algorithm of the distributed caching system, multiple write commands are packaged and sent to the cache server at once, and a reasonable expiration time is set to prevent data from permanently residing, thereby significantly reducing network round-trip latency.

[0103] In another embodiment, a backfilling mechanism based on asynchronous loading of local cache is adopted. While synchronously returning query results to the requester, the server submits all obtained subscription pairs to the background asynchronous thread pool. The background thread calls the batch loading interface of the local cache component to write the data into local memory and configures an eviction policy based on capacity or time to ensure that memory usage is kept at a safe level.

[0104] Step S2400: After writing the new subscription relationship list to the local cache, register to listen for metadata change events of the distributed message queue. When a metadata change event is detected, incrementally update the subscription relationship list in the local cache according to the change consumer group or change topic indicated by the metadata change event.

[0105] Metadata change events refer to notification signals triggered by changes in the status of members in a distributed message queue, such as the addition or deletion of a consumer group, the creation or deletion of a topic, or changes in the subscription relationship of a consumer group. Incremental updates refer to modifying only the changed local data, rather than fetching all data again for a complete overwrite. In the specific implementation of registering listeners and executing incremental updates, one embodiment adopts a listening and update mechanism based on the internal system topics of the message queue. The server, as a special consumer, subscribes to the internal system topics of the distributed message queue used to record metadata changes. When it consumes a change message containing a newly added consumer group or a newly subscribed topic, it parses the message content and appends the new subscription relationship pair to the local cache list; when it consumes a change message containing a deleted consumer group or a deleted topic, it uses the deletion or filtering interface of the cache component to remove the subscription relationship pair matching the consumer group or topic from the local cache.

[0106] In this embodiment, by introducing a local cache to intercept high-frequency query requests and combining it with an incremental update mechanism for metadata change events, the specific technical problems caused by high-frequency subscription relationship queries leading to network bandwidth exhaustion on the distributed message queue server and overload of computing resources on the controller node are effectively solved. This mechanism, while ensuring eventual consistency of subscription relationship data, reduces the response time of most query requests from hundreds of milliseconds of remote network calls to milliseconds or even microseconds of local memory reads, completely eliminating the interference of query operations on the core message queue sending and receiving services. This significantly improves the overall throughput of the distributed system, query response time, and the concurrent capacity of the operation and maintenance monitoring platform.

[0107] Please see Figure 2 This application provides a subscription relationship query device based on a distributed message queue, which is a functional embodiment of the subscription relationship query method based on a distributed message queue of this application. The device includes a request response module 1100, a data query module 1200, a subscription mapping module 1300, and a request response module 1400. The request response module 1100 is used to respond to subscription query requests and parse out the queue access address and subscription relationship member set from the subscription query information carried in the request. The data query module 1200 is used to query all consumer groups and their status values ​​in the distributed message queue corresponding to the queue access address. The subscription mapping module 1300 is used to query the detailed description information of each consumer group whose status value represents its active state, and extract the corresponding consumer group and its subscribed topics from each detailed description information to form subscription relationship pairs. The request response module 1400 is used to return all subscription relationship pairs containing the subscription relationship member for each subscription relationship member in the subscription relationship member set, in response to the subscription query request.

[0108] In a further embodiment, the data query module 1200 includes: a consumer group query submodule, used to construct and execute a consumer group query instruction applied to the corresponding distributed message queue based on the queue access address, to obtain a consumer group list containing all consumer groups in the queue; and a status query submodule, used to construct and execute a status query script applied to all consumer groups in the consumer group list, to obtain the status value of each consumer group.

[0109] In a further embodiment, the request-response module 1400 includes: a regular expression submodule, used to construct a query regular expression corresponding to each subscription member in the subscription member set; and a regular query submodule, used to apply the query regular expression to query all extracted subscription pairs to obtain all subscription pairs containing subscription members.

[0110] In a further embodiment, before the request response module 1100, there is an event response module, used to respond to a query parsing event, obtain the subscription query statement expressed in natural language corresponding to the event, and obtain a preset parsing prompt template. The parsing prompt template includes a preset queue access address corresponding to at least one distributed message queue and a list of its matching subscription relationships. The list of subscription relationships includes all subscription relationship pairs in the distributed message queue corresponding to the queue access address, which are pre-cached, consisting of each consumer group whose active status is represented by a status value and the topic it subscribes to. The query parsing module is used to construct parsing prompt text based on the subscription query statement and the parsing prompt template, and use it to guide the large language model to determine the queue access address and subscription relationship member set that match the subscription query statement, and to construct a subscription query request.

[0111] In a further embodiment, after the data query module 1200, the module includes: a duration determination submodule, used to obtain, for each consumer group whose status value represents an inactive state, the update timestamp corresponding to the partition displacement information of the most recent consumption update of each consumer in the consumer group, and the update interval duration from the current timestamp; a consumer group identification submodule, used to confirm that the consumer group is a long-term inactive consumer group if the sum of all update interval durations exceeds a preset cumulative duration; and a store-and-call submodule, used to determine whether each long-term inactive consumer group exists in a preset long-term mounted whitelist, and if so, to enable external storage of the partition displacement information of the most recent consumption update of each consumer in the corresponding consumer group, so that the corresponding partition can be used when the consumer group that was deleted and rebuilt continues to process the consumption.

[0112] In a further embodiment, after the subscription mapping module 1300, the module includes: an information parsing submodule, used to parse the details of each consumer and their subscribed topics in the description information of each active consumer group; a conflict judgment submodule, used to compare the subscribed topics of all consumers in the same consumer group to determine whether there are conflicting consumers with inconsistent subscription topics; an information generation submodule, used to extract the conflicting consumers and their corresponding differing topics if conflicting consumers exist, and generate subscription relationship conflict alarm information; and a result appending submodule, used to append the subscription relationship conflict alarm information and its corresponding subscription relationship repair suggestions to the response result of the subscription query request.

[0113] In a further embodiment, after the request response module 1100, the module includes: a cache query submodule, used to query whether there is a subscription relationship list in the local cache that matches the queue access address, the subscription relationship list including all subscription relationship pairs composed of each consumer group whose active status is represented by a status value and their subscribed topics in the distributed message queue corresponding to the queue access address, which are pre-cached; a cache call submodule, used to, if there is a subscription relationship list, directly execute each subscription relationship member in the corresponding subscription relationship member set based on the subscription relationship list and return all subscription relationship pairs containing the subscription relationship members; a local cache submodule, used to, if there is no subscription relationship list, continue to execute the steps of querying all consumer groups and their status values ​​in the distributed message queue corresponding to the queue access address and their subsequent steps, and after obtaining all subscription relationship pairs, write them as a new subscription relationship list to the local cache; and a cache update submodule, used to, after writing the new subscription relationship list to the local cache, register to listen for metadata change events of the distributed message queue, and when a metadata change event is listened to, incrementally update the subscription relationship list in the local cache according to the changed consumer group or changed topic indicated by the metadata change event.

[0114] To address the aforementioned technical problems, embodiments of this application also provide computer equipment. For example... Figure 3 The diagram shows the internal structure of a computer device. The computer device includes a processor, a computer-readable storage medium, a memory, and a network interface connected via a system bus. The computer-readable storage medium stores an operating system, data, and computer-readable instructions. The data may store a sequence of control information. When the computer-readable instructions are executed by the processor, they enable the processor to implement a subscription relationship query method based on a distributed message queue. The processor of the computer device provides computing and control capabilities, supporting the operation of the entire computer device. The memory of the computer device may store computer-readable instructions. When these computer-readable instructions are executed by the processor, they enable the processor to execute the subscription relationship query method based on a distributed message queue of this application. The network interface of the computer device is used for communication with a terminal. Those skilled in the art will understand that… Figure 3 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0115] In this embodiment, the processor is used to execute... Figure 2 The system contains the specific functions of each module and its sub-modules. The memory stores the program code and various data required to execute these modules or sub-modules. The network interface is used for data transmission between the user terminal and the server. In this embodiment, the memory stores the program code and data required to execute all modules / sub-modules in the distributed message queue-based subscription relationship query device of this application. The server can call the server's program code and data to execute the functions of all sub-modules.

[0116] This application also provides a storage medium storing computer-readable instructions, which, when executed by one or more processors, cause the one or more processors to perform the steps of the subscription relationship query method based on a distributed message queue according to any embodiment of this application.

[0117] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments of this application can be implemented by a computer program instructing related hardware. This computer program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the methods described above. The aforementioned storage medium can be a computer-readable storage medium such as a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM).

[0118] In summary, this application can efficiently and accurately query subscription relationships in a distributed message queue.

[0119] Those skilled in the art will understand that the steps, measures, and solutions in the various operations, methods, and processes discussed in this application can be alternated, modified, combined, or deleted. Furthermore, other steps, measures, and solutions in the various operations, methods, and processes discussed in this application can also be alternated, modified, rearranged, decomposed, combined, or deleted. Furthermore, steps, measures, and solutions in the prior art that are similar to those disclosed in this application can also be alternated, modified, rearranged, decomposed, combined, or deleted.

[0120] The above are only some embodiments of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.< / memberdescription>

Claims

1. A subscription relationship query method based on a distributed message queue, characterized in that, Includes the following steps: In response to a subscription query request, parse out the queue access address and subscription relationship member set from the subscription query information carried in the request; Query all consumer groups and their status values ​​in the distributed message queue corresponding to the queue access address; Query the detailed description information of each consumer group whose status value represents its active status, and extract the corresponding consumer group and its subscribed topics from each detailed description information to form a subscription relationship pair; For each subscription relationship member in the subscription relationship member set, return all subscription relationship pairs containing that subscription relationship member to respond to the subscription query request.

2. The subscription relationship query method based on a distributed message queue according to claim 1, characterized in that, Querying all consumer groups and their status values ​​in the distributed message queue corresponding to the queue access address includes the following steps: Based on the queue access address, construct and execute a consumer group query instruction that acts on the corresponding distributed message queue to obtain a consumer group list containing all consumer groups in the queue. Construct and execute a status query script that operates on all consumer groups in the consumer group list to obtain the status value of each consumer group.

3. The subscription relationship query method based on a distributed message queue according to claim 1, characterized in that, For each subscription relationship member in the subscription relationship member set, return all subscription relationship pairs containing that subscription relationship member, including the following steps: For each subscription member in the subscription member set, construct a corresponding query regular expression for that subscription member; The query regular expression is used to query all extracted subscription relationship pairs to obtain all subscription relationship pairs containing the subscription relationship member.

4. The subscription relationship query method based on a distributed message queue according to claim 1, characterized in that, Before responding to a subscription query request, the following steps are included: In response to a query parsing event, obtain the subscription query statement expressed in natural language corresponding to the event, and obtain a preset parsing prompt template. The parsing prompt template includes a preset queue access address corresponding to at least one distributed message queue and a list of its matching subscription relationships. The list of subscription relationships includes all subscription relationship pairs in the distributed message queue corresponding to the queue access address, which are pre-cached and consist of each consumer group whose active status is represented by a status value and the topic it subscribes to. Based on the subscription query statement and the parsing prompt template, a parsing prompt text is constructed, which guides the large language model to determine the queue access address and subscription relationship member set that match the subscription query statement, and is used to construct the subscription query request.

5. The subscription relationship query method based on a distributed message queue according to claim 1, characterized in that, After querying all consumer groups and their status values ​​in the distributed message queue corresponding to the queue access address, the following steps are included: For each consumer group whose state value represents an inactive state, obtain the update timestamp corresponding to the most recent consumption update partition displacement information of each consumer in the consumer group, and the update interval time from the current timestamp; If the sum of all update interval durations exceeds the preset cumulative duration, then the consumer group is confirmed as a long-term inactive consumer group. Determine whether each long-term inactive consumer group exists in the preset long-term mounted whitelist. If it does, enable external storage of the partition offset information of the most recent consumption update for each consumer in the corresponding consumer group, so that the corresponding partition can be used when the consumer group that was deleted and rebuilt continues to process consumption.

6. The subscription relationship query method based on a distributed message queue according to claim 1, characterized in that, After querying the detailed description information for each consumer group whose status value represents its active status, the following steps are included: For each active consumer group, analyze the detailed description information of each consumer and the topics they subscribe to; Compare the topics subscribed to by all consumers within the same consumer group to determine if there are any conflicting consumers with inconsistent subscription topics; If the conflicting consumer exists, extract the conflicting consumer and the corresponding differing topic, and generate subscription relationship conflict alarm information; The subscription relationship conflict alarm information and its corresponding subscription relationship repair suggestions are appended to the response result of the subscription query request.

7. The subscription relationship query method based on a distributed message queue according to claim 1, characterized in that, After parsing the queue access address and subscription relationship membership set from the subscription query information carried in the request, the following steps are included: Query whether there is a list of subscription relationships in the local cache that matches the queue access address. The list of subscription relationships includes all subscription relationship pairs in the distributed message queue corresponding to the queue access address, which are pre-cached and consist of each consumer group whose active status is represented by a status value and the topic they subscribe to. If the subscription relationship list exists, then the step of directly executing each subscription relationship member in the corresponding subscription relationship member set based on the subscription relationship list and returning all subscription relationship pairs containing the subscription relationship member is executed. If the subscription relationship list does not exist, continue to execute the step of querying all consumer groups and their status values ​​in the distributed message queue corresponding to the queue access address and subsequent steps, and after obtaining all subscription relationship pairs, write them as a new subscription relationship list to the local cache; After writing the new subscription relationship list to the local cache, register to listen for metadata change events of the distributed message queue. When the metadata change event is detected, incrementally update the subscription relationship list in the local cache according to the change consumer group or change topic indicated by the metadata change event.

8. A subscription relationship query device based on a distributed message queue, characterized in that, include: The request and response module is used to respond to subscription query requests and parse out the queue access address and subscription relationship member set from the subscription query information carried in the request. The data query module is used to query all consumer groups and their status values ​​in the distributed message queue corresponding to the queue access address; The subscription mapping module is used to query the detailed description information of each consumer group whose status value represents its active status, and extract the corresponding consumer group and its subscribed topics from each detailed description information to form a subscription relationship pair; The request and response module is used to return all subscription relationship pairs containing each subscription relationship member in the subscription relationship member set in order to respond to the subscription query request.

9. A computer device comprising a central processing unit and a memory, characterized in that, The central processing unit is used to invoke and run a computer program stored in the memory to perform the steps of the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, It stores, in the form of computer-readable instructions, a computer program implemented according to any one of claims 1 to 7, which, when invoked by a computer, executes the steps included in the corresponding method.

Images