Processing method, device, storage medium and program product of message queue system

Abstract

This disclosure provides a processing method, device, storage medium, and program product for a message queue system, applied to a proxy service node in the message queue system. The method involves receiving a first protocol data packet sent by a client; identifying a first protocol type; converting the first protocol data packet into a second protocol data packet using a first protocol adapter corresponding to the first protocol type, wherein the second protocol is a protocol supported by the message queue service cluster of the message queue system; determining a target message queue service node from the message queue service cluster based on the second protocol data packet and the metadata information of the message queue service cluster; and transmitting the second protocol data packet to the target message queue service node for processing. By using a proxy service node for protocol conversion, the compatibility and scalability of the message queue system with multiple protocols are improved, enabling it to adapt to diverse business scenarios, optimize resource utilization and cost, and meet user needs.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and in particular to a processing method, device, storage medium, and program product for a message queue system. Background Technology

[0002] A message queuing system is a container that provides message storage and transmission capabilities. Messages can be sent to a message queuing system, which needs to provide data saving and delivery functions. The main purpose of a message queuing system is to provide message routing and ensure message delivery. If the receiver is unavailable when a message is sent, the message queue will retain the message until it can be successfully delivered.

[0003] Currently, traditional message queue systems typically use fixed protocol patterns. However, as the online scale of message queue systems continues to expand, the protocols of message queue systems are becoming increasingly diverse, with more and more versions. Message queue systems using fixed protocol patterns cannot adapt to different protocol requirements, resulting in poor stability and scalability. Summary of the Invention

[0004] This disclosure provides a message queue system processing method, device, storage medium, and program product to improve the message queue system's compatibility and scalability with multiple protocols.

[0005] In a first aspect, embodiments of this disclosure provide a processing method for a message queue system, applied to a proxy service node in the message queue system, the method comprising:

[0006] Receive the first protocol data packet sent by the client;

[0007] The system identifies a first protocol type and converts the first protocol data packet into a second protocol data packet using a first protocol adapter corresponding to the first protocol type; wherein the second protocol is a protocol supported by the message queue service cluster of the message queue system; different first protocol adapters are configured for different first protocol types;

[0008] Based on the second protocol data packet and the metadata information of the message queue service cluster of the message queue system, the target message queue service node is determined from the message queue service cluster of the message queue system;

[0009] The second protocol data packet is transmitted to the target message queue service node for processing.

[0010] In a second aspect, embodiments of this disclosure provide a processing device for a message queue system, comprising:

[0011] The communication unit is used to receive the first protocol data packet sent by the client;

[0012] The protocol conversion unit is used to identify a first protocol type and convert the first protocol data packet into a second protocol data packet through a first protocol adapter corresponding to the first protocol type; wherein the second protocol is a protocol supported by the message queue service cluster of the message queue system; different first protocol adapters are configured for different first protocol types;

[0013] The node discovery unit is used to determine the target message queue service node from the message queue service cluster of the message queue system based on the second protocol data packet and the metadata information of the message queue service cluster of the message queue system.

[0014] The communication unit is also used to transmit the second protocol data packet to the target message queue service node for processing.

[0015] Thirdly, embodiments of this disclosure provide an electronic device, including: a processor and a memory;

[0016] The memory stores computer-executed instructions;

[0017] The processor executes computer execution instructions stored in the memory, causing the at least one processor to perform the processing method of the message queue system as described in the first aspect and various possible designs of the first aspect.

[0018] Fourthly, embodiments of this disclosure provide a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the message queue system processing method described in the first aspect and various possible designs of the first aspect.

[0019] Fifthly, embodiments of this disclosure provide a computer program product, including a computer program that, when executed by a processor, implements the message queue system processing method described in the first aspect and various possible designs of the first aspect.

[0020] The message queue system processing method, device, storage medium, and program product provided in this disclosure are applied to a proxy service node in the message queue system. The method involves receiving a first protocol data packet sent by a client; identifying a first protocol type; converting the first protocol data packet into a second protocol data packet using a first protocol adapter corresponding to the first protocol type, wherein the second protocol is a protocol supported by the message queue service cluster of the message queue system; configuring different first protocol adapters for different first protocol types; determining a target message queue service node from the message queue service cluster based on the second protocol data packet and the metadata information of the message queue service cluster; and transmitting the second protocol data packet to the target message queue service node for processing. By using a proxy service node for protocol conversion, the compatibility and scalability of the message queue system with multiple protocols are improved, enabling it to adapt to diverse business scenarios, optimize resource utilization and cost, and meet user needs. Attached Figure Description

[0021] To more clearly illustrate the technical solutions in the embodiments of this disclosure or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 A schematic diagram of a scenario for a message queue system processing method provided in an embodiment of this disclosure;

[0023] Figure 2 This is a schematic flowchart of a message queue system processing method provided in an embodiment of the present disclosure;

[0024] Figure 3a This is a schematic diagram of a message queue system architecture based on a connection pool, provided in one embodiment of the present disclosure.

[0025] Figure 3b This is a schematic diagram of another message queue system architecture based on a connection pool, provided in an embodiment of this disclosure.

[0026] Figure 4 This is a schematic diagram illustrating the metadata information rewriting method for a message queue system provided in an embodiment of this disclosure.

[0027] Figure 5 A structural block diagram of a processing device for a message queuing system provided in an embodiment of this disclosure;

[0028] Figure 6 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present disclosure. Detailed Implementation

[0029] To make the objectives, technical solutions, and advantages of the embodiments of this disclosure clearer, the technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this disclosure, and not all embodiments. Based on the embodiments of this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.

[0030] In existing technologies, traditional message queue systems typically use fixed protocol patterns. However, as the online scale of message queue systems continues to expand, the protocols of message queue systems become increasingly rich and the number of versions increases. Message queue systems that use fixed protocol patterns cannot adapt to different protocol requirements, resulting in poor stability and scalability.

[0031] For example, Kafka is a distributed message queue system. Its initial mission was as a log collection tool, but with its enhanced functionality and performance, Kafka has become a widely used data streaming platform in modern distributed systems for real-time data pipelines, stream processing, and message queues. Kafka's core characteristics are high throughput, low latency, distributed nature, strong scalability, and reliability, making it particularly suitable for handling large-scale data streams. While Kafka is a powerful, high-performance distributed messaging system widely used in various real-time data stream processing and message queue scenarios, it also has some disadvantages and limitations that may become pain points in certain application scenarios. These include complex cluster management, high operational costs, and high data storage costs.

[0032] Although Kafka has some disadvantages, its early development and high popularity mean that some new message queues still adopt Kafka-compatible approaches, using new kernels to support Kafka client protocols. However, these still cannot effectively adapt to different protocol requirements, resulting in poor stability and scalability.

[0033] To address the aforementioned technical problems, this disclosure provides a message queue system processing method for application in a proxy service node, thereby enabling the message queue system to be compatible with multiple protocols through the proxy service node.

[0034] In message queue systems, proxy technology refers to adding a proxy node between message producers and consumers and the core components of the message queue system (such as brokers and storage layers). This proxy node is used to send and receive messages, forward traffic, or execute specific tasks. The proxy acts as an intermediary layer for requests, providing standardized interfaces and hiding the complex underlying architecture and implementation details. By introducing proxy technology, a unified access point, dynamic load balancing, and security control can be provided without directly exposing the underlying message queue system, while reducing the client's understanding and adaptation costs for complex underlying protocols. However, existing open-source and public message queue proxy technologies only support simple reverse proxying, lacking support for data unpacking and verification, and protocol conversion, such as protocol conversion between Kafka and other Kafka-compatible message queues. Therefore, in this embodiment, improvements to the proxy service node enable it to convert between different protocols, thereby making the message queue system compatible with multiple protocols.

[0035] The application scenarios of the message queue system processing method of this disclosure embodiment are as follows: Figure 1 As shown, the message queue system deploys one or more proxy service nodes, which receive first protocol data packets sent by clients. The proxy service nodes identify the first protocol type and, through a first protocol adapter corresponding to the first protocol type, convert the first protocol data packet into a second protocol data packet, wherein the second protocol is a protocol supported by the message queue service cluster of the message queue system. Different first protocol adapters are configured for different first protocol types. Based on the second protocol data packet and the metadata information of the message queue service cluster of the message queue system, a target message queue service node is determined from the message queue service cluster of the message queue system. The second protocol data packet is then transmitted to the target message queue service node for processing.

[0036] The processing method of the message queue system disclosed herein will be described in detail below with reference to specific embodiments.

[0037] refer to Figure 2 , Figure 2 This is a schematic flowchart illustrating a processing method for a message queue system according to an embodiment of the present disclosure. The method of this embodiment can be applied to a proxy service node in a message queue system, and the processing method of this message queue system includes:

[0038] S201, Receive the first protocol data packet sent by the client.

[0039] In this embodiment, when the client needs to use the message queue service, it can send a first protocol data packet for the message queue service access request. The first protocol data packet may include, but is not limited to, data packets of various protocols such as Kafka protocol, Kafka REST protocol, MQTT protocol (Message Queuing Telemetry Transport), RocketMQ protocol, etc.

[0040] Optionally, when receiving the first protocol data packet, the data length of the first protocol data packet can be read first. Then, based on the previously read data length, the first protocol data packet can continue to be read until the complete first protocol data packet is obtained. This avoids the "packet merging" problem similar to that in HTTP protocol processing. This is especially true when the first protocol data packet is a Kafka protocol data packet. Considering that the Kafka protocol is an application protocol based on TCP long connections, and since the Kafka protocol is based on frames of specified data length while the TCP protocol is more stream-based, it is necessary to first read the data length of the first protocol data packet, and then, based on the previously read data length, continue to read the first protocol data packet until the complete Kafka protocol frame content is obtained. This can avoid the "packet merging" problem similar to that in HTTP protocol processing.

[0041] Furthermore, since the proxy service node is deployed as a server-side component in the message queue system, it needs to handle network connection requests from clients to establish network connections, receive initial protocol data packets or other data packets sent by clients, and return response data packets to clients. After a client sends a connection request, it can initialize the entire network connection configuration information, including recording remote connection point information and remote connection status. Moreover, for protocols such as Kafka that are based on long-lived connections, the proxy service node initializes the context information needed later during connection initialization to avoid repeatedly fetching configuration information with each request, thus eliminating the need for repeated fetching. In addition to preparing in advance during initialization, when a client disconnects, it also destroys the initialization configuration information to prevent the connection from continuing to occupy resources, thus managing the lifecycle of connection-related resources.

[0042] S202. Identify the first protocol type, and convert the first protocol data packet into a second protocol data packet through the first protocol adapter corresponding to the first protocol type; wherein the second protocol is a protocol supported by the message queue service cluster of the message queue system; different first protocol adapters are configured for different first protocol types.

[0043] In this embodiment, since the first protocol may be Kafka, Kafka REST, MQTT, RocketMQ, etc., to convert different protocols into protocols supported by the message queue service cluster of the message queue system, different protocol adapters can be configured in the proxy service node for different first protocol types to perform the conversion process. The first protocol adapter can be implemented in hardware or software. Upon receiving the first protocol data packet, the first protocol type can be identified through the packet header, which stores the first protocol type in key-value pairs. The first protocol type is determined by reading the key-value pairs. Optionally, the key in the key-value pair can be the first protocol type, and the value can be the version of the first protocol, etc. Further, the first protocol adapter corresponding to the first protocol type is called to perform the protocol conversion process, that is, to convert the first protocol data packet into a second protocol data packet, meeting the protocol requirements that the backend message queue service node can handle.

[0044] Optionally, after determining the first protocol type, the proxy service node can obtain the complete binary data included in the first protocol data packet from the first protocol data packet based on the first protocol type. For ease of distinction, this is referred to as the first binary data. At this point, the program cannot process this data, so a decoder is needed to further convert the first binary data into a first protocol format object. The protocol object is the structured representation of the binary data in the code. For example, converting the binary data in the first protocol data packet into a Kafka protocol format object (Kafka Request object). Further, the first protocol format object can be converted into a second protocol format object to achieve protocol conversion. Optionally, the first protocol and the second protocol can be completely different types of protocols; or the first protocol and the second protocol can be different versions of the same type of protocol. For example, taking a KafkaMetaData request as an example, a higher version client supports the V1 version of the Kafka protocol and will first send a V1 KafkaApiVersions request. At this time, the backend message queue service node may only support the V0 version of the Kafka protocol, that is, it can only process V0 version requests. The proxy service node can then process the V1 version Kafka request. The ApiVersions request is converted into a V0 version request, i.e., a V0 Kafka ApiVersions request, achieving the goal of request version conversion and thus ensuring compatibility with multiple protocols. Finally, an encoder can be used to convert the second protocol format object into binary data through the encoding process of the second binary data, denoted as the second binary data, thereby obtaining the second protocol data packet.

[0045] It should be noted that the conversion process from the first protocol data packet to the second protocol data packet in this embodiment is not limited to the examples in the above embodiments. It can also be implemented in any other feasible way, such as bridging connection, or by using other platforms or tools. This embodiment does not impose any limitations.

[0046] Furthermore, considering that the first binary data in the first protocol data packet may be based on different security protocols, i.e., different authentication mechanisms, such as PLAINTEXT, SSL, SASL_PLAINTEXT, and SASL_SSL, where PLAINTEXT and SSL protocols do not require authentication, while SASL_PLAINTEXT and SASL_SSL protocols do, after decoding the first binary data to convert it into a first protocol format object, it is also necessary to determine the authentication mechanism of the first binary data. If the authentication mechanism requires authentication, i.e., SASL_PLAINTEXT or SASL_SSL, then authentication is performed on the first protocol format object according to the configuration information of the authentication mechanism. If the first protocol format object fails authentication, an authentication error message is returned; if authentication passes, subsequent protocol conversion proceeds. If the authentication mechanism does not require authentication, authentication is skipped. This allows the proxy service node to support authentication functionality, ensuring security.

[0047] S203. Based on the second protocol data packet and the metadata information of the message queue service cluster of the message queue system, determine the target message queue service node from the message queue service cluster of the message queue system.

[0048] In this embodiment, the second protocol data packet and metadata information can be used to determine which message queue service node to send the second protocol data packet to. In other words, it is necessary to determine the target message queue service node of the second protocol data packet, which is equivalent to providing the client with an API interface to access the target message queue service node.

[0049] Optionally, the metadata information of the message queue service cluster of the message queue system may include the address information (such as host, port, ID, etc.) of different message queue service nodes, the distribution of different topic partition data on different message queue service nodes, the distribution of the full amount of data to be consumed by different consumer groups on different message queue service nodes, and so on.

[0050] Furthermore, by determining the target topic of the second protocol data packet or the target consumer group it belongs to, the target message queue service node and its address information can be queried from the metadata information. Specifically, as follows:

[0051] Determine the target topic of the second protocol data packet, and query the target message queue service node corresponding to the target topic based on the metadata information; or

[0052] The target consumer group to which the second protocol data packet belongs is determined, and the consumption progress information of the full data required by the second protocol data packet for the target consumer group is determined. Based on the metadata information, the target message queue service node corresponding to the consumption progress information is determined from the message queue service nodes corresponding to the full data. The consumption progress information is used to identify which part of the full data the second protocol data packet needs to read. Therefore, based on the distribution of the full data across different message queue service nodes in the metadata, the message queue service node where that part of the data is located can be determined, thus serving as the target message queue service node.

[0053] Optionally, considering that the metadata maintained by message queue systems of different protocol types are somewhat different, and the methods of obtaining metadata are also different, various forms of metadata can be uniformly summarized into three categories of metadata. The first category of metadata is the address information (such as host, port, ID, etc.) of different message queue service nodes. The second category of metadata is the distribution of data of different topic partitions on different message queue service nodes. The third category of metadata is the distribution of the full amount of data to be consumed by different consumer groups on different message queue service nodes. In this way, it is possible to support accessing metadata information in a unified way under different protocol conditions.

[0054] S204. The second protocol data packet is transmitted to the target message queue service node for processing.

[0055] In this embodiment, after determining the target message queue service node, the second protocol data packet can be transmitted to the target message queue service node for corresponding processing. Optionally, after completing the processing, the target message queue service node can also return a response data packet to the client through a proxy service node.

[0056] Optionally, when transmitting the second protocol data packet to the target message queue service node, a connection pool can be used. Specifically, in this embodiment, a connection channel between the proxy service node and any message queue service node can be created and added to the connection pool. The connection channel can be established in at least two of the following ways:

[0057] The first method is, for example Figure 3a As shown, only one connection channel is established between the proxy service node and each message queue service node. Different clients accessing the same message queue service node can share the same connection channel. This allows fewer connections to handle a large number of client requests, improving connection utilization.

[0058] The second way is, such as Figure 3b As shown, multiple connection channels can be established between the proxy service node and each message queue service node. Each connection channel can be dedicated to one client. That is, different clients accessing the same message queue service node use different connection channels, which can better maintain the request processing on a single client connection and avoid out-of-order situations during data packet transmission (for example, client A requests A_Request_1 and A_Request_2, and client B requests B_Request_1 and B_Request_2. During transmission, the same connection channel may be used alternately, resulting in out-of-order transmission).

[0059] Based on this, for the second protocol data packet, a target connection channel can be selected from the connection pool to connect to the target message queue service node. The second protocol data packet is then transmitted to the target message queue service node for processing via the target connection channel. Of course, the transmission of the second protocol data packet is not limited to using a connection pool; any other feasible method can also be used.

[0060] The message queue system processing method provided in this embodiment is applied to a proxy service node in the message queue system. It receives a first protocol data packet sent by a client; identifies a first protocol type; and converts the first protocol data packet into a second protocol data packet using a first protocol adapter corresponding to the first protocol type. The second protocol is a protocol supported by the message queue service cluster of the message queue system. Different first protocol adapters are configured for different first protocol types. Based on the second protocol data packet and the metadata information of the message queue service cluster, a target message queue service node is determined from the message queue service cluster. The second protocol data packet is then transmitted to the target message queue service node for processing. Protocol conversion via a proxy service node improves the compatibility and scalability of the message queue system for multiple protocols, adapts to diverse business scenarios, optimizes resource utilization and cost, and meets user needs.

[0061] Based on any of the above embodiments, the message queue system can support multi-tenant management, that is, multiple users can share a single underlying physical cluster (i.e., a message queue service node cluster). It is necessary to abstract the concept of the user-layer logical cluster and isolate the namespace on the proxy service node. On the one hand, this can reduce the intrusion into the core code of the message queue, and on the other hand, it can also quickly iterate some unique customized functions of the serverless architecture.

[0062] When implementing namespace isolation, since multiple users can share an underlying physical cluster, different message queue service nodes at the underlying level share the same underlying shared namespace. At the user level, different namespaces can be configured, that is, different dedicated namespaces can be configured for different clients. On the proxy service node, the conversion between dedicated namespaces and underlying shared namespaces can be performed.

[0063] Specifically, when transmitting the second protocol data packet to the target message queue service node for processing, it may include:

[0064] According to the conversion rules between the client-specific namespace and the underlying shared namespace, the second protocol data packet is converted into a second protocol data packet conforming to the underlying shared namespace, and the second protocol data packet conforming to the underlying shared namespace is transmitted to the target message queue service node for processing.

[0065] The conversion rules between the client-specific namespace and the underlying shared namespace include the following: When converting the client-specific namespace to the underlying shared namespace, certain fields in the client-specific namespace can be detached, such as Topic allocation information in the cluster, target message queue service node information, request information for obtaining the offset of each partition of a Topic, request information for producing messages, and request information for obtaining the progress of a Topic consumed by a Group. Conversely, when converting the underlying shared namespace to the client-specific namespace, certain fields can be added. Furthermore, the mapping relationship between fields in the client-specific namespace and the underlying shared namespace can be maintained. This embodiment does not limit the specific conversion rules.

[0066] In response, when the target message queue service node returns the first response data packet to the client, it can also convert the response data packet into a second response data packet that conforms to the client's private namespace according to the conversion rules between the client's private namespace and the underlying shared namespace, and then send it to the client.

[0067] In addition, a proxy service node or a message queue service node can handle the resources of a large number of tenants. For a public cluster, each tenant's resources are limited, such as inbound and outbound flow, number of connections, connection rate, number of partitions, etc. These resource limits are planned to be managed by quota configuration on the proxy service node, which can distribute the tenant's resources evenly to each message queue service node, or distribute them unevenly to each message queue service node.

[0068] Based on any of the above embodiments, in the architecture of a traditional message queue system, the client needs to connect to the message queue service nodes. Therefore, the message queue system exposes the client with the address information (e.g., IP addresses) of different message queue service nodes, resulting in the need to maintain the address information of multiple message queue service nodes, consuming significant resources. In this embodiment, however, only the address information of one load balancing node needs to be exposed to the client. Therefore, only one address information needs to be maintained. The client can directly establish a connection with the load balancing node, and the proxy service node can rewrite the address information according to the configuration and scenario, thereby enabling data forwarding between the client and different message queue service nodes. Specifically, as shown... Figure 4 As shown, when receiving the first protocol data packet sent by the client, it can be done as follows:

[0069] Obtain the first protocol data packet sent by the client to the load balancing node, wherein the access address information in the first protocol data packet is the address information of the load balancing node, and the first protocol data packet also includes the mapping identification information of the target message queue service node;

[0070] Based on the mapping relationship between the mapping identifier information of the target message queue service node and the corresponding metadata information, the access address information in the first protocol data packet is converted into the address information of the target message queue service node.

[0071] In this embodiment, when the client sends the first protocol data packet, it can directly send the first protocol data packet to the load balancing node based on the address information of the load balancing node. The first protocol data packet can also carry the mapping identifier information of the target message queue service node, such as Partition1, Partition2, etc. On the proxy service node, the access address information in the first protocol data packet can be converted into the address information of the target message queue service node according to the mapping relationship between the mapping identifier information of the target message queue service node and the corresponding metadata information. For example, Partition1 corresponds to the metadata information of message queue service node 1, and Partition2 corresponds to the metadata information of message queue service node 2. By rewriting the metadata information, it is possible to maintain only the address information of one load balancing node. The client can directly establish a connection with the load balancing node, and the proxy service node can rewrite the address information to realize data forwarding between the client and different message queue service nodes, reducing resource consumption.

[0072] Similarly, when the target message queue service node returns the first response data packet to the client, the source address information in the first response data packet, i.e., the address information of the target message queue service node, can be rewritten. Based on the mapping relationship between the mapping identifier information of the target message queue service node and the corresponding metadata information, the source address information of the first response data packet is converted into the address information of the load balancer node, and the mapping identifier information of the target message queue service node is written into the first response data packet; then it is transmitted to the client through the load balancer node.

[0073] Based on the above embodiments, traffic balancing of proxy service nodes can also be achieved through the metadata rewriting process. Specifically, if there are multiple proxy service nodes in the message queue system, traffic under a client-specific namespace is usually handled by one proxy service node, which may lead to uneven traffic distribution. In this embodiment, a proxy service node can be configured to handle traffic under one or more client-specific namespaces through metadata rewriting. Furthermore, a suitable target proxy service node can be selected to balance traffic based on the traffic of each proxy service node. For example, when receiving the first protocol data packet from the client, the proxy service node with the lowest traffic among the multiple proxy service nodes that can handle traffic under the client-specific namespace can receive and process the first protocol data packet. This allows the proxy service node to handle traffic under different client-specific namespaces simultaneously. In other words, the target proxy service node can return different metadata information for different client-specific namespaces through metadata rewriting, thus achieving traffic scheduling and a more balanced traffic distribution.

[0074] Based on any of the above embodiments, the message queue system can also adopt different allocation strategies when returning metadata information to the client, so that the traffic of the proxy service node is more balanced.

[0075] In one architecture, each user can exclusively enjoy a proxy service node cluster. The underlying message queue service node cluster can also use either a shared or exclusive mode. Users are only aware of the proxy service nodes, not the underlying message queue service nodes. If the underlying message queue service nodes adopt a shared mode, they can be oversold at a certain rate, greatly improving the utilization of underlying resources. Because each user has exclusive access to the proxy service node cluster, the overall function of the proxy service nodes is primarily reflected in the physical isolation of underlying resources.

[0076] In another architecture, the message queue system can also support a serverless mode. In serverless mode, to further improve resource utilization, all users will use the same broker service node cluster, with the cluster distinctions implemented logically. In serverless mode, each user within the same broker service node cluster will be assigned different ports. Isolation between users is achieved not only through ALC (Access Control List) but also through the exposed ports on the network side; that is, isolation is achieved through port numbers.

[0077] Based on any of the above embodiments, the message queue system can also dynamically access message queue service nodes. Currently, the service exposure rules of mainstream message queue products such as Kafka and Pulsar all require modifying configuration files and then restarting the service, which is unacceptable for multi-tenant scenarios in a cluster mode. Therefore, the proxy service node can support dynamic access points based on this scenario.

[0078] Currently, message queues like Kafka and Pulsar expose their message queue service nodes based on configuration files. To achieve dynamic access points, in addition to retrieving access points from configuration files, the ability to dynamically obtain access point information is also required. The methods for dynamically obtaining access points differ between cloud-native and conventional deployment scenarios, thus requiring separate support. Natively, access point configuration information uses configuration files. However, in conventional deployments, distributed storage systems like Redis and ZooKeeper might be used to store configuration information for dynamic updates (i.e., dynamic configuration information, Dynamind config). In cloud-native scenarios, configuration data objects from container orchestration engines, such as Kubernetes ConfigMaps, might be used. Therefore, a unified set of rules needs to be defined to describe the configuration information of message queue service nodes. For ease of expansion and understanding, an open-source access point information representation format can be chosen. Regardless of whether the access point originates from a file or other components, it will ultimately be described in a fixed format, including a unique identifier used to start the message queue service node, a unique identifier used to return client access, and protocol-related unique identifiers. Furthermore, multiple configuration methods are used to store the configuration information of the message queue service node. These multiple configuration methods include at least one of the following: configuration files, configuration data objects of the container orchestration engine, and dynamic configuration data stored in the distributed storage system. To avoid the loss or duplication of configuration information from different sources, i.e., different configuration methods, a unified API interface can be provided to obtain configuration information from different configuration methods. At the same time, to ensure that the message queue service node can still work normally when configuration information conflicts, a preset priority for different configuration methods is configured here. For example, the preset priority is determined according to the difficulty and scope of the change, i.e., dynaminc config > configMap > Config Files. When configuration information conflicts, the configuration information from the configuration method with higher priority is used.

[0079] Furthermore, message queue service nodes can be enabled or deleted by monitoring configuration information. For example, the status of message queue service node configuration information under various configuration methods can be monitored based on preset priorities. This status can include "added," "deleted," and "unmodified." Based on the status of the message queue service node's configuration information, it can be enabled or deleted. If the status is "added," the message queue service node is enabled; if the status is "deleted," the message queue service node is deleted; if the status is "unmodified," it remains unchanged. Optionally, version information can be used to determine if the configuration information of the message queue service node has changed. If the version has changed, the configuration information of the message queue service node is obtained from the latest version.

[0080] Corresponding to the message queue system processing method in the above embodiments, Figure 5 This is a structural block diagram of a processing device for a message queuing system provided in an embodiment of this disclosure. For ease of explanation, only the parts relevant to the embodiments of this disclosure are shown. (Refer to...) Figure 5 The processing device 500 of the message queue system includes: a communication unit 501, a protocol conversion unit 502, and a node discovery unit 503.

[0081] The communication unit 501 is used to receive the first protocol data packet sent by the client;

[0082] Protocol conversion unit 502 is used to identify a first protocol type and convert the first protocol data packet into a second protocol data packet through a first protocol adapter corresponding to the first protocol type; wherein the second protocol is a protocol supported by the message queue service cluster of the message queue system; different first protocol adapters are configured for different first protocol types;

[0083] The node discovery unit 503 is used to determine the target message queue service node from the message queue service cluster of the message queue system based on the second protocol data packet and the metadata information of the message queue service cluster of the message queue system.

[0084] The communication unit 501 is further configured to transmit the second protocol data packet to the target message queue service node for processing.

[0085] The processing device for a message queue system provided in this embodiment is applied to a proxy service node in the message queue system. It receives a first protocol data packet sent by a client; identifies a first protocol type; and converts the first protocol data packet into a second protocol data packet using a first protocol adapter corresponding to the first protocol type. The second protocol is a protocol supported by the message queue service cluster of the message queue system. Different first protocol adapters are configured for different first protocol types. Based on the second protocol data packet and the metadata information of the message queue service cluster, a target message queue service node is determined from the message queue service cluster. The second protocol data packet is then transmitted to the target message queue service node for processing. Protocol conversion via a proxy service node improves the compatibility and scalability of the message queue system for multiple protocols, adapts to diverse business scenarios, optimizes resource utilization and cost, and meets user needs.

[0086] In one or more embodiments of this disclosure, the protocol conversion unit 502, when converting the first protocol data packet into a second protocol data packet, is configured to:

[0087] Obtain first binary data from the first protocol data packet, decode the first binary data, and convert the first binary data into a first protocol format object;

[0088] The first protocol format object is converted into a second protocol format object, and the second protocol format object is encoded to convert the second protocol format object into second binary data, thereby obtaining a second protocol data packet.

[0089] In one or more embodiments of this disclosure, after decoding the first binary data to convert the first binary data into a first protocol format object, the protocol conversion unit 502 is further configured to:

[0090] Determine the authentication mechanism for the first binary data;

[0091] If the authentication mechanism is one that requires authentication, then the first protocol format object is authenticated according to the configuration information of the authentication mechanism.

[0092] If the first protocol format object fails authentication, an authentication error message will be returned.

[0093] In one or more embodiments of this disclosure, when the communication unit 501 transmits the second protocol data packet to the target message queue service node for processing, it is used to:

[0094] Select a target connection channel from the connection pool to connect to the target message queue service node, and transmit the second protocol data packet to the target message queue service node for processing through the target connection channel;

[0095] The connection pool includes a single connection channel between the proxy service node and any message queue service node, and different clients accessing the same message queue service node share the same connection channel; or the connection pool includes multiple connection channels between the proxy service node and any message queue service node, and different clients accessing the same message queue service node use different connection channels.

[0096] In one or more embodiments of this disclosure, when the node discovery unit 503 determines the target message queue service node from the message queue service cluster of the message queue system based on the second protocol data packet and the metadata information of the message queue service cluster of the message queue system, it is configured to:

[0097] Determine the target topic of the second protocol data packet, and query the target message queue service node corresponding to the target topic based on the metadata information; or

[0098] The target consumer group to which the second protocol data packet belongs is determined, and the consumption progress information of the full amount of data required by the second protocol data packet for the target consumer group is determined. Based on the metadata information, the target message queue service node corresponding to the consumption progress information is determined from the message queue service nodes corresponding to the full amount of data.

[0099] In one or more embodiments of this disclosure, when the communication unit 501 transmits the second protocol data packet to the target message queue service node for processing, it is used to:

[0100] According to the conversion rules between the client-specific namespace and the underlying shared namespace, the second protocol data packet is converted into a second protocol data packet conforming to the underlying shared namespace, and the second protocol data packet conforming to the underlying shared namespace is transmitted to the target message queue service node for processing; wherein different clients are configured with different dedicated namespaces, and different message queue service nodes share a single underlying shared namespace.

[0101] In one or more embodiments of this disclosure, the communication unit 501 is further configured to:

[0102] The system receives the first response data packet returned by the target message queue service node, and converts the first response data packet into a second response data packet that conforms to the client-specific namespace according to the conversion rules between the client-specific namespace and the underlying shared namespace, and sends it to the client.

[0103] In one or more embodiments of this disclosure, when the communication unit 501 receives a first protocol data packet sent by the client, it is configured to:

[0104] Obtain the first protocol data packet sent by the client to the load balancing node, wherein the access address information in the first protocol data packet is the address information of the load balancing node, and the first protocol data packet also includes the mapping identification information of the target message queue service node;

[0105] Based on the mapping relationship between the mapping identifier information of the target message queue service node and the corresponding metadata information, the access address information in the first protocol data packet is converted into the address information of the target message queue service node.

[0106] In one or more embodiments of this disclosure, the communication unit 501 is further configured to:

[0107] Receive a first response data packet returned by the target message queue service node, wherein the source address information of the first response data packet is the address information of the target message queue service node;

[0108] Based on the mapping relationship between the target message queue service node's mapping identifier information and the corresponding metadata information, the source address information of the first response data packet is converted into the address information of the load balancing node, and the mapping identifier information of the target message queue service node is written into the first response data packet.

[0109] In one or more embodiments of this disclosure, the communication unit 501 is further configured to:

[0110] If there are multiple proxy service nodes in the message queue system, the proxy service node with the lowest traffic will receive the first protocol data packet sent by the client.

[0111] In one or more embodiments of this disclosure, the communication unit 501 is further configured to:

[0112] For any message queue service node in the message queue system, the configuration information of the message queue service node is saved using multiple configuration methods; wherein the multiple configuration methods include at least one of the following: configuration file, configuration data object of container orchestration engine, and dynamic configuration data stored in distributed storage system;

[0113] Based on preset priority, the status of the configuration information of the message queue service node in various configuration methods is monitored, and the message queue service node is enabled or deleted according to the status of the configuration information of the message queue service node.

[0114] The device provided in this embodiment can be used to execute the technical solutions of the above method embodiments. Its implementation principle and technical effect are similar, and will not be described again here.

[0115] To implement the above embodiments, this disclosure also provides an electronic device.

[0116] refer to Figure 6 The diagram illustrates a structural schematic of an electronic device 600 suitable for implementing embodiments of the present disclosure. The electronic device 600 can be a terminal device or a server. The terminal device can include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, personal digital assistants (PDAs), tablet computers, portable media players (PMPs), and in-vehicle terminals (e.g., in-vehicle navigation terminals), as well as fixed terminals such as digital TVs and desktop computers. Figure 6 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0117] like Figure 6 As shown, electronic device 600 may include a processing unit (e.g., a central processing unit, a graphics processing unit, etc.) 601, which can perform various appropriate actions and processes according to a program stored in read-only memory (ROM) 602 or a program loaded from storage device 608 into random access memory (RAM) 603. RAM 603 also stores various programs and data required for the operation of electronic device 600. The processing unit 601, ROM 602, and RAM 603 are interconnected via bus 604. Input / output (I / O) interface 605 is also connected to bus 604.

[0118] Typically, the following devices can be connected to I / O interface 605: input devices 606 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 607 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 608 including, for example, magnetic tapes, hard disks, etc.; and communication devices 609. Communication device 609 allows electronic device 600 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 6 An electronic device 600 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively.

[0119] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable storage medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 609, or installed from a storage device 608, or installed from a ROM 602. When the computer program is executed by a processing device 601, it performs the functions defined in the methods of embodiments of this disclosure.

[0120] It should be noted that the computer-readable storage medium described in this disclosure can be a computer-readable signal medium, a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this disclosure, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may also be any computer-readable storage medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable storage medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0121] The aforementioned computer-readable storage medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device.

[0122] The aforementioned computer-readable storage medium carries one or more programs, which, when executed by the electronic device, cause the electronic device to perform the method shown in the above embodiments.

[0123] Computer program code for performing the operations of this disclosure can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a Local Area Network (LAN) or a Wide Area Network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0124] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0125] The units described in the embodiments of this disclosure can be implemented in software or in hardware. The name of a unit does not necessarily limit the unit itself; for example, the first acquisition unit can also be described as "a unit that acquires at least two Internet Protocol addresses".

[0126] The functions described above in this document can be performed, at least in part, by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: Field Programmable Gate Arrays (FPGAs), Application-Specific Integrated Circuits (ASICs), Application Standard Products (ASSPs), System-on-Chip (SoCs), Complex Programmable Logic Devices (CPLDs), and so on.

[0127] The electronic device, computer-readable storage medium, and computer program product provided in this disclosure are applied to a proxy service node in a message queue system. They receive a first protocol data packet sent by a client; identify a first protocol type; and convert the first protocol data packet into a second protocol data packet using a first protocol adapter corresponding to the first protocol type. The second protocol is a protocol supported by the message queue service cluster of the message queue system. Different first protocol adapters are configured for different first protocol types. Based on the second protocol data packet and the metadata information of the message queue service cluster, a target message queue service node is determined from the message queue service cluster of the message queue system. The second protocol data packet is then transmitted to the target message queue service node for processing. Protocol conversion via a proxy service node improves the compatibility and scalability of the message queue system for multiple protocols, adapts to diverse business scenarios, optimizes resource utilization and cost, and meets user needs.

[0128] In a first aspect, according to one or more embodiments of this disclosure, a processing method for a message queue system is provided, applied to a proxy service node in the message queue system, the method comprising:

[0129] Receive the first protocol data packet sent by the client;

[0130] The system identifies a first protocol type and converts the first protocol data packet into a second protocol data packet using a first protocol adapter corresponding to the first protocol type; wherein the second protocol is a protocol supported by the message queue service cluster of the message queue system; different first protocol adapters are configured for different first protocol types;

[0131] Based on the second protocol data packet and the metadata information of the message queue service cluster of the message queue system, the target message queue service node is determined from the message queue service cluster of the message queue system;

[0132] The second protocol data packet is transmitted to the target message queue service node for processing.

[0133] According to one or more embodiments of this disclosure, converting the first protocol data packet into a second protocol data packet includes:

[0134] Obtain first binary data from the first protocol data packet, decode the first binary data, and convert the first binary data into a first protocol format object;

[0135] The first protocol format object is converted into a second protocol format object, and the second protocol format object is encoded to convert the second protocol format object into second binary data, thereby obtaining a second protocol data packet.

[0136] According to one or more embodiments of this disclosure, after decoding the first binary data to convert the first binary data into a first protocol format object, the method further includes:

[0137] Determine the authentication mechanism for the first binary data;

[0138] If the authentication mechanism is one that requires authentication, then the first protocol format object is authenticated according to the configuration information of the authentication mechanism.

[0139] If the first protocol format object fails authentication, an authentication error message will be returned.

[0140] According to one or more embodiments of this disclosure, transmitting the second protocol data packet to the target message queue service node for processing includes:

[0141] Select a target connection channel from the connection pool to connect to the target message queue service node, and transmit the second protocol data packet to the target message queue service node for processing through the target connection channel;

[0142] The connection pool includes a single connection channel between the proxy service node and any message queue service node, and different clients accessing the same message queue service node share the same connection channel; or the connection pool includes multiple connection channels between the proxy service node and any message queue service node, and different clients accessing the same message queue service node use different connection channels.

[0143] According to one or more embodiments of this disclosure, determining the target message queue service node from the message queue service cluster of the message queue system based on the second protocol data packet and the metadata information of the message queue service cluster of the message queue system includes:

[0144] Determine the target topic of the second protocol data packet, and query the target message queue service node corresponding to the target topic based on the metadata information; or

[0145] The target consumer group to which the second protocol data packet belongs is determined, and the consumption progress information of the full amount of data required by the second protocol data packet for the target consumer group is determined. Based on the metadata information, the target message queue service node corresponding to the consumption progress information is determined from the message queue service nodes corresponding to the full amount of data.

[0146] According to one or more embodiments of this disclosure, transmitting the second protocol data packet to the target message queue service node for processing includes:

[0147] According to the conversion rules between the client-specific namespace and the underlying shared namespace, the second protocol data packet is converted into a second protocol data packet conforming to the underlying shared namespace, and the second protocol data packet conforming to the underlying shared namespace is transmitted to the target message queue service node for processing; wherein different clients are configured with different dedicated namespaces, and different message queue service nodes share a single underlying shared namespace.

[0148] According to one or more embodiments of this disclosure, the method further includes:

[0149] The system receives the first response data packet returned by the target message queue service node, and converts the first response data packet into a second response data packet that conforms to the client-specific namespace according to the conversion rules between the client-specific namespace and the underlying shared namespace, and sends it to the client.

[0150] According to one or more embodiments of this disclosure, the receiving client sends a first protocol data packet, comprising:

[0151] Obtain the first protocol data packet sent by the client to the load balancing node, wherein the access address information in the first protocol data packet is the address information of the load balancing node, and the first protocol data packet also includes the mapping identification information of the target message queue service node;

[0152] Based on the mapping relationship between the mapping identifier information of the target message queue service node and the corresponding metadata information, the access address information in the first protocol data packet is converted into the address information of the target message queue service node.

[0153] According to one or more embodiments of this disclosure, the method further includes:

[0154] Receive a first response data packet returned by the target message queue service node, wherein the source address information of the first response data packet is the address information of the target message queue service node;

[0155] Based on the mapping relationship between the target message queue service node's mapping identifier information and the corresponding metadata information, the source address information of the first response data packet is converted into the address information of the load balancing node, and the mapping identifier information of the target message queue service node is written into the first response data packet.

[0156] According to one or more embodiments of this disclosure, the method further includes:

[0157] If there are multiple proxy service nodes in the message queue system, the proxy service node with the lowest traffic will receive the first protocol data packet sent by the client.

[0158] According to one or more embodiments of this disclosure, the method further includes:

[0159] For any message queue service node in the message queue system, the configuration information of the message queue service node is saved using multiple configuration methods; wherein the multiple configuration methods include at least one of the following: configuration file, configuration data object of container orchestration engine, and dynamic configuration data stored in distributed storage system;

[0160] Based on preset priority, the status of the configuration information of the message queue service node in various configuration methods is monitored, and the message queue service node is enabled or deleted according to the status of the configuration information of the message queue service node.

[0161] Secondly, according to one or more embodiments of this disclosure, a processing device for a message queue system is provided, comprising:

[0162] The communication unit is used to receive the first protocol data packet sent by the client;

[0163] The protocol conversion unit is used to identify a first protocol type and convert the first protocol data packet into a second protocol data packet through a first protocol adapter corresponding to the first protocol type; wherein the second protocol is a protocol supported by the message queue service cluster of the message queue system; different first protocol adapters are configured for different first protocol types;

[0164] The node discovery unit is used to determine the target message queue service node from the message queue service cluster of the message queue system based on the second protocol data packet and the metadata information of the message queue service cluster of the message queue system.

[0165] The communication unit is also used to transmit the second protocol data packet to the target message queue service node for processing.

[0166] According to one or more embodiments of this disclosure, when the protocol conversion unit converts the first protocol data packet into a second protocol data packet, it is configured to:

[0167] Obtain first binary data from the first protocol data packet, decode the first binary data, and convert the first binary data into a first protocol format object;

[0168] The first protocol format object is converted into a second protocol format object, and the second protocol format object is encoded to convert the second protocol format object into second binary data, thereby obtaining a second protocol data packet.

[0169] According to one or more embodiments of this disclosure, after decoding the first binary data to convert the first binary data into a first protocol format object, the protocol conversion unit is further configured to:

[0170] Determine the authentication mechanism for the first binary data;

[0171] If the authentication mechanism is one that requires authentication, then the first protocol format object is authenticated according to the configuration information of the authentication mechanism.

[0172] If the first protocol format object fails authentication, an authentication error message will be returned.

[0173] According to one or more embodiments of this disclosure, when the communication unit transmits the second protocol data packet to the target message queue service node for processing, it is configured to:

[0174] Select a target connection channel from the connection pool to connect to the target message queue service node, and transmit the second protocol data packet to the target message queue service node for processing through the target connection channel;

[0175] The connection pool includes a single connection channel between the proxy service node and any message queue service node, and different clients accessing the same message queue service node share the same connection channel; or the connection pool includes multiple connection channels between the proxy service node and any message queue service node, and different clients accessing the same message queue service node use different connection channels.

[0176] According to one or more embodiments of this disclosure, when the node discovery unit determines a target message queue service node from the message queue service cluster of the message queue system based on the second protocol data packet and the metadata information of the message queue service cluster of the message queue system, it is configured to:

[0177] Determine the target topic of the second protocol data packet, and query the target message queue service node corresponding to the target topic based on the metadata information; or

[0178] The target consumer group to which the second protocol data packet belongs is determined, and the consumption progress information of the full amount of data required by the second protocol data packet for the target consumer group is determined. Based on the metadata information, the target message queue service node corresponding to the consumption progress information is determined from the message queue service nodes corresponding to the full amount of data.

[0179] According to one or more embodiments of this disclosure, when the communication unit transmits the second protocol data packet to the target message queue service node for processing, it is configured to:

[0180] According to the conversion rules between the client-specific namespace and the underlying shared namespace, the second protocol data packet is converted into a second protocol data packet conforming to the underlying shared namespace, and the second protocol data packet conforming to the underlying shared namespace is transmitted to the target message queue service node for processing; wherein different clients are configured with different dedicated namespaces, and different message queue service nodes share a single underlying shared namespace.

[0181] According to one or more embodiments of this disclosure, the communication unit is further configured to:

[0182] The system receives the first response data packet returned by the target message queue service node, and converts the first response data packet into a second response data packet that conforms to the client-specific namespace according to the conversion rules between the client-specific namespace and the underlying shared namespace, and sends it to the client.

[0183] According to one or more embodiments of this disclosure, when the communication unit receives a first protocol data packet sent by the client, it is configured to:

[0184] Obtain the first protocol data packet sent by the client to the load balancing node, wherein the access address information in the first protocol data packet is the address information of the load balancing node, and the first protocol data packet also includes the mapping identification information of the target message queue service node;

[0185] Based on the mapping relationship between the mapping identifier information of the target message queue service node and the corresponding metadata information, the access address information in the first protocol data packet is converted into the address information of the target message queue service node.

[0186] According to one or more embodiments of this disclosure, the communication unit is further configured to:

[0187] Receive a first response data packet returned by the target message queue service node, wherein the source address information of the first response data packet is the address information of the target message queue service node;

[0188] Based on the mapping relationship between the target message queue service node's mapping identifier information and the corresponding metadata information, the source address information of the first response data packet is converted into the address information of the load balancing node, and the mapping identifier information of the target message queue service node is written into the first response data packet.

[0189] According to one or more embodiments of this disclosure, the communication unit is further configured to:

[0190] If there are multiple proxy service nodes in the message queue system, the proxy service node with the lowest traffic will receive the first protocol data packet sent by the client.

[0191] According to one or more embodiments of this disclosure, the communication unit is further configured to:

[0192] For any message queue service node in the message queue system, the configuration information of the message queue service node is saved using multiple configuration methods; wherein the multiple configuration methods include at least one of the following: configuration file, configuration data object of container orchestration engine, and dynamic configuration data stored in distributed storage system;

[0193] Based on preset priority, the status of the configuration information of the message queue service node in various configuration methods is monitored, and the message queue service node is enabled or deleted according to the status of the configuration information of the message queue service node.

[0194] Thirdly, according to one or more embodiments of the present disclosure, an electronic device is provided, comprising: at least one processor and a memory;

[0195] The memory stores computer-executed instructions;

[0196] The at least one processor executes computer execution instructions stored in the memory, causing the at least one processor to perform the processing method of the message queue system as described in the first aspect and various possible designs of the first aspect.

[0197] Fourthly, according to one or more embodiments of the present disclosure, a computer-readable storage medium is provided, wherein computer-executable instructions are stored therein, and when a processor executes the computer-executable instructions, the processing method of the message queue system described in the first aspect and various possible designs of the first aspect is implemented.

[0198] Fifthly, according to one or more embodiments of the present disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements the processing method of the message queue system as described in the first aspect and various possible designs of the first aspect.

[0199] In summary, the process involves a proxy service node in the message queue system receiving a first protocol data packet sent by a client; identifying the first protocol type; and converting the first protocol data packet into a second protocol data packet using a first protocol adapter corresponding to the first protocol type. The second protocol is a protocol supported by the message queue service cluster of the message queue system. Different first protocol adapters are configured for different first protocol types. Based on the second protocol data packet and the metadata information of the message queue service cluster, a target message queue service node is determined from the message queue service cluster. The second protocol data packet is then transmitted to the target message queue service node for processing. This protocol conversion via a proxy service node improves the compatibility and scalability of the message queue system across multiple protocols, allowing it to adapt to diverse business scenarios, optimize resource utilization and cost, and meet user needs.

[0200] The above description is merely a preferred embodiment of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features disclosed in this disclosure that have similar functions.

[0201] Furthermore, while the operations are described in a specific order, this should not be construed as requiring these operations to be performed in the specific order shown or in a sequential order. In certain environments, multitasking and parallel processing may be advantageous. Similarly, while several specific implementation details are included in the above discussion, these should not be construed as limiting the scope of this disclosure. Certain features described in the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments.

[0202] Although the subject matter has been described using language specific to structural features and / or methodological logic, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are merely illustrative examples of implementing the claims.

Claims

1. A processing method of a message queue system, characterized by, The method, applied to a proxy service node in a message queue system, includes: Receive the first protocol data packet sent by the client; The system identifies a first protocol type and converts the first protocol data packet into a second protocol data packet using a first protocol adapter corresponding to the first protocol type; wherein the second protocol is a protocol supported by the message queue service cluster of the message queue system; different first protocol adapters are configured for different first protocol types; Based on the second protocol data packet and the metadata information of the message queue service cluster of the message queue system, the target message queue service node is determined from the message queue service cluster of the message queue system. The metadata information includes a first type of metadata, a second type of metadata, and a third type of metadata. The first type of metadata includes the address information of different message queue service nodes; the second type of metadata includes the distribution of different topic partition data on different message queue service nodes; and the third type of metadata includes the distribution of the full amount of data to be consumed by different consumer groups on different message queue service nodes. According to the conversion rules between the client-specific namespace and the underlying shared namespace, the second protocol data packet is converted into a second protocol data packet conforming to the underlying shared namespace, and the second protocol data packet conforming to the underlying shared namespace is transmitted to the target message queue service node for processing through a connection pool; wherein, the connection pool includes a connection channel between the proxy service node and any message queue service node; wherein different clients are configured with different dedicated namespaces, and different message queue service nodes share a common underlying shared namespace.

2. The method of claim 1, wherein, The step of converting the first protocol data packet into a second protocol data packet includes: Obtain first binary data from the first protocol data packet, decode the first binary data, and convert the first binary data into a first protocol format object; The first protocol format object is converted into a second protocol format object, and the second protocol format object is encoded to convert the second protocol format object into second binary data, thereby obtaining a second protocol data packet.

3. The method of claim 2, wherein, After decoding the first binary data to convert it into a first protocol format object, the process further includes: Determine the authentication mechanism for the first binary data; If the authentication mechanism is one that requires authentication, then the first protocol format object is authenticated according to the configuration information of the authentication mechanism. If the first protocol format object fails authentication, an authentication error message will be returned.

4. The method of claim 1, wherein, The step of transmitting the second protocol data packet to the target message queue service node for processing includes: Select a target connection channel from the connection pool to connect to the target message queue service node, and transmit the second protocol data packet to the target message queue service node for processing through the target connection channel; The connection pool includes a single connection channel between the proxy service node and any message queue service node, and different clients accessing the same message queue service node share the same connection channel; or the connection pool includes multiple connection channels between the proxy service node and any message queue service node, and different clients accessing the same message queue service node use different connection channels.

5. The method according to any one of claims 1 to 4, characterized in that, The step of determining the target message queue service node from the message queue service cluster of the message queue system based on the second protocol data packet and the metadata information of the message queue service cluster of the message queue system includes: Determine the target topic of the second protocol data packet, and query the target message queue service node corresponding to the target topic based on the metadata information; or The target consumer group to which the second protocol data packet belongs is determined, and the consumption progress information of the full amount of data required by the second protocol data packet for the target consumer group is determined. Based on the metadata information, the target message queue service node corresponding to the consumption progress information is determined from the message queue service nodes corresponding to the full amount of data.

6. The method according to any one of claims 1 to 4, characterized in that, The method further includes: The system receives the first response data packet returned by the target message queue service node, and converts the first response data packet into a second response data packet that conforms to the client-specific namespace according to the conversion rules between the client-specific namespace and the underlying shared namespace, and sends it to the client.

7. The method according to any one of claims 1 to 4, characterized in that, The first protocol data packet sent by the receiving client includes: Obtain the first protocol data packet sent by the client to the load balancing node, wherein the access address information in the first protocol data packet is the address information of the load balancing node, and the first protocol data packet also includes the mapping identification information of the target message queue service node; Based on the mapping relationship between the mapping identifier information of the target message queue service node and the corresponding metadata information, the access address information in the first protocol data packet is converted into the address information of the target message queue service node.

8. The method according to claim 7, characterized in that, The method further includes: Receive a first response data packet returned by the target message queue service node, wherein the source address information of the first response data packet is the address information of the target message queue service node; Based on the mapping relationship between the target message queue service node's mapping identifier information and the corresponding metadata information, the source address information of the first response data packet is converted into the address information of the load balancing node, and the mapping identifier information of the target message queue service node is written into the first response data packet.

9. The method according to any one of claims 1-4, characterized in that, The first protocol data packet sent by the receiving client includes: If there are multiple proxy service nodes in the message queue system, the proxy service node with the lowest traffic will receive the first protocol data packet sent by the client.

10. The method according to any one of claims 1 to 4, characterized in that, The method further includes: For any message queue service node in the message queue system, the configuration information of the message queue service node is saved using multiple configuration methods; wherein the multiple configuration methods include at least one of the following: configuration file, configuration data object of container orchestration engine, and dynamic configuration data stored in distributed storage system; Based on preset priority, the status of the configuration information of the message queue service node in various configuration methods is monitored, and the message queue service node is enabled or deleted according to the status of the configuration information of the message queue service node.

11. A processing device for a message queue system, characterized in that, include: The communication unit is used to receive the first protocol data packet sent by the client; The protocol conversion unit is used to identify a first protocol type and convert the first protocol data packet into a second protocol data packet through a first protocol adapter corresponding to the first protocol type; wherein the second protocol is a protocol supported by the message queue service cluster of the message queue system; different first protocol adapters are configured for different first protocol types; The node discovery unit is used to determine the target message queue service node from the message queue service cluster of the message queue system based on the second protocol data packet and the metadata information of the message queue service cluster of the message queue system. The metadata information includes a first type of metadata, a second type of metadata, and a third type of metadata. The first type of metadata includes the address information of different message queue service nodes; the second type of metadata includes the distribution of different topic partition data on different message queue service nodes; and the third type of metadata includes the distribution of the full amount of data to be consumed by different consumer groups on different message queue service nodes. The communication unit is further configured to convert the second protocol data packet into a second protocol data packet conforming to the underlying shared namespace according to the conversion rules between the client-specific namespace and the underlying shared namespace, and transmit the second protocol data packet conforming to the underlying shared namespace to the target message queue service node for processing through a connection pool; wherein the connection pool includes a connection channel between the proxy service node and any message queue service node; wherein different clients are configured with different dedicated namespaces, and different message queue service nodes share a single underlying shared namespace.

12. An electronic device, comprising: include: Processor and memory; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory, causing the processor to perform the method as described in any one of claims 1-10.

13. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, implement the method as described in any one of claims 1-10.

14. A computer program product comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1-10.

Images