Cloud service discovery method and device, storage medium and computer program product

Abstract

The invention discloses a cloud service discovery method and device, a storage medium and a computer program product, and relates to the technical field of data processing, and the method comprises the steps: requesting cluster service configuration information from a Kafka management and control platform according to a pre-established communication connection channel, and receiving a metadata request sent by a client; judging the request type of the metadata request, and forwarding the metadata request to a Kafka cluster based on the request type and cluster service configuration information, so that the Kafka cluster returns response data according to the metadata request; and receiving response data, packaging the response data, and sending the packaged response data to the client, so that the client performs data writing and / or data reading according to the response data. Metadata requests are forwarded through the Kafka metadata gateway, the Kafka metadata gateway serves as a service discovery intermediary, a client does not need to concern details of a bottom layer cluster and depends on the centralized control and forwarding capacity of the Kafka metadata gateway serving as an intermediary layer, and unified entry non-inductive access and load balancing of the Kafka service on the cloud are achieved.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a cloud service discovery method, device, storage medium, and computer program product. Background Art

[0002] In enterprise-level applications, numerous Kafka clusters are deployed in the cloud to meet the demands of data production and consumption. Currently, Kafka's cloud service discovery mechanism primarily relies on clients directly configuring the actual Kafka cluster address, or using custom clients for cloud service discovery. Due to the numerous and dispersed Kafka clusters in the cloud and the lack of a unified entry point, clients must manually configure the corresponding Kafka cluster address for each business scenario, resulting in a complex and error-prone configuration process. Once misconfigured, clients will be unable to access the Kafka cluster properly and may even frequently establish and disconnect connections, impacting cluster stability. The second approach requires clients to be retrofitted to integrate cloud service discovery functionality, which increases client complexity and dependency, making management more difficult. Subsequent upgrades require the involvement of a large number of clients, resulting in high costs and challenges with unified management.

[0003] Therefore, how to achieve seamless unified access and load balancing for Kafka services on the cloud has become an urgent problem that needs to be solved in this application.

[0004] The above content is only used to assist in understanding the technical solution of this application and does not constitute an admission that the above content is prior art. Summary of the Invention

[0005] The main purpose of this application is to provide a cloud service discovery method, device, storage medium and computer program product, aiming to solve the technical problem of how to achieve unified entrance and seamless access and load balancing of Kafka services on the cloud.

[0006] To achieve the above objectives, this application proposes a cloud service discovery method applied to a Kafka metadata gateway, the method comprising:

[0007] Requests cluster service configuration information from the Kafka management and control platform through the pre-established communication connection channel, and receives metadata requests sent by the client;

[0008] Determining a request type of the metadata request, and forwarding the metadata request to a Kafka cluster based on the request type and the cluster service configuration information, so that the Kafka cluster returns response data according to the metadata request;

[0009] The response data is received, and the response data is encapsulated and sent to the client, so that the client can write data and / or read data according to the response data.

[0010] In one embodiment, before the step of requesting cluster service configuration information from the Kafka management and control platform according to the pre-established communication connection channel and receiving the metadata request sent by the client, the following steps are further included:

[0011] Obtain a communication configuration parameter set, and initiate a connection request to the Kafka management and control platform based on the communication configuration parameter set, so that the Kafka management and control platform establishes a bidirectional connection according to the connection request;

[0012] Performing encryption protocol handshake, identity authentication, and application layer protocol version alignment based on the bidirectional connection to generate the communication connection channel;

[0013] Responding to the heart detection packet sent by the Kafka management and control platform, and determining the health status of the communication connection channel according to the response result of the heart detection packet;

[0014] Under the condition that the health status is available, the communication connection channel is enabled.

[0015] In one embodiment, the step of determining the request type of the metadata request and forwarding the metadata request to a Kafka cluster based on the request type and the cluster service configuration information, so that the Kafka cluster returns response data according to the metadata request, includes:

[0016] Parsing a request header field of the metadata request, and determining a request type of the metadata request according to a result of parsing the request header field, wherein the request type includes a producer client metadata request and a consumer client metadata request;

[0017] If the request type is a producer client metadata request, forwarding the producer client metadata request to the Kafka cluster based on the cluster service configuration information, so that the Kafka cluster returns first response data according to the producer client metadata request;

[0018] If the request type is a consumer client metadata request, the consumer client metadata request is forwarded to the Kafka cluster based on the cluster service configuration information, so that the Kafka cluster returns second response data according to the consumer client metadata request.

[0019] In one embodiment, the client includes a producer client and a consumer client, and the steps of receiving the response data, encapsulating the response data, and sending the response data to the client so that the client can write data and / or read data according to the response data include:

[0020] Receive the first response data, encapsulate the first response data and send it to the producer client, so that the producer client initiates creation of a connection to the Kafka cluster according to the first response data, and writes data according to the created connection; and / or

[0021] receiving the second response data, encapsulating the second response data and sending it to the consumer client, so that the consumer client can send a consumption coordinator search request according to the second response data;

[0022] Obtain the consumer coordinator search request, and forward the consumer coordinator search request to the Kafka cluster, so that the Kafka cluster returns consumer coordinator information according to the coordinator search request;

[0023] The consumer coordinator information is obtained and forwarded to the consumer client, so that the consumer client can initiate a consumption request to the Kafka cluster to read data according to the consumer coordinator information.

[0024] In addition, to achieve the above objectives, the present application also proposes a cloud service discovery method, which is applied to a client and includes:

[0025] Sending a metadata request to the Kafka metadata gateway, so that the Kafka metadata gateway determines the request type of the metadata request and forwards the metadata request to the Kafka cluster based on the request type and cluster service configuration information, where the cluster configuration service information is requested by the Kafka metadata gateway from the Kafka management and control platform via a pre-established communication connection channel;

[0026] Receive the response data returned by the Kafka metadata gateway, and write and / or read data according to the response data, wherein the response data is encapsulated and sent by the Kafka metadata gateway after the Kafka cluster returns it to the Kafka metadata gateway according to the metadata request.

[0027] In one embodiment, the step of writing data and / or reading data according to the response data includes:

[0028] Initiate creation of a connection to the Kafka cluster according to the response data, and write data according to the created connection; and / or

[0029] Sending a consumer coordinator search request to the Kafka metadata gateway according to the response data, so that the Kafka metadata gateway forwards the consumer coordinator search request to the Kafka cluster to obtain consumer coordinator information;

[0030] Obtain the consumer coordinator information sent by the Kafka metadata gateway, and send a group entry request to the Kafka cluster according to the consumer coordinator information;

[0031] Sending a grouping request to the Kafka cluster based on the grouping request, and synchronizing consumption allocation plan information according to the grouping request;

[0032] Sending a partition data request to the Kafka cluster based on the consumption allocation scheme information, so that the Kafka cluster returns partition data information according to the partition data request;

[0033] A message pull request is sent according to the partition data information, and data is read according to the message pull request.

[0034] In one embodiment, after the step of writing data and / or reading data according to the response data, the method further includes:

[0035] Sending a heartbeat request to the Kafka cluster regularly, so that the Kafka cluster can keep the session alive according to the heartbeat request;

[0036] Submit an offset submission request according to pre-configured submission parameters, and save consumption location information according to the offset submission request.

[0037] In addition, to achieve the above-mentioned purpose, the present application also proposes a cloud service discovery method device, which includes: a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the computer program is configured to implement the steps of the cloud service discovery method as described above.

[0038] In addition, to achieve the above-mentioned purpose, the present application also proposes a storage medium, which is a computer-readable storage medium, and stores a computer program on the storage medium. When the computer program is executed by the processor, the steps of the cloud service discovery method described above are implemented.

[0039] In addition, to achieve the above-mentioned purpose, the present application also provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, it implements the steps of the cloud service discovery method as described above.

[0040] One or more technical solutions proposed in this application have at least the following technical effects:

[0041] The Kafka metadata gateway requests cluster service configuration information from the Kafka management platform based on a pre-established communication connection channel, and receives metadata requests sent by the client; determines the request type of the metadata request, and forwards the metadata request to the Kafka cluster based on the request type and the cluster service configuration information, so that the Kafka cluster can return response data based on the metadata request; receives the response data, encapsulates the response data, and sends it to the client, so that the client can write and / or read data based on the response data. First, the Kafka metadata gateway is "disguised" as a server node, processing the client metadata request in an intermediary role, forwarding the metadata request to the real Kafka cluster to obtain the response data and then returning it to the client, so that the client is unaware of the real cluster and can access it without modification. Furthermore, metadata requests are forwarded based on the request type and cluster configuration, and requests are evenly distributed to different Kafka nodes to avoid overloading a single node, thereby improving the overall throughput and stability of the cluster and achieving load balancing. In summary, this application forwards metadata requests through the Kafka metadata gateway, so that the client does not need to care about the details of the underlying cluster. It relies on the centralized control and forwarding capabilities of the Kafka metadata gateway as an intermediary layer to achieve unified entry and seamless access and load balancing of Kafka services on the cloud. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0043] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0044] Figure 1 A flowchart of the first embodiment of the cloud service discovery method provided in this application;

[0045] Figure 2 This is a diagram of the interaction between the Kafka metadata gateway, the Kafka management and control platform, and the Kafka cluster;

[0046] Figure 3 A flowchart of the second embodiment of the cloud service discovery method provided in this application;

[0047] Figure 4 The producer client provided for this application is implemented through the Kafka gateway;

[0048] Figure 5 The consumer client provided for this application is implemented through the Kafka gateway;

[0049] Figure 6 A flowchart of the fourth embodiment of the cloud service discovery method of this application is provided;

[0050] Figure 7 Provides a Kafka cloud platform module interaction diagram;

[0051] Figure 8 Provides a diagram of the Kafka gateway deployment architecture;

[0052] Figure 9 This is a schematic diagram of the module structure of the Kafka cloud service discovery device in an embodiment of the present application;

[0053] Figure 10 This is a schematic diagram of the device structure of the hardware operating environment involved in the cloud service discovery method in the embodiment of the present application.

[0054] The purpose, features and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0055] It should be understood that the specific embodiments described herein are merely used to explain the technical solutions of the present application and are not intended to limit the present application.

[0056] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0057] The main solution of the embodiment of the present application is: using the Kafka metadata gateway as a forwarding intermediary, the Kafka metadata gateway requests cluster service configuration information from the Kafka management and control platform according to a pre-established communication connection channel, and receives the metadata request sent by the client; judging the request type of the metadata request, and forwarding the metadata request to the Kafka cluster based on the request type and the cluster service configuration information, so that the Kafka cluster can return response data according to the metadata request; receiving the response data, encapsulating the response data and sending it to the client, so that the client can write and / or read data according to the response data.

[0058] The embodiments of the present application take into account that Kafka is a distributed message queue, and its core function is to enable the "party that sends data" (producer) and the "party that receives data" (consumer) to efficiently transmit messages through the middle "Kafka cluster". The Kafka cluster consists of multiple Brokers (server nodes), which are responsible for storing and managing messages to ensure reliable data transmission. In traditional locally deployed Kafka clusters, because the Broker address is fixed, producers / consumers can directly connect to the Broker node. In a cloud environment, Kafka clusters are usually "elastically scalable", which is specifically manifested in: when the business volume increases, the cloud platform will automatically add a new Broker node and the address will change; when a node fails, the cloud platform will replace the new node and the old address will become invalid. As a result, producers / consumers need to frequently update the Broker address, so it is very troublesome to implement the service discovery function and process client metadata requests.

[0059] The embodiments of the present application also take into account that: in enterprise-level applications, a large number of Kafka clusters are deployed on the cloud to meet the needs of data production and consumption. The current cloud service discovery method mechanism mainly relies on the client to directly configure the real Kafka cluster address, or to perform the cloud service discovery method through a customized client. In the first method, since there are many and scattered Kafka clusters on the cloud and there is a lack of a unified entrance, the client needs to manually configure the corresponding Kafka cluster address for each business scenario, resulting in a complex and error-prone configuration process. Once the configuration is wrong, the client will not be able to access the Kafka cluster normally, and may even frequently establish and disconnect connections, affecting the stability of the cluster. The second method requires the client to be modified to integrate the cloud service discovery method function, which increases the complexity and dependency of the client, makes management difficult, and requires subsequent upgrades to involve a large number of clients, which is costly and difficult to manage uniformly.

[0060] Therefore, the present application provides a solution, in which the Kafka metadata gateway requests cluster service configuration information from the Kafka management platform according to a pre-established communication connection channel, and receives the metadata request sent by the client; determines the request type of the metadata request, and forwards the metadata request to the Kafka cluster based on the request type and the cluster service configuration information, so that the Kafka cluster can return response data according to the metadata request; receives the response data, encapsulates the response data and sends it to the client, so that the client can write and / or read data according to the response data. First, the Kafka metadata gateway is "disguised" as a server node, and processes the client metadata request in an intermediary role, forwards the metadata request to the real Kafka cluster to obtain the response data and then returns it to the client, so that the client is unaware of the real cluster and can access it without modification. Furthermore, the metadata request is forwarded according to the request type and cluster configuration, and the request is evenly distributed to different Kafka nodes to avoid overloading of a single node, thereby improving the overall throughput and stability of the cluster and achieving load balancing. To summarize, this application forwards metadata requests through the Kafka metadata gateway, which acts as a "service discovery intermediary" so that the client does not need to worry about the details of the underlying cluster. It relies on the centralized control and forwarding capabilities of the Kafka metadata gateway as an intermediary layer to achieve unified entry and seamless access and load balancing of Kafka services on the cloud.

[0061] Based on this, the embodiment of the present application provides a cloud service discovery method, which is applied to the Kafka metadata gateway. Figure 1 , Figure 1 This is a flowchart of the first embodiment of the cloud service discovery method of this application.

[0062] In this embodiment, the cloud service discovery method includes steps S10 to S30:

[0063] Step S10: Request cluster service configuration information from the Kafka management and control platform through the pre-established communication connection channel, and receive the metadata request sent by the client;

[0064] The communication connection channel is a dedicated communication link established between the Kafka metadata gateway and the Kafka management platform, as well as the Kafka cluster, through a persistent TCP connection or scheduled HTTP long polling. Figure 2 , Figure 2 This is an interaction diagram between the Kafka metadata gateway, the Kafka management and control platform, and the Kafka cluster. Figure 2As shown in the figure, the Kafka metadata gateway, as a component implementing Kafka service discovery, provides unified metadata management and certain request degradation capabilities. It supports the Kafka messaging protocol but does not act as a Kafka server to store data sent by clients. In other words, service discovery can only handle a subset of Kafka requests, and it can be simply understood as a semi-realistic Kafka broker. It only handles four types of requests from Kafka producer / consumer clients: METADATA (metadata requests), FIND_COORDINATOR (consumer group coordinator search), API_VERSIONS (API version negotiation), and INIT_PRODUCER_ID (initializing the producer ID).

[0065] The Kafka metadata gateway periodically requests cluster service configuration information from the Kafka management platform through a pre-established communication connection channel. Specific requests include: cluster node list (Broker IP:Port), topic partition distribution (including Leader / Replica nodes), node load status (CPU / memory / network usage), faulty node markings, dynamic scaling information, etc., ensuring that the Kafka metadata gateway always maintains the latest and accurate Kafka cluster configuration status. It is understandable that when users perform operations such as instance creation, topic creation, user creation, and ACL management on the Kafka management platform, some Kafka-related service configuration data will be stored on the Kafka management platform after the operations are completed.

[0066] Additionally, the Kafka metadata gateway receives metadata requests from clients. Clients are producer / consumer applications that access the Kafka service. Clients only need to configure the gateway's unified access address, not specific Kafka cluster node information. Metadata requests are essential prerequisites for client interaction with Kafka. These requests include querying the existence of a topic, obtaining the node address of a partition leader, obtaining the distribution of partition replicas, and confirming the protocol versions supported by the cluster. All metadata requests pass through the Kafka gateway rather than being sent directly to the Kafka cluster nodes.

[0067] Therefore, the Kafka metadata gateway becomes the only traffic entrance between the client and the cluster, shielding the complex topology of the underlying multi-cluster / multi-node topology and simplifying client configuration. The client does not need to directly perceive the cluster topology, such as changes in Broker IP and newly added nodes, thus supporting "seamless access".

[0068] Step S20: determining the request type of the metadata request, and forwarding the metadata request to a Kafka cluster based on the request type and the cluster service configuration information, so that the Kafka cluster returns response data according to the metadata request;

[0069] The request types of metadata requests are divided into "data production requests" and "data consumption requests", which correspond to different business intentions of the client. Among them, data production requests refer to the client (producer) needing to obtain metadata such as the partition leader node information, number of partitions, and replica distribution of the target topic; data consumption requests refer to the client (consumer) needing to obtain the consumer group coordinator (GroupCoordinator) address, partition leader / replica nodes, and consumer rebalancing (Rebalance) related metadata.

[0070] The Kafka metadata gateway can accurately identify the request type by parsing the request protocol fields or request parameters. The cluster service configuration information obtained from the Kafka management and control platform is used as the data source for forwarding decisions, and metadata requests are forwarded to the Kafka cluster based on the request type and cluster service configuration information. For example, the request type is a data production request sent by the producer client. Due to the requirements of the Kafka write mechanism, the production request must be sent to the leader node of the target partition. Therefore, the Kafka metadata gateway locates the leader node of the partition through the cluster service configuration information and randomly selects a node to forward the metadata request. The client does not need to perceive the specific forwarding logic, but only needs to send a request to the gateway. The gateway automatically completes the node matching and shields the dynamic changes of the underlying cluster (such as node failures and leader switching). The client does not need to modify any configuration.

[0071] Step S30: receiving the response data, encapsulating the response data and sending it to the client, so that the client can write and / or read data according to the response data.

[0072] The response data is the result of the Kafka cluster processing the metadata request. It contains information required by the client, such as the address and port of the broker where the topic resides, and other detailed metadata. The encapsulation process involves the system converting the response data format, encrypting it, and adding necessary header information to ensure that it conforms to the client's expected format and ensures the security and integrity of data transmission. Producer clients and consumer clients are responsible for writing and reading data, respectively.

[0073] When the Kafka metadata gateway forwards the metadata request to the Kafka cluster, the Kafka cluster performs the corresponding query or processing operations based on the request content and encapsulates the results into response data and returns it to the Kafka metadata gateway. The Kafka metadata gateway receives the response data returned by the Kafka cluster and encapsulates the response data to reduce data transmission volume and improve response speed.

[0074] This embodiment provides a cloud service discovery method, in which a Kafka metadata gateway requests cluster service configuration information from a Kafka management platform based on a pre-established communication connection channel, and receives a metadata request sent by a client; determines the request type of the metadata request, and forwards the metadata request to a Kafka cluster based on the request type and the cluster service configuration information, so that the Kafka cluster can return response data based on the metadata request; receives the response data, encapsulates the response data, and sends it to the client, so that the client can write and / or read data based on the response data. First, the Kafka metadata gateway is "disguised" as a server node, processing the client metadata request in an intermediary role, forwarding the metadata request to the real Kafka cluster to obtain the response data, and then returning it to the client, so that the client is unaware of the real cluster and can access it without modification. Furthermore, metadata requests are forwarded based on the request type and cluster configuration, and requests are evenly distributed to different Kafka nodes to avoid overloading a single node, thereby improving the overall throughput and stability of the cluster and achieving load balancing. In summary, this application forwards metadata requests through the Kafka metadata gateway, so that the client does not need to care about the details of the underlying cluster. It relies on the centralized control and forwarding capabilities of the Kafka metadata gateway as an intermediary layer to achieve unified entry and seamless access and load balancing of Kafka services on the cloud.

[0075] In a feasible implementation, step S20 may include steps S21 to S23:

[0076] Step S21, parsing the request header field of the metadata request, and determining the request type of the metadata request according to the parsing result of the request header field, wherein the request type includes producer client metadata request and consumer client metadata request;

[0077] Metadata requests follow the Kafka protocol format, and their structure includes a "Request Header" and a "Request Body." The Kafka metadata gateway quickly determines the request type by parsing specific fields in the request header. Specific fields in the request header include the core api_key field used to identify the request type in the Kafka protocol. Based on the value of api_key, the Kafka metadata gateway clearly divides metadata requests into two categories: producer client metadata requests: used to obtain "write-related metadata," such as whether a topic exists, the address of the partition leader node, the number of partitions, etc. (to support subsequent data production); consumer client metadata requests: used to obtain "consumption-related metadata," such as the consumer group coordinator address, partition allocation scheme, offset submission location, etc. (to support subsequent data consumption).

[0078] It's also important to note that the Kafka metadata gateway employs fault-tolerance mechanisms and data validation measures when parsing request header fields to ensure accuracy and stability. The format of request header fields is strictly validated, and any requests that don't conform to the specifications are returned with an error message and logged for subsequent investigation and resolution.

[0079] Step S22: If the request type is a producer client metadata request, forwarding the producer client metadata request to the Kafka cluster based on the cluster service configuration information, so that the Kafka cluster returns first response data according to the producer client metadata request;

[0080] The first response data refers to the response data returned by the Kafka cluster after receiving the producer client's metadata request. It contains detailed information about the broker where the topic is located based on the request content. The Kafka metadata gateway uses the cluster service configuration information to forward the producer client's metadata request to the Kafka cluster to obtain the first response data returned by the Kafka cluster. It can then select a broker node at will for processing and be directly processed by any broker.

[0081] After the Kafka cluster processes the request, it returns response data containing detailed metadata such as "partition Broker address, replica distribution, ISR list (synchronous replica set)", which is the first response data.

[0082] Step S23: If the request type is a consumer client metadata request, forward the consumer client metadata request to the Kafka cluster based on the cluster service configuration information, so that the Kafka cluster returns second response data according to the consumer client metadata request.

[0083] The second response data refers to the response data containing the consumption-related metadata required by the consumer client, which is queried and returned by the Kafka cluster based on the request content after receiving the metadata request from the consumer client. The forwarding target of consumer metadata requests (such as consumer group coordination and partition allocation) is different from that of producers, and needs to be dynamically selected based on the characteristics of the consumer group and the cluster load status. For example, consumer group coordination requests (such as FindCoordinatorRequest and JoinGroupRequest): forwarded to the GroupCoordinator node responsible for the consumer group in the Kafka cluster. The GroupCoordinator node is determined by the cluster based on the hash of the consumer group ID, and the gateway obtains its address through the cluster configuration information. Message pull requests (such as FetchRequest): can be forwarded to the replica node of the partition (not necessarily the Leader node). The gateway selects a low-load node based on the real-time load of the replica node (such as read request QPS, disk IO) to achieve "read-write separation".

[0084] The Kafka cluster returns second response data containing consumption-related metadata required by the consumer client based on the metadata request of the consumer client.

[0085] In this implementation, the client does not need to distinguish the forwarding logic of producer / consumer requests. The gateway automatically classifies and processes them based on the request header, shielding the node selection details of the underlying cluster and enhancing seamless access.

[0086] Based on the first embodiment of the present application, the second embodiment of the present application is proposed. In the second embodiment of the present application, the same or similar contents as those of the first embodiment can be referred to the above introduction and will not be repeated hereafter.

[0087] On this basis, please refer to Figure 3 , Figure 3 This is a flow chart of the second embodiment of this application. Figure 3 As shown, before step S10, the cloud service discovery method further includes steps S01 to S04:

[0088] Step S01: obtaining a communication configuration parameter set, and initiating a connection request to the Kafka management and control platform based on the communication configuration parameter set, so that the Kafka management and control platform establishes a bidirectional connection according to the connection request;

[0089] The communication configuration parameter set contains various parameters required to establish a communication connection with the Kafka control platform, such as the Kafka control platform's service address, connection timeout, authentication method (such as username, password, and API key), encryption protocol type, and retry strategy. The communication configuration parameter set can be obtained from the gateway's local configuration file (static parameters) or the distributed configuration center. The distributed configuration center supports dynamic parameter updates.

[0090] Based on the address information in the communication configuration parameter set, the Kafka metadata gateway initiates a TCP three-way handshake with the Kafka management and control platform to establish a two-way connection at the basic transport layer. After the connection is established, both the gateway and the management and control platform can actively send data to each other, supporting the subsequent management and control platform to actively push cluster configuration changes, such as node failure notifications.

[0091] Step S02, performing encryption protocol handshake, identity authentication, and application layer protocol version alignment based on the bidirectional connection to generate the communication connection channel;

[0092] Based on the established TCP bidirectional connection, three layers of "enhancement operations" are performed to upgrade the original connection to a secure and reliable application layer communication channel. The three layers of "enhancement operations" include: performing encryption protocol handshake, identity authentication, and application layer protocol version alignment. Specifically:

[0093] The encryption protocol handshake refers to the gateway and the management and control platform negotiating encryption algorithms (such as AES-256) and exchanging certificates through the TLS / SSL protocol, upgrading TCP plaintext transmission to encrypted transmission, and preventing configuration information (such as cluster node IP and authentication keys) from being eavesdropped during transmission.

[0094] Identity authentication refers to the gateway submitting authentication credentials (such as the username and password in the parameter set, SSL client certificate) to the management and control platform. The management and control platform will only allow subsequent communication after verification to prevent illegal gateway access.

[0095] Application layer protocol version alignment means that the gateway and the management and control platform negotiate the application layer communication protocol version to ensure consistency in request / response formats, field definitions, error codes, etc., to avoid configuration information parsing failures due to protocol incompatibility.

[0096] Step S03, responding to the heart detection packet sent by the Kafka management and control platform, and determining the health status of the communication connection channel according to the response result of the heart detection packet;

[0097] Heart detection packets are special data packets sent periodically or irregularly by the Kafka management and control platform to monitor the health of communication channels. Health status provides comprehensive information about whether the communication channel is functioning properly and whether data transmission is stable and reliable.

[0098] The Kafka management and control platform sends heartbeat packets (data packets with an empty payload or a fixed identifier) ​​to the gateway at a fixed interval (e.g., 5 seconds). Upon receiving these packets, the gateway must respond within a timeout threshold (e.g., 2 seconds). The health status is determined based on the response to the heartbeat packets. If N consecutive heartbeat packets (e.g., 3) are responded to normally without timeouts or packet loss, the health status is "available." If the heartbeat packets time out without a response, the response format is incorrect, or the packet loss rate exceeds a preset threshold, the health status is "unavailable."

[0099] Step S04: activating the communication connection channel under the condition that the health status is available.

[0100] Only when the gateway determines that "Health Status = Available" will it enable the communication channel and allow metadata requests and cluster service configuration information to be forwarded. This prevents business traffic (such as cluster service configuration requests) from being directed into unstable channels, ensuring the real-time and accuracy of the obtained cluster service configuration information.

[0101] It is understandable that if you need to implement Figure 2 The interaction between the Kafka metadata gateway, the Kafka management and control platform, and the Kafka cluster shown in the figure can refer to the communication connection channel between the Kafka metadata gateway and the management and control platform to build the communication connection channel between the Kafka metadata gateway, the Kafka cluster, and the client.

[0102] In this embodiment, a "safe, reliable, real-time and controllable" communication connection channel is jointly constructed through parameter configuration, security reinforcement, health detection, and conditional activation, laying the foundation for the realization of upper-layer business functions (load balancing and seamless access).

[0103] Based on the first embodiment and / or the second embodiment of the present application, the third embodiment of the present application is proposed. In the third embodiment of the present application, the same or similar contents as those of the first embodiment and / or the second embodiment can be referred to above and will not be repeated hereafter.

[0104] In this embodiment, the client includes a producer client and a consumer client. The step S30 of receiving the response data, encapsulating the response data and sending it to the client so that the client can write data and / or read data according to the response data includes steps S31 to S34:

[0105] Step S31: receiving the first response data, encapsulating the first response data and sending it to the producer client, so that the producer client initiates creation of a connection to the Kafka cluster according to the first response data, and writes data according to the created connection;

[0106] refer to Figure 4 , Figure 4 The producer client provided for this application is implemented through the Kafka gateway, such as Figure 4 As shown, the Kafka gateway and Kafka cluster periodically and proactively request the Kafka management platform to obtain cluster service configuration information and synchronize security authentication information. Cluster service configuration information includes two core pieces of information: the service address of each Kafka cluster (such as the IP and port of the broker node), which ensures that the gateway knows the real-time cluster topology; and service discovery configuration (such as topic routing rules, load balancing policies, and node health thresholds), which is used for dynamic decision-making when forwarding subsequent requests.

[0107] The producer client first sends a METADATA request (metadata request) to the Kafka gateway to obtain basic information about the target topic (such as the number of partitions, leader node location, replica distribution, etc.). After receiving the request, the gateway forwards the request to the corresponding Kafka cluster based on the synchronized cluster service configuration information. After processing, the cluster returns the first response data containing detailed metadata. The gateway encapsulates the first response data and returns it to the producer. This is the initial handshake between the producer client and the Kafka cluster, which allows the client to clearly understand the target direction of subsequent data writes.

[0108] Furthermore, after the producer client receives the first METADATA response, it initiates another connection creation request to the Kafka metadata gateway based on the Leader node information in the first response data, and sends the METADATA request again after the connection is established. The purpose of the second request is to confirm the real-time status of the cluster (to avoid node failure or partition migration in the cluster after the first query) and to ensure that the obtained Leader node address, ISR list (synchronous replica set) and other information are the latest and valid. After the gateway forwards the second request, the cluster returns the updated METADATA response, and the producer finally determines the target node for data writing based on this.

[0109] After completing metadata confirmation and connection creation, the producer sends business messages (such as user behavior logs and transaction data) directly to the real Kafka cluster for data writing. In order to cope with dynamic changes in the Kafka cluster (such as Leader node switching, partition expansion, and node failure recovery), the producer will periodically (such as every 5 minutes) automatically send METADATA requests to the gateway to refresh the locally cached metadata information. After forwarding the request, the gateway passes the latest status returned by the cluster (such as the new Leader node address and changes in the number of partitions) to the producer, allowing the client to adjust the write target in real time and avoid sending messages to faulty nodes. The "active update" mechanism ensures the continuous availability of the producer when the cluster changes dynamically.

[0110] Step S32: receiving the second response data, encapsulating the second response data and sending it to the consumer client, so that the consumer client can send a consumption coordinator search request according to the second response data;

[0111] refer to Figure 5 , Figure 5 The consumer client provided for this application is implemented through the Kafka gateway, such as Figure 5 As shown in the figure, the Kafka management and control platform serves as the cluster configuration center, storing static information such as cluster topology (Broker list, controller address), service discovery rules (such as Topic routing policy), etc. Therefore, the Kafka metadata gateway and Kafka cluster periodically pull cluster service configuration information from the management and control platform.

[0112] Furthermore, the consumer client first sends a METADATA request to the Kafka gateway to obtain basic information about the target topic (such as the number of partitions, leader node location, replica distribution, etc.). After receiving the request, the gateway forwards the request to the corresponding Kafka cluster based on the synchronized cluster service configuration information. After processing, the cluster returns a second response data containing detailed metadata. The gateway encapsulates the second response data and returns it to the consumer client.

[0113] Furthermore, since each consumer group designates a Broker as the coordinator, responsible for managing the consumer group member status, partition allocation scheme, and the submission and synchronization of consumption offsets, after receiving the second response data, the consumer client sends a FIND_COORDINATOR (consumer group coordinator search) request to the Kafka cluster based on the second response data, which is then forwarded by the Kafka metadata gateway.

[0114] Step S33: Obtain the consumer coordinator search request, and forward the consumer coordinator search request to the Kafka cluster, so that the Kafka cluster returns consumer coordinator information according to the coordinator search request;

[0115] The Kafka metadata gateway forwards the consumer coordinator lookup request sent by the consumer client, and forwards the consumer coordinator lookup request to the Kafka cluster. The Kafka cluster returns the consumer coordinator information based on the coordinator lookup request.

[0116] Step S34: Obtain the consumer coordinator information and forward the consumer coordinator information to the consumer client, so that the consumer client can initiate a consumption request to the Kafka cluster to read data according to the consumer coordinator information.

[0117] The Kafka metadata gateway obtains the consumer coordinator (Group Coordinator) information returned by the Kafka cluster based on the consumer group coordinator search request and forwards it to the consumer client. The consumer client sends a SynGroup (grouping) request to the coordinator based on the consumer coordinator information, synchronizes the consumer group's consumer partition allocation plan, and sends a METADATA request to the Kafka cluster based on the synchronized consumer partition allocation plan. The METADATA request here is a partition metadata request to obtain real-time metadata of the Topic partition for which it is responsible, that is, partition metadata information.

[0118] Based on the real-time partition metadata information obtained, the consumer client sends a FETCH request to the target Broker of the Kafka cluster to pull messages from the assigned partitions, thereby reading data.

[0119] In addition, the consumer client periodically sends Heartbeat requests to the Kafka cluster to keep it alive and submits OffsetCommit requests to save the consumed offset location information.

[0120] In this embodiment, the interaction process between the producer / consumer client and the cluster is uniformly controlled through the Kafka metadata gateway. On the one hand, the gateway encapsulates response data (such as the first response data and the second response data) and forwards key requests (such as the consumer group coordinator search request), so that the client does not need to directly perceive the underlying topology changes of the Kafka cluster, reducing the client's implementation complexity and simplifying configuration management; on the other hand, the gateway, as an intermediate layer, can centrally handle authentication, request filtering and routing distribution, which not only ensures the security of cluster access, but also dynamically optimizes the request forwarding strategy based on the timed synchronization of cluster metadata, thereby improving the system's availability and fault tolerance; at the same time, by separating the producer data writing and consumer coordinator search, consumption request and other processes, the interaction logic is clearer, which facilitates subsequent function expansion and performance tuning, and ultimately builds a set of client-unaware, cluster-highly available, and integrated management and control Kafka metadata access architecture.

[0121] In addition, the present invention provides a cloud service discovery method, which is applied to the client, referring to Figure 6 , Figure 6 A flowchart of the fourth embodiment of the cloud service discovery method provided in this application.

[0122] like Figure 6 As shown, in this embodiment, the cloud service discovery method includes steps A10 to A20:

[0123] Step A10: Send a metadata request to the Kafka metadata gateway, so that the Kafka metadata gateway determines the request type of the metadata request and forwards the metadata request to the Kafka cluster based on the request type and cluster service configuration information. The cluster configuration service information is requested by the Kafka metadata gateway from the Kafka management and control platform based on a pre-established communication connection channel.

[0124] Clients do not need to directly perceive the Kafka cluster address and instead send metadata requests (such as obtaining topic partition information and finding consumer group coordinators) to the Kafka metadata gateway. The Kafka metadata gateway first determines the specific type of request and the business purpose of the request (for example, a producer needs to establish a connection, a consumer needs to join a consumer group).

[0125] Furthermore, the gateway determines which Kafka cluster to forward the request to based on the cluster service configuration information pre-synchronized from the Kafka management and control platform (which is periodically pulled through the "communication connection channel" and includes broker addresses, load balancing policies, permission rules, etc.). The gateway forwards the request to the corresponding Kafka cluster according to the cluster configuration rules, achieving unified access without the client being aware of the cluster topology.

[0126] Step A20, receiving the response data returned by the Kafka metadata gateway, and writing and / or reading data according to the response data. The response data is encapsulated and sent by the Kafka metadata gateway after the Kafka cluster returns it to the Kafka metadata gateway according to the metadata request.

[0127] After receiving a request, the Kafka cluster returns the original response data (such as topic metadata and coordinator address), which is then received by the gateway. The gateway then standardizes and encapsulates the original response data (for example, by standardizing the format, adding routing information, and filtering sensitive fields) to generate "first response data" (for producers) or "second response data" (for consumers) that can be directly parsed by the client.

[0128] The producer client uses the encapsulated "first response data" to identify the target broker address, initiate a connection request, and complete data writing. The consumer client uses the encapsulated "second response data" to initiate a consumer group coordinator lookup request, then forwards the information to the gateway to obtain the coordinator information, and finally initiates a consumption request to the cluster and reads the data.

[0129] This embodiment provides a cloud service discovery method, which "disguises" the Kafka metadata gateway as a server node, processes client metadata requests in an intermediary role, forwards the metadata request to the real Kafka cluster to obtain response data, and then returns it to the client, so that the client is unaware of the real cluster and can access it without modification. Furthermore, metadata requests are forwarded according to the request type and cluster configuration, and requests are evenly distributed to different Kafka nodes to avoid overloading a single node, thereby improving the overall throughput and stability of the cluster and achieving load balancing. In summary, this application forwards metadata requests through the Kafka metadata gateway, so that the client does not need to care about the details of the underlying cluster, and relies on the centralized management and forwarding capabilities of the Kafka metadata gateway as the intermediary layer to achieve unified entrance seamless access and load balancing for Kafka services on the cloud.

[0130] In a feasible implementation, step A20 may include steps A21 to A26:

[0131] Step A21: Create a connection to the Kafka cluster according to the response data, and write data according to the created connection;

[0132] After the producer client receives the first METADATA response data, it initiates another connection creation request to the Kafka metadata gateway based on the Leader node information in the first response data, and sends the METADATA request again after the connection is established. The purpose of the second request is to confirm the real-time status of the cluster (to avoid node failure or partition migration in the cluster after the first query) and to ensure that the obtained Leader node address, ISR list (synchronous replica set) and other information are the latest and valid. After the gateway forwards the second request, the cluster returns the updated METADATA response, and the producer finally determines the target node for data writing based on this.

[0133] After completing metadata confirmation and connection creation, the producer sends business messages (such as user behavior logs and transaction data) directly to the real Kafka cluster for data writing. In order to cope with dynamic changes in the Kafka cluster (such as Leader node switching, partition expansion, and node failure recovery), the producer will periodically (such as every 5 minutes) automatically send METADATA requests to the gateway to refresh the locally cached metadata information. After forwarding the request, the gateway passes the latest status returned by the cluster (such as the new Leader node address and changes in the number of partitions) to the producer, allowing the client to adjust the write target in real time and avoid sending messages to faulty nodes. The "active update" mechanism ensures the continuous availability of the producer when the cluster changes dynamically.

[0134] Step A22: Send a consumer group coordinator lookup request to the Kafka metadata gateway according to the response data, so that the Kafka metadata gateway forwards the consumer group coordinator lookup request to the Kafka cluster to obtain consumer coordinator information;

[0135] The consumer client first sends a METADATA request to the Kafka gateway to obtain basic information about the target topic (such as the number of partitions, leader node location, replica distribution, etc.). After receiving the request, the gateway forwards the request to the corresponding Kafka cluster based on the synchronized cluster service configuration information. The cluster processes the request and returns a second response data containing detailed metadata. The gateway encapsulates the second response data and returns it to the consumer client.

[0136] Furthermore, since each consumer group designates a Broker as the coordinator, responsible for managing the consumer group member status, partition allocation scheme, and the submission and synchronization of consumption offsets, after receiving the second response data, the consumer client sends a FIND_COORDINATOR (consumer group coordinator search) request to the Kafka cluster based on the second response data, which is then forwarded by the Kafka metadata gateway.

[0137] Step A23: Obtain the consumer coordinator information sent by the Kafka metadata gateway, and send a group entry request to the Kafka cluster according to the consumer coordinator information;

[0138] The Kafka metadata gateway obtains the consumer coordinator (Group Coordinator) information returned by the Kafka cluster based on the consumer group coordinator search request and forwards it to the consumer client. The consumer client sends a JoinGroup request to the coordinator based on the consumer coordinator information, declaring to join the consumer group. The coordinator elects a Leader and collects member subscription information.

[0139] Step A24: sending a grouping request to the Kafka cluster based on the grouping request, and synchronizing consumption allocation plan information according to the grouping request;

[0140] After sending a join request, the consumer client selects a leader based on the coordinator and collects member subscription information. After the leader formulates a partition allocation plan, it sends a SyncGroup request to the Kafka cluster to synchronize the consumption allocation plan information.

[0141] Step A25: Sending a partition metadata request to the Kafka cluster based on the consumption allocation scheme information, so that the Kafka cluster returns partition metadata information according to the partition metadata request;

[0142] Based on the consumption allocation plan information, the consumer client sends a partition-level METADATA request to the Kafka cluster, and obtains the partition metadata information returned by the Kafka cluster based on the partition metadata request. The partition metadata information includes the latest leader address of the partition, ISR list and other real-time routing information.

[0143] Step A26: Send a message pull request according to the partition data information, and read data according to the message pull request.

[0144] The consumer client sends a FETCH (message pull) request based on the partition data information, and pulls the message data by directly sending a FETCH request to the Leader Broker of the target partition to realize data reading.

[0145] Based on the fourth embodiment of the present application, a fifth embodiment of the present application is proposed. In the fifth embodiment of the present application, after step A20 of writing and / or reading data according to the response data, steps A30 to A40 are further included:

[0146] Step A30: regularly sending a heartbeat request to the Kafka cluster, so that the Kafka cluster can keep the session alive according to the heartbeat request;

[0147] "Heartbeat request" refers to the status detection signal that the consumer client sends periodically to the Kafka cluster (specifically, the consumer group coordinator); "session keepalive" refers to maintaining the effective connection status between the client and the cluster through continuous heartbeat interaction to avoid being judged as abnormally offline.

[0148] Kafka consumer groups rely on a coordinator to manage member status. If a consumer fails to send a heartbeat for an extended period, the coordinator deems it faulty and triggers a rebalance of the consumer group, leading to partition reallocation and potentially causing duplicate message consumption or interrupted processing. Regularly sending heartbeats ensures the coordinator is aware of client liveness and maintains consumer group stability.

[0149] Therefore, the consumer client periodically sends heartbeat requests to the consumer group coordinator in the Kafka cluster; after receiving the request, the consumer group coordinator updates the "last active time" of the consumer. If the heartbeat is continuously received within the default time, its consumer group membership is maintained to avoid triggering rebalancing.

[0150] Step A40: Submit an offset submission request according to pre-configured submission parameters, and save consumption location information according to the offset submission request.

[0151] An OffsetCommit request is a request from a consumer client to send the location information (Offset) of consumed messages to the cluster. The "consumption location information" is the Offset, which refers to the sequence number of the last message successfully consumed by the consumer in the partition. The "pre-configured commit parameters" include the commit method (automatic / manual) and the commit interval.

[0152] Submitting offsets is used to persist consumption progress. When a consumer restarts or rebalances, it can continue consuming based on the most recently submitted offset, avoiding duplicate processing or message loss. Configuration parameters allow for flexible control over the timing of commits, balancing data consistency and performance.

[0153] Therefore, the consumer client triggers an OffsetCommit request based on pre-set commit parameters: if it is an automatic commit, the current consumption position is committed periodically; if it is a manual commit, an explicit API call is required to commit. The request is sent to the coordinator, and the cluster associates the offset with the consumer group and partition to ensure that the consumption status can be accurately restored after failure recovery.

[0154] In this embodiment, by sending heartbeat requests at regular intervals, the Kafka cluster is ensured to be aware of the survival status of the consumer client in real time, avoiding unnecessary consumer group rebalancing caused by connection timeouts, thereby reducing message processing interruptions caused by partition reallocation and improving the continuity and stability of the consumption process; by submitting offset requests to persist consumer site information, it is ensured that consumers can continue to consume from the most recently submitted Offset after a failure restart or rebalancing, effectively avoiding duplicate message processing or data loss. The two together ensure the high reliability, data consistency and system fault tolerance of the Kafka consumption link.

[0155] For example, to help understand the implementation process of the cloud service discovery method obtained by combining this embodiment with the above embodiments, please refer to Figure 7 , Figure 7 Provides a Kafka cloud platform module interaction diagram, such as Figure 7 As shown in the figure, users perform operations such as instance creation, topic creation, user creation, and ACL management on the Kafka management and control platform. After the operations are completed, some metadata related to the Kafka cluster will be stored on the Kafka management and control platform.

[0156] Furthermore, the Kafka gateway and Kafka cluster will periodically obtain relevant cluster service configuration metadata from the Kafka management platform. The metadata here includes cluster service address, user information, permission information, and dynamic configuration information for service discovery. The deployment architecture of the Kafka gateway is as follows: Figure 8 As shown, Figure 8 This article provides a diagram of the Kafka gateway's deployment architecture. The Kafka gateway architecture uses the global domain name as the entry point, configures polling rules in the DNS service, and evenly distributes requests to the SLB corresponding to the Kafka gateway. The SLB then forwards the requests to the Kafka gateway node. The Kafka gateway supports multi-AZ and multi-node deployment to achieve load balancing and AZ-level fault isolation.

[0157] Furthermore, the client will first request the Kafka gateway to obtain Topic metadata information, etc. The Kafka gateway will forward the request to the Kafka cluster and obtain the metadata information, and then return the metadata information directly to the client.

[0158] Furthermore, after the client obtains the Topic metadata responded by the Kafka cluster, it will produce / consume to the Broker where the Kafka cluster Topic is located. Correspondingly, after receiving the client's production / consumption request, the Kafka cluster will perform read / write permission control and flow control.

[0159] It should be noted that the above examples are only used to understand this application and do not constitute a limitation on the cloud service discovery method of this application. More simple transformations based on this technical concept are all within the scope of protection of this application.

[0160] This application also provides a Kafka cloud service discovery device, please refer to Figure 9 , the Kafka cloud service discovery device includes:

[0161] Receiving module 10, used to request cluster service configuration information from the Kafka management and control platform according to the pre-established communication connection channel, and receive metadata requests sent by the client;

[0162] a forwarding module 20 configured to determine a request type of the metadata request and, based on the request type and the cluster service configuration information, forward the metadata request to a Kafka cluster, so that the Kafka cluster returns response data according to the metadata request;

[0163] The sending module 30 is configured to receive the response data, encapsulate the response data, and send the response data to the client, so that the client can write and / or read data according to the response data.

[0164] The Kafka cloud service discovery device provided in this application adopts the cloud service discovery method in the above-mentioned embodiment to solve the technical problem of Kafka cloud service discovery. Compared with the prior art, the beneficial effects of the Kafka cloud service discovery device provided in this application are the same as the beneficial effects of the cloud service discovery method provided in the above-mentioned embodiment, and the other technical features of the Kafka cloud service discovery device are the same as the features disclosed in the above-mentioned embodiment method, which are not repeated here.

[0165] The present application provides a Kafka cloud service discovery device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the cloud service discovery method in the above-mentioned embodiment one.

[0166] Reference below Figure 10, which shows a schematic diagram of the structure of a Kafka cloud service discovery device suitable for implementing an embodiment of the present application. The Kafka cloud service discovery device in the embodiment of the present application may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), in-vehicle terminals (such as in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 10 The Kafka cloud service discovery device shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of the present application.

[0167] like Figure 10 As shown, the Kafka cloud service discovery device may include a processing device 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes based on programs stored in a read-only memory 1002 or programs loaded from a storage device 1003 into a random access memory 1004. Random access memory 1004 also stores various programs and data required for the operation of the Kafka cloud service discovery device. The processing device 1001, the read-only memory 1002, and the random access memory 1004 are connected to each other via a bus 1005. An input / output interface 1006 is also connected to the bus. Typically, the following systems can be connected to the input / output interface 1006: an input device 1007 including, for example, a touch screen, touchpad, keyboard, mouse, image sensor, microphone, accelerometer, gyroscope, etc.; an output device 1008 including, for example, a liquid crystal display (LCD), speaker, vibrator, etc.; a storage device 1003 including, for example, a magnetic tape, hard disk, etc.; and a communication device 1009. Communication device 1009 can allow the Kafka cloud service discovery device to communicate with other devices wirelessly or wired to exchange data. Although the figure shows a Kafka cloud service discovery device with various systems, it should be understood that it is not required to implement or have all of the systems shown. More or fewer systems can be implemented or provided instead.

[0168] In particular, according to the embodiments disclosed in the present application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, the embodiments disclosed in the present application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program comprising program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network via a communication device, or installed from a storage device 1003, or installed from a read-only memory 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are performed.

[0169] The Kafka cloud service discovery device provided in this application adopts the cloud service discovery method in the above embodiment to solve the technical problems of Kafka cloud service discovery. Compared with the prior art, the beneficial effects of the Kafka cloud service discovery device provided in this application are the same as the beneficial effects of the cloud service discovery method provided in the above embodiment, and the other technical features of the Kafka cloud service discovery device are the same as the features disclosed in the above embodiment method, which are not repeated here.

[0170] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0171] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

[0172] The present application provides a computer-readable storage medium having computer-readable program instructions (ie, a computer program) stored thereon, and the computer-readable program instructions are used to execute the cloud service discovery method in the above-mentioned embodiment.

[0173] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems or devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0174] The above-mentioned computer-readable storage medium may be included in the Kafka cloud service discovery device; or it may exist independently without being assembled into the Kafka cloud service discovery device.

[0175] The above-mentioned computer-readable storage medium carries one or more programs. When the above-mentioned one or more programs are executed by the Kafka cloud service discovery device, the Kafka cloud service discovery device: requests cluster service configuration information from the Kafka management and control platform according to a pre-established communication connection channel, and receives the metadata request sent by the client; determines the request type of the metadata request, and forwards the metadata request to the Kafka cluster based on the request type and the cluster service configuration information, so that the Kafka cluster can return response data according to the metadata request; receives the response data, encapsulates the response data and sends it to the client, so that the client can write and / or read data according to the response data.

[0176] Computer program code for performing the operations of the present application may be written in one or more programming languages, or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on the remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0177] The flow charts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present application. In this regard, each box in the flow chart or block diagram can represent a module, program segment or a part of code, and the module, program segment or a part of code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order than that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented by a dedicated hardware-based system that performs the specified function or operation, or can be implemented by a combination of dedicated hardware and computer instructions.

[0178] The modules described in the embodiments of the present application may be implemented in software or hardware, wherein the name of a module does not necessarily limit the unit itself.

[0179] The computer-readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., a computer program) for executing the aforementioned cloud service discovery method, which can solve the technical problem of Kafka cloud service discovery. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as those of the cloud service discovery method provided in the aforementioned embodiment, and are not further elaborated here.

[0180] The present application also provides a computer program product, including a computer program, which implements the steps of the cloud service discovery method as described above when executed by a processor. The computer program product provided by the present application can solve the technical problem of Kafka cloud service discovery. Compared with the prior art, the beneficial effects of the computer program product provided by the present application are the same as the beneficial effects of the cloud service discovery method provided by the above embodiment, and will not be elaborated here. The above is only a partial embodiment of the present application, and does not limit the patent scope of the present application. All equivalent structural transformations made by using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect application in other related technical fields are included in the patent protection scope of the present application.

Claims

1. A cloud service discovery method, characterized in that: Applied to the Kafka metadata gateway, the cloud service discovery method includes: Requests cluster service configuration information from the Kafka management and control platform through the pre-established communication connection channel, and receives metadata requests sent by the client; Determining a request type of the metadata request, and forwarding the metadata request to a Kafka cluster based on the request type and the cluster service configuration information, so that the Kafka cluster returns response data according to the metadata request; The response data is received, and the response data is encapsulated and sent to the client, so that the client can write data and / or read data according to the response data.

2. The cloud service discovery method according to claim 1, wherein: Before the step of requesting cluster service configuration information from the Kafka management and control platform according to the pre-established communication connection channel and receiving the metadata request sent by the client, the following steps are also included: Obtain a communication configuration parameter set, and initiate a connection request to the Kafka management and control platform based on the communication configuration parameter set, so that the Kafka management and control platform establishes a bidirectional connection according to the connection request; Performing encryption protocol handshake, identity authentication, and application layer protocol version alignment based on the bidirectional connection to generate the communication connection channel; Responding to the heart detection packet sent by the Kafka management and control platform, and determining the health status of the communication connection channel according to the response result of the heart detection packet; Under the condition that the health status is available, the communication connection channel is enabled.

3. The cloud service discovery method according to claim 1, wherein: The step of determining the request type of the metadata request and forwarding the metadata request to the Kafka cluster based on the request type and the cluster service configuration information, so that the Kafka cluster returns response data according to the metadata request, includes: Parsing a request header field of the metadata request, and determining a request type of the metadata request according to a result of parsing the request header field, wherein the request type includes a producer client metadata request and a consumer client metadata request; If the request type is a producer client metadata request, forwarding the producer client metadata request to the Kafka cluster based on the cluster service configuration information, so that the Kafka cluster returns first response data according to the producer client metadata request; If the request type is a consumer client metadata request, the consumer client metadata request is forwarded to the Kafka cluster based on the cluster service configuration information, so that the Kafka cluster returns second response data according to the consumer client metadata request.

4. The cloud service discovery method according to claim 3, wherein: The client includes a producer client and a consumer client. The steps of receiving the response data, encapsulating the response data and sending it to the client so that the client can write data and / or read data according to the response data include: Receive the first response data, encapsulate the first response data and send it to the producer client, so that the producer client can initiate a connection creation request to the Kafka cluster according to the first response data, and write data according to the connection creation request; and / or receiving the second response data, encapsulating the second response data and sending it to the consumer client, so that the consumer client can send a consumer group coordinator search request according to the second response data; Obtain the consumer group coordinator search request, and forward the consumer group coordinator search request to the Kafka cluster, so that the Kafka cluster returns consumer coordinator information according to the consumer group coordinator search request; The consumer coordinator information is obtained and forwarded to the consumer client, so that the consumer client can initiate a consumption request to the Kafka cluster to read data according to the consumer coordinator information.

5. A cloud service discovery method, characterized in that: Applied to the client, the cloud service discovery method includes: Sending a metadata request to the Kafka metadata gateway, so that the Kafka metadata gateway determines the request type of the metadata request and forwards the metadata request to the Kafka cluster based on the request type and cluster service configuration information, where the cluster configuration service information is requested by the Kafka metadata gateway from the Kafka management and control platform via a pre-established communication connection channel; Receive the response data returned by the Kafka metadata gateway, and write and / or read data according to the response data, wherein the response data is encapsulated and sent by the Kafka metadata gateway after the Kafka cluster returns it to the Kafka metadata gateway according to the metadata request.

6. The cloud service discovery method according to claim 5, wherein: The step of writing data and / or reading data according to the response data includes: Initiate creation of a connection to the Kafka cluster according to the response data, and write data according to the created connection; and / or Sending a consumer group coordinator lookup request to the Kafka metadata gateway according to the response data, so that the Kafka metadata gateway forwards the consumer group coordinator lookup request to the Kafka cluster to obtain consumer coordinator information; Obtain the consumer coordinator information sent by the Kafka metadata gateway, and send a group entry request to the Kafka cluster according to the consumer coordinator information; Sending a grouping request to the Kafka cluster based on the grouping request, and synchronizing consumption allocation plan information according to the grouping request; Sending a partition metadata request to the Kafka cluster based on the consumption allocation scheme information, so that the Kafka cluster returns partition metadata information according to the partition metadata request; A message pull request is sent according to the partition data information, and data is read according to the message pull request.

7. The cloud service discovery method according to claim 5, wherein: After the step of writing data and / or reading data according to the response data, the following steps are further included: Sending a heartbeat request to the Kafka cluster regularly, so that the Kafka cluster can keep the session alive according to the heartbeat request; Submit an offset submission request according to pre-configured submission parameters, and save consumption location information according to the offset submission request.

8. A Kafka cloud service discovery device, characterized in that: The device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the cloud service discovery method according to any one of claims 1 to 7.

9. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the cloud service discovery method according to any one of claims 1 to 7 are implemented.

10. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the steps of the cloud service discovery method according to any one of claims 1 to 7 are implemented.

Images