Message publishing method and device, electronic equipment and computer readable storage medium

Abstract

The invention discloses a message publishing method and device, electronic equipment and a computer readable storage medium. The method is executed by a message producer of a Kafka cluster, and comprises the following steps: when a message fails to be sent in a target message partition, updating an available message partition of the message; determining whether a new target message partition exists in the updated available message partition according to a preset partition strategy; if the new target message partition exists, sending the message to the new target message partition; and if the new target message partition does not exist, stopping sending the message to the Kafka cluster. According to the technical scheme provided by the invention, the message producer is encapsulated, and the encapsulated message producer is utilized to update the available message partition of the message, so that the message is ensured to be retry on the available message partition, and the transmission success probability of the in-transit message is increased.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular, to a message publishing method, apparatus, electronic device, and computer-readable storage medium. Background Art

[0002] The Kafka system is an open-source stream processing platform developed by the Apache Software Foundation. It is a high-throughput distributed publish-subscribe messaging system. The Kafka partitioning mechanism is the secret weapon for Kafka to achieve high availability and high throughput. A Kafka cluster can have multiple topics, each topic has multiple message partitions, and each message partition can be distributed to different servers. Kafka allows servers in the cluster to fail. When some servers fail, messages can be retried to other available servers within the message partition. When all servers of a certain message partition in the Kafka cluster fail, the in-transit messages of this message partition will be sent failed, affecting business continuity.

[0003] In the prior art, the message publishing process is generally as follows: First, calculate the target message partition for this message sending, send the message to the primary server node of the target message partition. If the message sending fails, continue to send the message to the original target message partition according to the configured number of retries. If the message still fails after retrying, determine that all servers of this message partition have failed, and feedback a notice of failure of this message sending to the client.

[0004] It is found that the prior art calculates the target message partition according to the partition information in the local cache. If the target message partition fails before calculating the target message partition, it does not affect this message sending. If the target message partition fails after calculating the target message partition but before sending the message, the in-transit message of this time will be sent failed; or if the actual message partition has failed and the local cache has not been updated, it will also cause the calculated target message partition to have failed, and the message sending of this time will fail. Summary of the Invention

[0005] The purpose of the embodiments of this application aims to at least solve one of the above technical defects, and particularly provides a message publishing method, apparatus, electronic device, and computer-readable storage medium to improve the success rate of message sending.

[0006] The embodiments of this application adopt the following technical solutions:

[0007] In a first aspect, the embodiments of this application provide a message publishing method, which is executed by a message producer of a Kafka cluster. The message publishing method includes:

[0008] When a message fails to be sent in the target message partition, update the available message partitions of the message;

[0009] Determine whether there is a new target message partition in the updated available message partitions according to the preset partitioning strategy;

[0010] If there is a new target message partition, send the message to the new target message partition;

[0011] If there is no new target message partition, stop sending the message to the Kafka cluster.

[0012] Optionally, when the message fails to be sent in the target message partition, updating the available message partitions of the message includes:

[0013] Obtain the metadata update status cached locally;

[0014] If the locally cached metadata has been updated, obtain the updated available message partitions through the updated metadata;

[0015] If the locally cached metadata has not been updated, mark the original target message partition as a faulty message partition in the unupdated metadata, and update the available message partitions of the message according to the faulty message partition.

[0016] Optionally, if the locally cached metadata has not been updated, marking the original target message partition as a faulty message partition in the unupdated metadata and updating the available message partitions of the message according to the faulty message partition includes:

[0017] Obtain the available Topic partitions in the metadata that are not marked as faulty message partitions, and use the available Topic partitions as the updated available message partitions.

[0018] Optionally, if the locally cached metadata has been updated, obtaining the updated available message partitions through the updated metadata includes:

[0019] Obtain the available Topic partitions of the message according to the updated metadata, and use the available Topic partitions as the updated available message partitions.

[0020] Optionally, the preset partitioning strategy indicates the message type and the preset partitioning calculation algorithm. Determining whether there is a new target message partition in the updated available message partitions according to the preset partitioning strategy includes:

[0021] Obtain the message type of the message, and determine whether there is a reference message partition corresponding to the message type in the available message partitions;

[0022] If there is no reference message partition corresponding to the message type, determine that there is no new target message partition in the updated available message partitions;

[0023] If there is a reference message partition corresponding to the message type, determine that there is a new target message partition in the updated available message partitions, and obtain the new target message partition from the reference message partitions according to the preset partition calculation algorithm.

[0024] Optionally, determining whether there is a reference message partition corresponding to the message type in the available message partitions includes:

[0025] Determine whether the message specifies a message partition according to the message type;

[0026] If the message partition is not specified, determine that there is a reference message partition corresponding to the message type in the available message partitions, and use the available message partitions as the reference message partitions;

[0027] If the message partition is specified, determine whether the specified message partition is included in the available message partitions. When the specified message partition is included in the available message partitions, determine that there is a reference message partition corresponding to the message type in the available message partitions, and use the specified message partition as the reference message partition; when the specified message partition is not included in the available message partitions, determine that there is no reference message partition corresponding to the message type in the available message partitions.

[0028] Optionally, if there is a new target message partition, sending the message to the new target message partition includes:

[0029] Obtain whether the message is sent successfully;

[0030] When the message sending fails, continue to send the message to the new target message partition according to the configured message retry policy;

[0031] If it still fails to send, determine that the message fails to be sent in the new target message partition;

[0032] If the sending is successful, determine that the message is successfully published.

[0033] In a second aspect, an embodiment of the present application further provides a message publishing device, which is applied to a message producer of a Kafka cluster. The message publishing device includes:

[0034] A partition update unit, configured to update the available message partitions of the message when the message fails to be sent in the target message partition;

[0035] A partition determination unit, configured to determine whether there is a new target message partition in the updated available message partitions according to a preset partition policy;

[0036] A message sending unit, configured to send the message to the new target message partition if there is a new target message partition;

[0037] A message processing unit, configured to stop sending the message to the Kafka cluster if there is no new target message partition.

[0038] Thirdly, an embodiment of the present application further provides an electronic device, which includes:

[0039] A memory, storing computer-executable instructions;

[0040] A processor, when the computer-executable instructions are executed, causing the processor to execute the message publishing method.

[0041] Fourthly, an embodiment of the present application further provides a computer-readable storage medium, which stores one or more programs. When the one or more programs are executed by an electronic device including a plurality of application programs, the electronic device is caused to execute the message publishing method.

[0042] The above at least one technical solution adopted in the embodiment of the present application can achieve the following beneficial effects:

[0043] When the message sending fails in the target message partition, the message producer in the embodiment of the present application updates the available message partitions of the message, and determines whether there is a new target message partition in the updated available message partitions according to the preset partition policy. When there is a new target message partition, the message is sent through the new target message partition, ensuring that the message is retried on the available message partitions, increasing the sending success probability of the in-transit message. When there is no new target message partition, the message sending is stopped to avoid unlimited retries and entering an infinite loop. Description of the Drawings

[0044] The drawings described herein are used to provide a further understanding of the present application, and constitute a part of the present application. The illustrative embodiments and descriptions thereof of the present application are used to explain the present application, and do not constitute an improper limitation to the present application. In the drawings:

[0045] Figure 1 Is a flowchart of a message publishing method shown in an embodiment of the present application;

[0046] Figure 2 Is a schematic diagram of metadata shown in an embodiment of the present application;

[0047] Figure 3 Is a schematic structural diagram of a message publishing device shown in an embodiment of the present application;

[0048] Figure 4 Is a schematic structural diagram of an electronic device shown in an embodiment of the present application. Detailed implementation manners

[0049] To make the objectives, technical solutions and advantages of the present application clearer, the technical solutions of the present application will be clearly and completely described below in conjunction with specific embodiments of the present application and the corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present application.

[0050] The following will, with reference to the drawings, detail the technical solutions provided by each embodiment of the present application.

[0051] An embodiment of the present application provides a message publishing method, which is executed by a message producer of a Kafka cluster. The message producer runs on a client that publishes messages to the Kafka cluster. The message producer in the embodiment of the present application is encapsulated, and the messages of the client are published to the Kafka cluster by the encapsulated message producer according to a multi-round message resending strategy, thereby improving the success rate of message publishing.

[0052] As Figure 1 shown, a flowchart of a message publishing method in an embodiment of the present application is provided. The method at least includes the following steps S110 to S140:

[0053] Step S110: When a message fails to be sent in a target message partition, update the available message partitions of the message.

[0054] In the prior art, when a message producer receives a message publishing request from a client, it calculates the target message partition for this message sending according to the message publishing request, and sends the message to the target message partition. If the message fails to be sent for the first time, and according to the message retry strategy configured by the producer, it continues to send the message to the original target message partition. If it still fails to be sent, it is determined that the message fails to be sent in the target message partition.

[0055] It can be seen that the message producer in the prior art actually sends a round of messages to the Kafka cluster. If this round of message sending fails, the message producer directly generates a message publishing failure notice and feeds it back to the client.

[0056] As mentioned above, in one scenario, if the target message partition fails after the target message partition is calculated, then only one round of message sending will cause all in-transit messages in the target message partition to fail to be sent. In another scenario, when the message partition has failed, but the locally cached metadata has not been updated, resulting in the calculated target message partition being a faulty message partition, only one round of message sending will also cause the in-transit messages in the target message partition to fail to be sent.

[0057] In view of the above problems, the embodiments of the present application encapsulate the message producer. When the message sending in the target message partition fails, the encapsulated message producer updates the available message partitions of the message, so as to remove the faulty message partitions from the available message partitions to a certain extent, and obtain new target message partitions from the updated available message partitions for the next round of message sending. In this way, through the update of the available message partitions and multiple rounds of message retransmission, the messages in the above two scenarios can be successfully sent, improving the publishing success rate of such messages.

[0058] It can be seen from this that the multiple rounds of message retransmission in the embodiments of the present application are different from the "message retry policy" in the prior art. The message retry policy in the prior art is to repeatedly send the message to the original target message partition multiple times according to the configured number of retries after calculating the target message partition, while the multiple rounds of message retransmission in the embodiments of the present application mean that the message should be retransmitted multiple times in the available message partitions next, rather than repeatedly sending in the faulty message partitions.

[0059] Step S120: Determine whether there is a new target message partition in the updated available message partitions according to the preset partitioning policy.

[0060] After obtaining the available message partitions of the message, the embodiments of the present application judge whether there is a new target message partition in the available message partitions according to a certain policy to accurately obtain the new target message partition.

[0061] For example, the embodiments of the present application can obtain a new target message partition for message retransmission through a combined policy of the message type and the set partitioning calculation algorithm, so that the new target message partition meets the message sending conditions of various types, and the message publishing method of the embodiments of the present application is applicable to various scenarios.

[0062] When the embodiments of the present application obtain a new target message partition, the message is retransmitted through the new target message partition to ensure that the message is retried on the available message partitions rather than the faulty message partitions, increasing the sending success probability of the in-transit messages. If no new target message partition is obtained, the message sending is stopped to avoid unlimited retries and entering an infinite loop.

[0063] Step S130: If there is a new target message partition, send the message to the new target message partition.

[0064] Step S140: If there is no new target message partition, stop sending the message to the Kafka cluster.

[0065] Such as Figure 1It can be seen from the message publishing method shown that when the message fails to be sent in the target message partition, the message producer of the embodiment of the present application updates the available message partition of the message, and determines whether there is a new target message partition after the updated available message partition according to the preset partition strategy. When there is a new target message partition, the message is sent through the new target message partition to ensure that the message is retried on the available message partition, thereby increasing the probability of successful sending of in-transit messages. When there is no new target message partition, the message sending is stopped to avoid unlimited retries and entering an infinite loop.

[0066] In some embodiments of the present application, the above step S110 determines that when the message fails to be sent in the target message partition, the available message partition of the message is updated, specifically including:

[0067] Get the metadata update status of the local cache;

[0068] If the locally cached metadata has been updated, the updated available message partition is obtained through the updated metadata;

[0069] If the metadata cached locally is not updated, the original target message partition is marked as a fault message partition in the metadata that has not been updated, and the available message partition of the message is updated according to the fault message partition.

[0070] The purpose of updating the available message partition in the embodiment of the present application is to ensure that the message is effectively retried on the available message partition rather than the fault message partition. Therefore, the embodiment of the present application should ensure that there is no fault message partition in the available message partition as much as possible.

[0071] In the embodiment of the present application, the fault message partition can be obtained in two ways. The first is to obtain the fault message partition through the latest metadata periodically pulled by the message producer from the Kafka cluster. The newly pulled metadata is closer to the current survival status of the server node in the Kafka cluster than the previously pulled metadata. Therefore, when the embodiment of the present application determines that the metadata of the local cache has been updated, it is preferred to obtain the updated available message partition through the updated metadata. When the metadata of the local cache is not updated, the fault message partition is obtained in combination with the target message partition where the message fails to be sent, so as to update the available message partition of the message through the fault message partition.

[0072] In some application scenarios, the following steps are used to obtain the metadata update status of the local cache:

[0073] Get the metadata version information when calculating the original target message partition, as well as the current metadata version information;

[0074] If the version information of the two is different, it is determined that the metadata of the local cache has been updated, otherwise it is determined that the metadata of the local cache has not been updated.

[0075] In some possible implementation solutions of this application scenario, the encapsulated message producer updates the locally cached metadata through the following steps:

[0076] Generate a metadata pull request and send the metadata pull request to the Kafka cluster to obtain the request response of the metadata pull request, where the request response indicates that the metadata pull is successful or the metadata fails;

[0077] If the request response indicates that the metadata pull is successful, update the locally cached metadata with the newly pulled metadata;

[0078] If the request response indicates that the metadata pull fails, do not update the locally cached metadata.

[0079] For example, assume that the Kafka cluster is deployed through ZooKeeper, and ZooKeeper manages the Kafka cluster. The management content includes obtaining metadata regularly. If ZooKeeper's locally cached metadata is still the previous version when receiving the metadata pull request, that is, the previous version of the metadata cached by ZooKeeper has not reached the expiration time, at this time, the request response sent by ZooKeeper to the message producer indicates that the metadata pull fails.

[0080] In this embodiment, the message producer can generate a metadata pull request regularly or generate a metadata pull request when it is determined that the message sending of the target message partition fails.

[0081] It can be seen that the "locally cached metadata has been updated" described in the embodiments of this application means that the original target message partition is obtained based on the metadata at time T1, and at time T2 when the message sending of the original target message partition fails, the locally cached metadata has been updated at least once, and the metadata versions before and after the update are different.

[0082] In some possible implementation solutions of this embodiment, the updated available message partitions are obtained through the updated metadata, specifically including:

[0083] Obtain the available Topic partitions of the message according to the updated metadata, and use the available Topic partitions as the updated available message partitions.

[0084] The metadata in the embodiments of this application includes relevant data such as message partition information, node survival information, node address information, and backup information. The available Topic partitions can be obtained through the message partition information and node survival information of the metadata. At this time, all available Topic partitions are used as the updated available message partitions.

[0085] Take Figure 2 as an example. There are six server nodes in the Kafka cluster. A certain Topic is deployed according to six message partitions and three backup nodes. Assume Figure 2 the metadata shown is the updated metadata. If all the server nodes with server IDs 1, 2, and 3 are faulty according to the node survival information, it is determined that message partition 0 is faulty. At this time, message partitions 1, 2, 3, 4, and 5 all include surviving servers. These five message partitions are the available Topic partitions. At this time, these five message partitions are used as the updated available message partitions.

[0086] In some other possible implementation solutions of this embodiment, updating the available message partitions of the messages according to the faulty message partitions described above specifically includes:

[0087] Obtain the available Topic partitions in the metadata that are not marked as faulty message partitions, and use the available Topic partitions as the updated available message partitions.

[0088] Still take Figure 2 as an example. Assume that the locally cached metadata has not been updated. If Figure 2 all six message partitions are available Topic partitions, and the original target message partitions 0 and 2 are marked as faulty message partitions, then the available Topic partitions 1, 3, 4, and 5 are used as the updated available message partitions.

[0089] It should be noted that the embodiment of the present application shows an implementation solution for determining whether there are available message partitions in the Kafka cluster based on the marking information. In other embodiments of the present application, other solutions can also be used to implement the determination process of the available message partitions. For example, delete the faulty message partitions from the available Topic partitions of the metadata, and use the remaining available Topic partitions in the metadata after the deletion process as the updated available message partitions. If there are no available Topic partitions in the metadata after the deletion process, it is determined that there are no available message partitions for the message. At this time, stop sending messages to the Kafka cluster and send a message sending failure notification to the client.

[0090] In some embodiments of the present application, the preset partitioning policy indicates the message type and the preset partitioning calculation algorithm. The above step S120 determines whether there is a new target message partition in the updated available message partitions according to the preset partitioning policy, specifically including:

[0091] Obtain the message type of the message, and determine whether there is a reference message partition corresponding to the message type in the available message partitions;

[0092] If there is no reference message partition corresponding to the message type, it is determined that there is no new target message partition in the updated available message partitions;

[0093] If there is a reference message partition corresponding to the message type, it is determined that there is a new target message partition in the updated available message partitions, and a new target message partition is obtained from the reference message partition according to the preset partition calculation algorithm.

[0094] In some application scenarios of this embodiment, determining whether there is a reference message partition corresponding to the message type in the available message partitions specifically includes:

[0095] Determining whether the message specifies a message partition according to the message type;

[0096] If the message does not specify a message partition, it is determined that there is a reference message partition corresponding to the message type in the available message partitions, and the available message partition is used as the reference message partition;

[0097] If the message specifies a message partition, it is determined whether the available message partitions include the specified message partition. When the available message partitions include the specified message partition, it is determined that there is a reference message partition corresponding to the message type in the available message partitions, and the specified message partition is used as the reference message partition; when the available message partitions do not include the specified message partition, it is determined that there is no reference message partition corresponding to the message type in the available message partitions.

[0098] The "whether the message specifies a message partition" described in the embodiments of this application means whether the client specifies that the message needs to be sent to a specific message partition when generating a message publishing request. Assuming that it is specified to be sent to the first message partition, the message can only be sent to the first message partition and cannot be sent to other message partitions; if it is not specified to be sent to a specific message partition, the message can be sent to any / any number of message partitions under the message Topic.

[0099] Still taking Figure 2Taking the following as an example, assume that the message is for an unspecified message partition, and the updated available message partitions include message partitions 1, 2, 3, and 4. At this time, these four message partitions are determined as the reference message partitions. Assume that the specified message partition of the message is 1, and the updated available message partitions include message partitions 1, 2, 3, and 4. At this time, since the updated available message partitions include the specified message partition, message partition 1 is used as the reference message partition. Assume that the specified message partition of the message is 1, and the updated available message partitions include message partitions 0, 2, 3, and 4. At this time, since the updated available message partitions do not include the specified message partition and it is determined that the updated available message partitions do not include the reference message partition, the message producer in this embodiment stops sending messages to the Kafka cluster and feeds back a message sending failure notice to the client.

[0100] In some other application scenarios of this embodiment, obtaining a new target message partition from the reference message partitions according to the preset partition calculation algorithm specifically includes:

[0101] Obtain the preset Key value of the message;

[0102] Obtain the hash value of the preset Key value according to the hash algorithm;

[0103] Perform modulo operation on the number of reference message partitions according to the hash value, and obtain a new target message partition according to the modulo result.

[0104] In some embodiments of the present application, if there is a new target message partition in the above step S130, sending the message to the new target message partition specifically includes:

[0105] Obtain whether the message is sent successfully;

[0106] When the message sending fails, continue to send the message to the new target message partition according to the configured message retry policy;

[0107] If it still fails to send, it is determined that the message fails to be sent in the new target message partition;

[0108] If the sending is successful, it is determined that the message is successfully published.

[0109] In this embodiment, when obtaining a new target message partition, if the first message sending to the new target message partition fails, continue to send the message to the new target message partition according to the configured message retry policy to avoid message sending failure caused by accidental factors such as network jitter, and improve the probability of successful message sending through the message retry policy.

[0110] It can be seen from the above embodiments of the present application that the message publishing method of the embodiments of the present application has at least the following advantages:

[0111] First, after the message producer encapsulated in the embodiment of the present application fails to send a message to a partition, it updates the available message partitions of the message to determine a new target message partition in the available message partitions for message resending. Through multiple rounds of message resending, the probability of successful message sending is increased.

[0112] Second, after each round of message resending by the message producer encapsulated in the embodiment of the present application, it either updates the available message partitions with the failed message partitions to ensure that the message is resent on the available message partitions next, increasing the probability of successful message sending in the next round; or updates the available message partitions with the updated metadata, increasing the probability of successful sending of the in-transit message.

[0113] Third, when there is no new target message partition in the updated available message partitions, the message producer encapsulated in the embodiment of the present application stops sending messages to the Kafka cluster to avoid unlimited resending and entering an infinite loop.

[0114] The embodiment of the present application also provides a message publishing device, which is applied to a message producer. As Figure 3 shown, a schematic structural diagram of a message publishing device in the embodiment of the present application is provided. The device 300 includes a first determination unit 310, a second determination unit 320, a first control unit 330, and a second control unit 340;

[0115] The partition update unit 310 is configured to update the available message partitions of the message when the message fails to be sent to the target message partition;

[0116] The partition determination unit 320 is configured to determine whether there is a new target message partition in the updated available message partitions according to a preset partition policy;

[0117] The message sending unit 330 is configured to send the message to the new target message partition if there is a new target message partition;

[0118] The message processing unit 340 is configured to stop sending the message to the Kafka cluster if there is no new target message partition.

[0119] In some embodiments of the present application, the partition update unit 310 is specifically configured to obtain the update situation of the metadata cached locally; if the metadata cached locally has been updated, obtain the updated available message partitions through the updated metadata; if the metadata cached locally has not been updated, mark the original target message partition as a failed message partition in the unupdated metadata, and update the available message partitions of the message according to the failed message partition.

[0120] In some embodiments of the present application, the partition update unit 310 includes a first update module;

[0121] A first update module, configured to obtain available Topic partitions in the metadata that are not marked as faulty message partitions, and use the available Topic partitions as updated available message partitions.

[0122] In some embodiments of the present application, the partition update unit 310 further includes a second update module;

[0123] The second update module is configured to obtain available Topic partitions of the message according to the updated metadata, and use the available Topic partitions as updated available message partitions.

[0124] In some embodiments of the present application, the preset partition policy indicates a message type and a preset partition calculation algorithm. The partition determination unit 320 is specifically configured to obtain the message type of the message, and determine whether there is a reference message partition corresponding to the message type in the available message partitions; if there is no reference message partition corresponding to the message type, it is determined that there is no new target message partition in the updated available message partitions; if there is a reference message partition corresponding to the message type, it is determined that there is a new target message partition in the updated available message partitions, and a new target message partition is obtained from the reference message partitions according to the preset partition calculation algorithm.

[0125] In some embodiments of the present application, the partition determination unit 320 is further specifically configured to determine whether the message specifies a message partition according to the message type;

[0126] If the message partition is not specified, it is determined that there is a reference message partition corresponding to the message type in the available message partitions, and the available message partitions are used as the reference message partitions; if the message partition is specified, it is determined whether the specified message partition is included in the available message partitions. When the specified message partition is included in the available message partitions, it is determined that there is a reference message partition corresponding to the message type in the available message partitions, and the specified message partition is used as the reference message partitions; when the specified message partition is not included in the available message partitions, it is determined that there is no reference message partition corresponding to the message type in the available message partitions.

[0127] In some embodiments of the present application, the message sending unit 330 is specifically configured to obtain whether the message is sent successfully;

[0128] When the message sending fails, the message is continuously sent to the new target message partition according to the configured message retry policy; if it still fails to be sent, it is determined that the message fails to be sent in the new target message partition; if it is sent successfully, it is determined that the message is successfully published.

[0129] It can be understood that the above message publishing device can implement each step of the message publishing method provided in the foregoing embodiments. The relevant explanations regarding the message publishing method are applicable to the message publishing device and will not be elaborated here.

[0130] Figure 4 This is a schematic structural diagram of an electronic device shown in an embodiment of the present application. Please refer to Figure 4 , at the hardware level, the electronic device includes a processor, and optionally also includes an internal bus, a network interface, and a memory. Among them, the memory may include a memory, such as a high-speed random access memory (Random-Access Memory, RAM), and may also include a non-volatile memory, such as at least one disk memory, etc. Of course, the cross-border payment system may also include other hardware required for other services.

[0131] The processor, network interface, and memory can be interconnected through an internal bus. The internal bus can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus, or an EISA (Extended Industry Standard Architecture) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For the sake of simplicity of representation, Figure 4 only a bidirectional arrow is used in [the figure], but it does not mean that there is only one bus or one type of bus.

[0132] The memory is used to store programs. Specifically, the program may include program code, and the program code includes computer operation instructions. The memory can include a memory and a non-volatile memory, and provide instructions and data to the processor.

[0133] The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it, forming a message publishing device at the logical level. The processor executes the program stored in the memory and executes the message publishing method, and specifically executes the following steps:

[0134] When the message fails to be sent in the target message partition, update the available message partitions of the message;

[0135] Determine whether there is a new target message partition in the updated available message partitions according to the preset partition policy;

[0136] If there is a new target message partition, send the message to the new target message partition;

[0137] If there is no new target message partition, stop sending the message to the Kafka cluster.

[0138] The above as in this application Figure 1 The method executed by the message publishing device disclosed in the embodiments shown above can be applied to or implemented by a processor. The processor may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the integrated logic circuit in the hardware of the processor or by instructions in the form of software. The above processor may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. It can implement or execute the various methods, steps, and logic block diagrams disclosed in the embodiments of the present application. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc. The steps of the method disclosed in combination with the embodiments of the present application can be directly embodied as being executed and completed by a hardware decoding processor, or by a combination of hardware and software modules in the decoding processor. The software module may be located in a mature storage medium in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. This storage medium is located in the memory, and the processor reads the information in the memory and combines its hardware to complete the steps of the above message publishing method.

[0139] The electronic device can also execute Figure 1 the method executed by the message publishing device in Figure 1 and implement the functions of the message publishing device in the embodiments shown above. The embodiments of the present application will not be elaborated here.

[0140] The embodiments of the present application also propose a computer-readable storage medium. The computer-readable storage medium stores one or more programs. The one or more programs include instructions that, when executed by an electronic device including a plurality of application programs, can enable the electronic device to execute Figure 1 the method executed by the message publishing device in the embodiments shown above, and specifically execute the following steps:

[0141] When the message fails to be sent in the target message partition, update the available message partitions of the message;

[0142] Determine whether there is a new target message partition in the updated available message partitions according to the preset partitioning strategy;

[0143] If there is a new target message partition, send the message to the new target message partition;

[0144] If there is no new target message partition, stop sending the message to the Kafka cluster.

[0145] Those skilled in the art should understand that the embodiments of the present invention can be provided as a method, a system, or a computer program product. Therefore, the present invention can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present invention can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk memory, CD-ROM, optical memory, etc.) containing computer-usable program code.

[0146] The present invention is described with reference to the flowcharts and / or block diagrams of methods, apparatuses (systems), and computer program products according to the embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, and the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to generate a machine, such that the instructions executed by the processor of the computer or other programmable data processing devices generate means for implementing the functions specified in Figure 1 one process or multiple processes and / or blocks Figure 1 one block or multiple blocks.

[0147] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, such that the instructions stored in the computer-readable memory generate a manufactured article including instruction means that implement the functions specified in Figure 1 one process or multiple processes and / or blocks Figure 1 one block or multiple blocks.

[0148] These computer program instructions can also be loaded onto a computer or other programmable data processing device, such that a series of operation steps are executed on the computer or other programmable device to generate a computer-implemented process, so that the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in Figure 1 one process or multiple processes and / or blocks Figure 1 one block or multiple blocks.

[0149] In a typical configuration, a computing device includes one or more processors (CPUs), an input / output interface, a network interface, and memory.

[0150] The memory may include non-permanent memory in the form of computer-readable media, random access memory (RAM), and / or non-volatile memory such as read-only memory (ROM) or flash memory (flash RAM). Memory is an example of computer-readable media.

[0151] Computer-readable media includes both permanent and non-permanent, removable and non-removable media implemented by any method or technology for storing information. The information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices, or any other non-transitory media that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transitory media such as modulated data signals and carrier waves.

[0152] It should also be noted that the term "comprising", "including" or any other variation thereof is intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but also other elements not expressly listed or elements inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the statement "comprising a..." does not exclude the presence of additional identical elements in the process, method, article, or apparatus that comprises the element.

[0153] Those skilled in the art will appreciate that the embodiments of the present application may be provided as a method, system, or computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0154] The above are only embodiments of the present application and are not intended to limit the present application. For those skilled in the art, various changes and modifications can be made to the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included within the scope of the claims of the present application.

Claims

1. A message publishing method, which is executed by a message producer of a Kafka cluster, wherein, The described message publishing method includes: When a message fails to be sent in the target message partition, update the available message partitions of the message; Determine whether there is a new target message partition in the updated available message partitions according to a preset partition policy; If there is a new target message partition, send the message to the new target message partition; If there is no new target message partition, stop sending the message to the Kafka cluster.

2. The message publishing method according to claim 1, wherein, The step of "When a message fails to be sent in the target message partition, update the available message partitions of the message" includes: Obtain the metadata update status cached locally; If the locally cached metadata has been updated, obtain the updated available message partitions through the updated metadata; If the locally cached metadata has not been updated, mark the original target message partition as a faulty message partition in the unupdated metadata, and update the available message partitions of the message according to the faulty message partition.

3. The message publishing method according to claim 2, wherein, The step of "If the locally cached metadata has not been updated, mark the original target message partition as a faulty message partition in the unupdated metadata, and update the available message partitions of the message according to the faulty message partition" includes: Obtain the available Topic partitions in the metadata that are not marked as faulty message partitions, and use the available Topic partitions as the updated available message partitions.

4. The message publishing method according to claim 2, wherein, The step of "If the locally cached metadata has been updated, obtain the updated available message partitions through the updated metadata" includes: Obtain the available Topic partitions of the message according to the updated metadata, and use the available Topic partitions as the updated available message partitions.

5. The message publishing method according to claim 1, wherein The preset partition policy includes a message type and a preset partition calculation algorithm. The step of "Determine whether there is a new target message partition in the updated available message partitions according to a preset partition policy" includes: Obtain the message type of the message, and determine whether there is a reference message partition corresponding to the message type in the available message partitions; If there is no reference message partition corresponding to the message type, determine that there is no new target message partition in the updated available message partitions; If there is a reference message partition corresponding to the message type, determine that there is a new target message partition in the updated available message partitions, and obtain the new target message partition from the reference message partitions according to the preset partition calculation algorithm.

6. The message publishing method according to claim 5, wherein, The step of "Determine whether there is a reference message partition corresponding to the message type in the available message partitions" includes: Determine whether the message specifies a message partition according to the message type; If the message does not specify a message partition, determine that there is a reference message partition corresponding to the message type in the available message partitions, and use the available message partitions as the reference message partitions; If a specified message partition is designated, determine whether the specified message partition is included in the available message partitions. When the specified message partition is included in the available message partitions, determine that there is a reference message partition corresponding to the message type in the available message partitions, and use the specified message partition as the reference message partition; when the specified message partition is not included in the available message partitions, determine that there is no reference message partition corresponding to the message type in the available message partitions.

7. The message publishing method according to claim 1, wherein, If there is a new target message partition, sending the message to the new target message partition includes: Obtain whether the message is sent successfully; When the message sending fails, continue to send the message to the new target message partition according to the configured message retry policy; If it still fails to send, determine that the message fails to be sent in the new target message partition; If the sending is successful, determine that the message is successfully published.

8. A message publishing device, applied to a message producer of a Kafka cluster, wherein, The message publishing device includes: A partition update unit, configured to update the available message partitions of the message when the message fails to be sent in the target message partition; A partition determination unit, configured to determine whether there is a new target message partition in the updated available message partitions according to a preset partition policy; A message sending unit, configured to send the message to the new target message partition if there is a new target message partition; A message processing unit, configured to stop sending the message to the Kafka cluster if there is no new target message partition.

9. An electronic device, wherein, The electronic device includes: A memory, storing computer-executable instructions; A processor, when the computer-executable instructions are executed, causing the processor to execute the message publishing method according to any one of claims 1 to 7.

10. A computer-readable storage medium, wherein, The computer-readable storage medium stores one or more programs, which, when executed by an electronic device including a plurality of application programs, cause the electronic device to execute the message publishing method according to any one of claims 1 to 7.

Images