Message fault processing method and apparatus, computing device, and computer storage medium
By combining fault frequency and heartbeat detection results, the message sending fault type is identified and targeted fault handling strategies are executed, which solves the batch processing blocking problem caused by message faults and achieves efficient message retry and order guarantee without fault awareness.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- CHINA MOBILE (SUZHOU) SOFTWARE TECH CO LTD
- Filing Date
- 2022-11-16
- Publication Date
- 2026-07-28
AI Technical Summary
Existing message failure handling methods may block normal batch processing, causing anomalies in upstream services.
By obtaining the failure frequency of message sending failures and the heartbeat detection results from the production end, the failure type is determined, and corresponding failure handling strategies are executed, including different handling strategies for short-term failure types and long-term failure types, such as selecting the normal partition to send messages, storing and retransmitting the failure message, and switching the partition topic.
It effectively identifies the types of message sending failures, increases the probability of successful retry of failed messages, ensures message order, and makes the production and consumption ends unaware of cluster failures, without affecting upstream and downstream businesses.
Smart Images
Figure CN116264535B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of Internet technology, specifically to a message fault handling method, apparatus, computing device, and computer storage medium. Background Technology
[0002] Message queues often reside in complex distributed systems, and message sending failures are likely to occur due to factors such as network fluctuations, service outages, and program anomalies. Current technologies typically employ timed retrying for failed message delivery. Taking Flume as an example, Flume is a data acquisition tool. Its data acquisition architecture includes data sources, channels, and sink components. Data sources can collect data from various sources such as log files, network ports, and Kafka clusters, encapsulate it, and write it to channels. After data is successfully written to a channel, the sink component actively pulls data from the channel and writes it to various big data components such as HDFS, HBase, Hive, and Elasticsearch. Channels act as passive storage, responsible for temporarily storing data. Flume channels include memory channels, which use memory to cache data, resulting in extremely high throughput.
[0003] However, when processing large volumes of messages in real time, repeated retries and resulting failures can block normal batch processing. When Flume uses the memory channel mode to sink Kafka for retries, large-scale retries can cause memory channel blockage, affecting Flume resource writes and causing anomalies in upstream services. Summary of the Invention
[0004] In view of the above problems, this application proposes a message fault handling method, apparatus, computing device and computer storage medium to solve the following problem: existing message fault handling methods may block normal batch processing, causing abnormalities in upstream services.
[0005] According to one aspect of the embodiments of this application, a message fault handling method is provided, including:
[0006] Obtain the failure frequency of message sending failures and perform heartbeat detection on the production end to obtain the heartbeat detection results;
[0007] Based on the failure frequency and heartbeat detection results, determine the failure type corresponding to the message sending failure;
[0008] Based on the fault type, execute the fault handling strategy corresponding to the fault type to eliminate the message sending failure.
[0009] Furthermore, the heartbeat detection results obtained from heartbeat detection at the production end further include:
[0010] The heartbeat detection thread performs heartbeat detection on the production end and obtains the heartbeat response results of the production end in the session;
[0011] The heart rate detection result is determined based on the heart rate response.
[0012] Furthermore, based on the heartbeat response results, the heartbeat detection results further include:
[0013] If the heartbeat response result indicates that the session contains heartbeat response information, then the heartbeat detection result indicates that the session is normal.
[0014] If the heartbeat response result indicates that the session does not contain heartbeat response information, then the heartbeat detection result is determined to be a session abnormality.
[0015] Furthermore, the fault types include: short-term fault types and long-term fault types;
[0016] Based on the failure frequency and heartbeat detection results, the failure type corresponding to the message sending failure is further determined by:
[0017] If the failure frequency is less than the preset frequency threshold and the heartbeat detection result is that the session is normal, then the failure type corresponding to the message sending failure is determined to be a short-term failure type.
[0018] If the fault frequency is greater than or equal to the preset frequency threshold and the heartbeat detection result is a session abnormality, then the fault type corresponding to the message sending fault is determined to be a long-term fault type.
[0019] Furthermore, after determining that the fault type corresponding to the message sending failure is a short-term fault type, the method also includes: determining the subtype of the short-term fault type to which the message sending failure belongs based on the fault frequency;
[0020] Based on the fault type, the corresponding fault handling strategy is executed to eliminate the message sending failure, which further includes:
[0021] If the subtype is network jitter fault subtype, then select a partition that is in normal working condition for the message group that has experienced message sending failure, and send the message group to the partition;
[0022] If the subtype is the leader node election subtype, the message that caused the message sending failure will be stored in the local cache on the production side. After the leader node election is completed, the message that caused the message sending failure stored in the local cache will be resent.
[0023] Furthermore, based on the fault type, the corresponding fault handling strategy is executed to eliminate message transmission failures, which further includes:
[0024] If the fault type is a long-term fault, the current working partition will be switched from the partition corresponding to the primary topic to the partition corresponding to the backup topic, so that the production end uses the partition corresponding to the backup topic to send messages, and the consumer end uses the partition corresponding to the backup topic to receive messages.
[0025] Furthermore, after switching the current working partition from the partition corresponding to the primary topic to the partition corresponding to the backup topic, so that the producer uses the partition corresponding to the backup topic to send messages and the consumer uses the partition corresponding to the backup topic to receive messages, the method also includes:
[0026] Use a scheduled task to check if the session has returned to normal.
[0027] If the session resumes normally, the current working partition will be restored from the partition corresponding to the backup topic to the partition corresponding to the primary topic, so that the producer uses the partition corresponding to the primary topic to send messages and the consumer uses the partition corresponding to the primary topic to receive messages.
[0028] According to another aspect of the embodiments of this application, a message fault handling apparatus is provided, including:
[0029] The detection module is used to obtain the failure frequency of message sending failures and to perform heartbeat detection on the production end to obtain the heartbeat detection results.
[0030] The type determination module is used to determine the fault type corresponding to the message sending failure based on the fault frequency and heartbeat detection results;
[0031] The fault handling module is used to execute the fault handling strategy corresponding to the fault type to eliminate message sending failures.
[0032] According to another aspect of the embodiments of this application, a computing device is provided, including: a processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface communicate with each other through the communication bus;
[0033] The memory is used to store at least one executable instruction, which causes the processor to perform the operation corresponding to the above-mentioned message fault handling method.
[0034] According to another aspect of the embodiments of this application, a computer storage medium is provided, wherein the storage medium stores at least one executable instruction, which causes a processor to perform an operation corresponding to the message fault handling method described above.
[0035] According to the message fault handling method, apparatus, computing device, and computer storage medium provided in the embodiments of this application, a heartbeat detection mechanism is added to the production end for message sending faults in the cluster. The fault frequency of message sending faults and the heartbeat detection results are fully combined to effectively identify the fault type corresponding to the message sending fault. Furthermore, a correspondence between fault types and fault handling strategies is set. For different fault types, different fault handling strategies are executed to eliminate message sending faults. This not only increases the probability of successful retry of failed messages while ensuring message order, but also makes the business on the production end and the business on the consumer end unaware of cluster faults, without affecting upstream and downstream businesses.
[0036] The above description is merely an overview of the technical solutions of the embodiments of this application. In order to better understand the technical means of the embodiments of this application and to implement them in accordance with the contents of the specification, and to make the above and other objects, features and advantages of the embodiments of this application more obvious and understandable, specific implementation methods of the embodiments of this application are described below. Attached Figure Description
[0037] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the embodiments of this application. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings:
[0038] Figure 1 A flowchart illustrating a message fault handling method according to an embodiment of this application is shown;
[0039] Figure 2 A flowchart illustrating a message fault handling method according to another embodiment of this application is shown;
[0040] Figure 3 A structural block diagram of a message fault handling apparatus according to an embodiment of this application is shown;
[0041] Figure 4 A schematic diagram of the structure of a computing device according to an embodiment of this application is shown. Detailed Implementation
[0042] Exemplary embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.
[0043] Figure 1A flowchart illustrating a message fault handling method according to an embodiment of this application is shown, as follows: Figure 1 As shown, the method includes the following steps:
[0044] Step S101: Obtain the failure frequency of message sending failure and perform heartbeat detection on the production end to obtain the heartbeat detection result.
[0045] Step S102: Determine the fault type corresponding to the message sending failure based on the fault frequency and heartbeat detection results.
[0046] Step S103: Based on the fault type, execute the fault handling strategy corresponding to the fault type to eliminate the message sending fault.
[0047] According to the message fault handling method provided in the embodiments of this application, a heartbeat detection mechanism is added to the production end for message sending faults in the cluster. The fault frequency and heartbeat detection results are fully combined to effectively identify the fault type corresponding to the message sending fault. Furthermore, a correspondence between fault types and fault handling strategies is set. For different fault types, different fault handling strategies are executed to eliminate message sending faults. This not only increases the probability of successful retry of failed messages while ensuring message order, but also makes the business on the production end and the business on the consumer end unaware of cluster faults, without affecting upstream and downstream businesses.
[0048] Figure 2 A flowchart illustrating a message fault handling method according to another embodiment of this application is shown, as follows: Figure 2 As shown, the method includes the following steps:
[0049] Step S201: Obtain the failure frequency of message sending failure and perform heartbeat detection on the production end to obtain the heartbeat detection result.
[0050] When a message sending failure occurs in the cluster, the frequency of such failures needs to be determined. Taking a Kafka cluster as an example, common Kafka cluster failures include single-node disk failures and the absence of a leader node for a topic's corresponding partition, causing producers to fail to send messages to that partition. To accurately determine the type of message sending failure, it is necessary to obtain the frequency of these failures.
[0051] In existing Kafka implementations, heartbeat detection is typically only configured for the consumer. In this embodiment, however, heartbeat detection is also enabled for the producer, instead of using the number of failed message transmissions to determine the type of failure. This effectively avoids inaccurate analysis caused by varying message transmission frequencies. In this embodiment, the fault type corresponding to a message transmission failure is determined by combining the failure frequency and the producer's heartbeat detection results. Therefore, in step S201, the producer's heartbeat detection results also need to be obtained.
[0052] Specifically, a heartbeat detection thread performs heartbeat checks on the production end to obtain the heartbeat response results within the session. Based on these responses, the heartbeat detection result is determined. If the heartbeat response result indicates that the session contains heartbeat response information, the session is considered normal; if the heartbeat response result indicates that the session does not contain heartbeat response information, the session is considered abnormal. For example, if the heartbeat detection thread sends heartbeat detection threads at fixed intervals, a single session can contain multiple heartbeat responses. If a heartbeat response is completed within a single session, the connection is considered normal, i.e., the session is normal.
[0053] Step S202: Determine the fault type corresponding to the message sending failure based on the fault frequency and heartbeat detection results. If the fault type is a short-term fault, proceed to step S203; if the fault type is a long-term fault, proceed to step S206.
[0054] In this embodiment, the fault type can be divided into short-term fault type and long-term fault type according to the duration of the message sending failure. The short-term fault type and long-term fault type are classified according to the duration of the message sending failure. Specifically, if the duration of the message sending failure is less than a preset duration, the message sending failure is classified as a short-term fault type; if the duration of the message sending failure is greater than or equal to the preset duration, the message sending failure is classified as a long-term fault type.
[0055] Specifically, short-term faults may include:
[0056] (1) Severe network jitter between a certain production node and the leader node of a certain partition caused the production node to fail to send messages to this partition for a period of time.
[0057] (2) The election of a single leader node within the cluster may cause all messages sent to this partition by the producers to fail to be sent for a period of time.
[0058] (3) If the leader node distribution in the cluster does not meet the load balancing conditions due to the cumulative restarts of the leader node, the leader node can be manually or automatically redistributed. This situation may also cause the cluster to be unavailable to the outside world for a short period of time.
[0059] Prolonged failure scenarios may include: To ensure message reliability and balance cluster performance, two replicas are typically used. However, in actual production environments, due to inadequate operation and maintenance inspections, single-section service failures may not be repaired in a timely manner, resulting in the failure of both nodes of the two replicas of a partition. This may lead to a situation where there is no available leader node for a long time, i.e., the leader id is equal to -1, indicating that a leader has not been successfully elected.
[0060] For short-term failures, using infinite retries can alleviate the immediate transmission failure, but it will impact write operations. For long-term failures, if maintenance personnel cannot detect and repair them promptly, the failure will persist for an extended period, making timely repair difficult. Furthermore, using infinite retries fails to effectively resolve the problem and may even have a significant adverse impact on both production and consumption operations.
[0061] Although a single partition may not function properly, the cluster as a whole may still function normally. However, in business scenarios where Flume is only used in memory channel mode to write data to Kafka, this may cause Flume's memory channel to become blocked, thereby affecting upstream services and causing them to malfunction.
[0062] Short-term failures typically only impact production-side operations, with minimal impact on consumer-side operations. Long-term failures, however, often involve failures across all nodes containing partitions, significantly detrimental to both production and consumer operations. This embodiment provides different fault handling strategies for short-term and long-term failures, ensuring that production and consumer operations are unaware of cluster failures and that upstream and downstream Kafka services are not affected.
[0063] In this embodiment, the fault type corresponding to a message sending failure is determined by combining the failure frequency and the heartbeat detection results from the production end. If the failure frequency is less than a preset frequency threshold and the heartbeat detection result indicates a normal session, the fault type corresponding to the message sending failure is determined to be a short-term failure. If the failure frequency is greater than or equal to the preset frequency threshold and the heartbeat detection result indicates an abnormal session, the fault type corresponding to the message sending failure is determined to be a long-term failure. Specifically, if the session is normal but message sending failures occur frequently, it is considered a short-term failure; if the session is abnormal and message sending failures continue to occur, it is considered a long-term failure.
[0064] Step S203: Based on the failure frequency, determine the subtype of the short-term failure type to which the message sending failure belongs. If the subtype is the network jitter failure subtype, proceed to step S204; if the subtype is the leader node election subtype, proceed to step S205.
[0065] In real-world scenarios, short-term fault types can be further divided into multiple subtypes based on their causes. Specifically, short-term fault types include: network jitter fault subtype and leader node election subtype.
[0066] Specifically, Kafka needs to be configured with a non-zero retry count, typically set to 3-5 times. If the message is successfully sent within the retry count, it indicates a minor network jitter, and there's no need to activate the fault handling strategy for short-term failures, thus avoiding frequent activation of this strategy. If the message fails to send within the retry count, it indicates a more serious network jitter, and the fault handling strategy for short-term failures needs to be activated.
[0067] If the sending failure is caused by severe network jitter, the corresponding situation is frequent and intermittent sending failures within a short period of time, but the network from the consumer to the corresponding Kafka broker (message broker) may be normal. If the sending failure is caused by leader election, the corresponding situation is continuous sending failures within a short period of time, in which case the consumer will also be unable to consume messages normally. Therefore, a preset range can be set based on the above situations. If the change in failure frequency conforms to the preset range, it indicates frequent and intermittent sending failures within a short period of time, and the subtype of short-term failure to which the message sending failure belongs is determined to be the network jitter failure subtype. If the change in failure frequency does not conform to the preset range, it indicates continuous sending failures within a short period of time, and the subtype of short-term failure to which the message sending failure belongs is determined to be the leader election subtype.
[0068] Step S204: Select a partition that is in normal working condition for the message group that has experienced a message sending failure, and send the message group to the partition.
[0069] Since messages are sent in batches, if the subtype is network jitter failure, then for message batches that failed to send, a normally functioning partition can be selected and sent using methods such as round-robin. Instead of splitting the message batch and resending each message, this does not affect the overall sending throughput.
[0070] Step S205: Store the message that caused the message sending failure in the local cache of the production end. After the leader node is selected, resend the message that caused the message sending failure stored in the local cache.
[0071] If the subtype is a leader node election subtype, messages can be temporarily stored in third-party storage. Since the sending failures are short-lived, there won't be many failed messages. To avoid adding system components, this embodiment uses the local disk where the production end resides as third-party storage to temporarily store failed messages, effectively avoiding increased system and operational complexity. After the election is complete and a message is successfully sent, a separate production end is started to send the disk-cached messages to the corresponding partition.
[0072] For short-term failures, the two failure handling strategies mentioned above can effectively resolve cluster service failures caused by single-node failures.
[0073] Step S206: Switch the current working partition from the partition corresponding to the primary topic to the partition corresponding to the backup topic, so that the producer uses the partition corresponding to the backup topic to send messages and the consumer uses the partition corresponding to the backup topic to receive messages.
[0074] In cases of prolonged failures, if operations and maintenance personnel fail to detect and resolve them promptly, the failure may persist for an extended period. If fault handling strategies designed for short-term failures are still applied, the accumulated messages over a long period may lead to an imbalance in the Kafka partition's data load after recovery, resulting in a relatively large amount of data in that partition. If the downstream system of Kafka is a Spark Streaming or Flink real-time computing system, consuming Kafka will cause data skew in the real-time computing system, impacting the overall computing speed.
[0075] Generally, a single node failure renders a partition unwritable, which also means that partition becomes unreadable. To improve consumption throughput, the number of consumers can be set to equal the number of topic partitions. If one partition cannot be consumed, it means one consumer will not be able to consume data. For example, if each consumer has a throughput of 20MB / s, the overall throughput of 5 partitions is 100MB / s. If one consumer cannot consume data, the overall throughput will decrease by 20MB / s, dropping to 80MB / s. If a sender failure triggers a retry mechanism, the sending throughput will not be significantly affected, but the consumption throughput will be greatly impacted, potentially causing message backlog over time.
[0076] If the failure type is a long-term failure, a dual-topic partition switching mechanism is used to handle it. After starting production with topic:topic-1, the backend automatically creates a topic:topic-1-temp as a backup topic for topic-1. topic-1-temp and topic-1 have the same number of partitions, and the brokers for the partitions of each topic are different, in order to ensure that the partitions will not fall on the failed node after the switch.
[0077] In the case of a prolonged failure, the production side enables a dual-topic partitioning mechanism to migrate traffic from the failed partition to the partition corresponding to the backup topic. The production, broker, and consumer sides all need to perform corresponding switches to ensure load balancing across the cluster partitions and prevent impact on downstream services. Specifically, messages sent to partition n of topic-1 (the primary topic) are sent to partition n of topic-1-temp (the backup topic). The production side caches the last successfully sent offset for each partition. The production side notifies the Kafka broker of the offset of the message sent to partition n of topic-1-temp, and the production side caches the last successfully sent offset for each partition as the end offset. After successfully receiving the message, the Kafka broker notifies the corresponding consumer to start consuming messages from partition n of topic-1-temp. This embodiment implements the switching of the current working partition from the partition corresponding to the primary topic to the partition corresponding to the backup topic, enabling the production side to send messages using the partition corresponding to the backup topic, and the consumer side to receive messages using the partition corresponding to the backup topic.
[0078] Step S207: Check whether the session has returned to normal through a scheduled task; if yes, proceed to step S208; if no, continue to proceed to step S207.
[0079] Step S208: Restore the current working partition from the partition corresponding to the backup topic to the partition corresponding to the primary topic, so that the production end uses the partition corresponding to the primary topic to send messages, and the consumer end uses the partition corresponding to the primary topic to receive messages.
[0080] After implementing the dual-topic partition failover mechanism, the Kafka broker starts a timed thread that checks at fixed intervals whether partition n of topic-1 is usable. If the failed partition is found to be usable, data is synchronized from partition n of topic-1-temp. Once synchronization is complete, the producer and consumer are notified to switch to partition n of topic-1 for production and consumption. Consumers subsequently consuming the primary topic are unaware of this.
[0081] According to the message failure handling method provided in this application embodiment, a heartbeat detection mechanism is added to the production end for message sending failures in the cluster, effectively avoiding the problem of inaccurate analysis caused by different message sending frequencies. The method effectively identifies the failure type corresponding to the message sending failure by fully combining the failure frequency and heartbeat detection results, and sets a correspondence between failure types and failure handling strategies. For different failure types, different failure handling strategies are executed to eliminate message sending failures. Furthermore, short-term failure types are further subdivided into network jitter failure subtypes and leader node election subtypes, with different failure handling strategies executed for different short-term failure subtypes, effectively solving the cluster's external service failure caused by single-node failures. This solution not only increases the probability of successful retry of failed messages while ensuring message order, but also ensures that the business on the production end and the business on the consumer end are unaware of cluster failures, without affecting upstream and downstream businesses.
[0082] Figure 3 A structural block diagram of a message fault handling apparatus according to an embodiment of this application is shown, as follows: Figure 3 As shown, the device includes: a detection module 310, a type determination module 320, and a fault handling module 330.
[0083] The detection module 310 is used to: obtain the failure frequency of message sending failures, and perform heartbeat detection on the production end to obtain the heartbeat detection results.
[0084] The type determination module 320 is used to determine the fault type corresponding to the message sending failure based on the fault frequency and heartbeat detection results.
[0085] The fault handling module 330 is used to: execute a fault handling strategy corresponding to the fault type to eliminate message sending failures.
[0086] Optionally, the detection module 310 is further configured to: perform heartbeat detection on the production end through the heartbeat detection thread, obtain the heartbeat response result of the production end in the session; and determine the heartbeat detection result based on the heartbeat response result.
[0087] Optionally, the detection module 310 is further configured to: if the heartbeat response result is that the session contains heartbeat response information, then determine that the heartbeat detection result is that the session is normal; if the heartbeat response result is that the session does not contain heartbeat response information, then determine that the heartbeat detection result is that the session is abnormal.
[0088] Optionally, the fault types include: short-term fault type and long-term fault type. The type determination module 320 is further configured to: if the fault frequency is less than a preset frequency threshold and the heartbeat detection result is that the session is normal, then determine that the fault type corresponding to the message sending fault is a short-term fault type; if the fault frequency is greater than or equal to the preset frequency threshold and the heartbeat detection result is that the session is abnormal, then determine that the fault type corresponding to the message sending fault is a long-term fault type.
[0089] Optionally, the type determination module 320 is further configured to: determine the subtype of the short-term fault type to which the message sending failure belongs based on the fault frequency. Optionally, the fault handling module 330 is further configured to: if the subtype is a network jitter fault subtype, select a partition in normal working condition for the message group that experienced the message sending failure, and send the message group to the partition; if the subtype is a leader node election subtype, store the message that experienced the message sending failure in the local cache of the production end, and resend the message that experienced the message sending failure stored in the local cache after the leader node election is completed.
[0090] Optionally, the fault handling module 330 is further configured to: if the fault type is a long-term fault type, switch the current working partition from the partition corresponding to the primary topic to the partition corresponding to the backup topic, so that the production end uses the partition corresponding to the backup topic to send messages, and the consumer end uses the partition corresponding to the backup topic to receive messages.
[0091] Optionally, the fault handling module 330 is further configured to: detect whether the session has returned to normal through a scheduled task; if the session has returned to normal, restore the current working partition from the partition corresponding to the backup topic to the partition corresponding to the primary topic, so that the production end uses the partition corresponding to the primary topic to send messages and the consumer end uses the partition corresponding to the primary topic to receive messages.
[0092] The descriptions of the above modules refer to the corresponding descriptions in the method embodiments, and will not be repeated here.
[0093] According to the message fault handling device provided in this application embodiment, a heartbeat detection mechanism is added to the production end for message sending faults in the cluster, effectively avoiding the problem of inaccurate analysis caused by different message sending frequencies. The device effectively identifies the fault type corresponding to the message sending fault by fully combining the fault frequency and heartbeat detection results, and sets a correspondence between fault types and fault handling strategies. For different fault types, different fault handling strategies are executed to eliminate message sending faults. Furthermore, short-term fault types are further subdivided into network jitter fault subtypes and leader node election subtypes, with different fault handling strategies executed for different short-term fault subtypes, effectively solving the cluster's external service failure caused by single-node failures. This solution not only increases the probability of successful retry of failed messages while ensuring message order, but also ensures that the business on the production end and the business on the consumer end are unaware of cluster faults, without affecting upstream and downstream businesses.
[0094] This application also provides a non-volatile computer storage medium storing at least one executable instruction that can execute the message fault handling method in any of the above method embodiments.
[0095] Figure 4 The diagram shows a structural schematic of a computing device according to an embodiment of the present application. The specific embodiments of the present application do not limit the specific implementation of the computing device.
[0096] like Figure 4 As shown, the computing device may include: a processor 402, a communications interface 404, a memory 406, and a communications bus 408.
[0097] in:
[0098] The processor 402, communication interface 404, and memory 406 communicate with each other via communication bus 408.
[0099] Communication interface 404 is used to communicate with other network elements such as clients or other servers.
[0100] The processor 402 is used to execute program 410, specifically to execute the relevant steps in the above-described message fault handling method embodiment.
[0101] Specifically, program 410 may include program code that includes computer operation instructions.
[0102] Processor 402 may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of this application. The computing device includes one or more processors, which may be processors of the same type, such as one or more CPUs; or processors of different types, such as one or more CPUs and one or more ASICs.
[0103] Memory 406 is used to store program 410. Memory 406 may include high-speed RAM memory, and may also include non-volatile memory, such as at least one disk storage device.
[0104] Specifically, program 410 can be used to cause processor 402 to execute the message fault handling method in any of the above method embodiments. The specific implementation of each step in program 410 can be found in the corresponding descriptions of the steps and units in the above message fault handling embodiments, and will not be repeated here. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the devices and modules described above can be referred to the corresponding process descriptions in the foregoing method embodiments, and will not be repeated here.
[0105] The algorithms and displays provided herein are not inherently related to any particular computer, virtual system, or other device. Various general-purpose systems can also be used in conjunction with the teachings herein. The required structure for constructing such systems is apparent from the above description. Furthermore, the embodiments of this application are not directed to any particular programming language. It should be understood that the contents of the embodiments of this application described herein can be implemented using various programming languages, and the above description of specific languages is for the purpose of disclosing the best implementation of the embodiments of this application.
[0106] Numerous specific details are set forth in the specification provided herein. However, it will be understood that embodiments of this application may be practiced without these specific details. In some instances, well-known methods, structures, and techniques have not been shown in detail so as not to obscure the understanding of this specification.
[0107] Similarly, it should be understood that, in order to simplify this disclosure and aid in understanding one or more of the various inventive aspects, in the foregoing description of exemplary embodiments of the present application, various features of the present application embodiments are sometimes grouped together into a single embodiment, figure, or description thereof. However, this approach to disclosure should not be construed as reflecting an intention that the claimed embodiments of the present application require more features than expressly recited in each claim. Rather, as reflected in the following claims, inventive aspects lie in fewer than all features of a single foregoing disclosed embodiment. Therefore, the claims following the detailed description are hereby expressly incorporated into that detailed description, wherein each claim itself is a separate embodiment of the present application.
[0108] Those skilled in the art will understand that modules in the device of the embodiments can be adaptively changed and placed in one or more devices different from that embodiment. Modules, units, or components in the embodiments can be combined into a single module, unit, or component, and further, they can be divided into multiple sub-modules, sub-units, or sub-components. Except where at least some of such features and / or processes or units are mutually exclusive, any combination can be used to combine all features disclosed in this specification (including the accompanying claims, abstract, and drawings) and all processes or units of any method or device so disclosed. Unless expressly stated otherwise, each feature disclosed in this specification (including the accompanying claims, abstract, and drawings) may be replaced by an alternative feature that serves the same, equivalent, or similar purpose.
[0109] Furthermore, those skilled in the art will understand that although some embodiments described herein include certain features but not others included in other embodiments, combinations of features from different embodiments are meant to be within the scope of the embodiments of this application and form different embodiments. For example, in the following claims, any one of the claimed embodiments can be used in any combination.
[0110] The various component embodiments of this application can be implemented in hardware, or as software modules running on one or more processors, or a combination thereof. Those skilled in the art will understand that microprocessors or digital signal processors (DSPs) can be used in practice to implement some or all of the functions of some or all of the components according to the embodiments of this application. The embodiments of this application can also be implemented as device or apparatus programs (e.g., computer programs and computer program products) for performing part or all of the methods described herein. Such programs implementing the embodiments of this application can be stored on a computer-readable medium, or can be in the form of one or more signals. Such signals can be downloaded from an Internet website, provided on a carrier signal, or provided in any other form.
[0111] It should be noted that the above embodiments are illustrative of the embodiments of this application and not limiting of the embodiments of this application, and those skilled in the art can devise alternative embodiments without departing from the scope of the appended claims. In the claims, any reference signs placed between parentheses should not be construed as limiting the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. Embodiments of this application can be implemented by means of hardware comprising several different elements and by means of a suitably programmed computer. In the unit claims enumerating several means, several of these means may be embodied by the same item of hardware. The use of the words first, second, and third, etc., does not indicate any order. These words can be interpreted as names.
Claims
1. A message fault handling method, characterized in that, include: Obtain the failure frequency of message sending failures and perform heartbeat detection on the production end to obtain the heartbeat detection results; Based on the fault frequency and the heartbeat detection results, the fault type corresponding to the message sending failure is determined; The fault types include: short-term fault types and long-term fault types; Based on the fault type, execute the fault handling strategy corresponding to the fault type to eliminate the message sending fault; If the message sending failure is determined to be a short-term failure, the method further includes: configuring a retry count; if the message is not successfully sent within the retry count, then activating the fault handling strategy corresponding to the short-term failure type; determining the subtype of the short-term failure type to which the message sending failure belongs based on the failure frequency; if the change range of the failure frequency meets a preset range, then determining the subtype to be a network jitter failure subtype; if the change range of the failure frequency does not meet the preset range, then determining the subtype to be a leader node election subtype. The step of executing a fault handling strategy corresponding to the fault type to eliminate the message sending failure further includes: If the subtype is a network jitter fault subtype, then select a partition that is in normal working condition for the message group that experienced the message sending failure, and send the message group to the partition. If the subtype is a leader node election subtype, the message that caused the message sending failure will be stored in the local cache of the production end. After the leader node election is completed, the message that caused the message sending failure stored in the local cache will be resent.
2. The method according to claim 1, characterized in that, The step of obtaining heartbeat detection results by performing heartbeat detection on the production end further includes: The heartbeat detection thread performs heartbeat detection on the production end and obtains the heartbeat response result of the production end in the session. The heartbeat detection result is determined based on the heartbeat response result.
3. The method according to claim 2, characterized in that, The step of determining the heartbeat detection result based on the heartbeat response result further includes: If the heartbeat response result indicates that the session contains heartbeat response information, then the heartbeat detection result is determined to be that the session is normal. If the heartbeat response result indicates that the session does not contain heartbeat response information, then the heartbeat detection result is determined to be a session abnormality.
4. The method according to any one of claims 1-3, characterized in that, The step of determining the fault type corresponding to the message sending failure based on the fault frequency and the heartbeat detection result further includes: If the fault frequency is less than a preset frequency threshold and the heartbeat detection result is that the session is normal, then the fault type corresponding to the message sending fault is determined to be a short-term fault type. If the fault frequency is greater than or equal to a preset frequency threshold and the heartbeat detection result is a session abnormality, then the fault type corresponding to the message sending fault is determined to be a long-term fault type.
5. The method according to claim 4, characterized in that, The step of executing a fault handling strategy corresponding to the fault type to eliminate the message sending failure further includes: If the fault type is a long-term fault type, the current working partition will be switched from the partition corresponding to the primary topic to the partition corresponding to the backup topic, so that the production end uses the partition corresponding to the backup topic to send messages, and the consumer end uses the partition corresponding to the backup topic to receive messages.
6. The method according to claim 5, characterized in that, After switching the current working partition from the partition corresponding to the primary topic to the partition corresponding to the backup topic, so that the producer uses the partition corresponding to the backup topic to send messages and the consumer uses the partition corresponding to the backup topic to receive messages, the method further includes: Use a scheduled task to check if the session has returned to normal. If the session resumes normally, the current working partition will be restored from the partition corresponding to the backup topic to the partition corresponding to the primary topic, so that the producer uses the partition corresponding to the primary topic to send messages, and the consumer uses the partition corresponding to the primary topic to receive messages.
7. A message fault handling device, characterized in that, include: The detection module is used to obtain the failure frequency of message sending failures and to perform heartbeat detection on the production end to obtain the heartbeat detection results. The type determination module is used to determine the fault type corresponding to the message sending fault based on the fault frequency and the heartbeat detection result; The fault types include: short-term fault types and long-term fault types; The fault handling module is used to execute a fault handling strategy corresponding to the fault type according to the fault type, so as to eliminate the message sending fault; The type determination module is further configured to: if the fault type corresponding to the message sending failure is determined to be a short-term fault type, configure the number of retries; if the message is not successfully sent within the number of retries, activate the fault handling strategy corresponding to the short-term fault type; determine the subtype of the short-term fault type to which the message sending failure belongs based on the fault frequency; if the change range of the fault frequency meets the preset range, determine the subtype as a network jitter fault subtype; if the change range of the fault frequency does not meet the preset range, determine the subtype as a leader node election subtype. The fault handling module is further used for: If the subtype is a network jitter fault subtype, then select a partition that is in normal working condition for the message group that experienced the message sending failure, and send the message group to the partition. If the subtype is a leader node election subtype, the message that caused the message sending failure will be stored in the local cache of the production end. After the leader node election is completed, the message that caused the message sending failure stored in the local cache will be resent.
8. A computing device, comprising: The processor, memory, communication interface, and communication bus are provided, wherein the processor, memory, and communication interface communicate with each other via the communication bus. The memory is used to store at least one executable instruction, which causes the processor to perform the operation corresponding to the message fault handling method as described in any one of claims 1-6.
9. A computer storage medium storing at least one executable instruction that causes a processor to perform an operation corresponding to the message fault handling method as described in any one of claims 1-6.