Disaster recovery switching method and device of consumption machine, computer equipment and medium

Abstract

The invention relates to a disaster recovery switching method and device for consumption machines, computer equipment and a medium, and the method comprises the steps: recovering business data on a first consumption machine when it is monitored that the heartbeat stop duration of the first consumption machine reaches a preset heartbeat cycle; when it is monitored that the second consumption machine sends the exclusive lock, independently sending service data to the second consumption machine to complete switching operation of the consumption machines; the second consumption machine is the first consumption machine which sends the exclusive lock to the Redis end in the rest of the plurality of consumption machines; wherein when the first consumption machine crashes, or an internal program is abnormal, the first consumption machine stops sending the heartbeat. The disaster recovery switching scheme of the consumption machine is designed, it is ensured that the service data can only be consumed by a single consumption machine in the distributed environment, when problems occur, the disaster recovery switching operation is executed to avoid the problem of incremental data synchronization delay, and normal operation of services is ensured.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of computer technology, and in particular to a disaster recovery switching method, device, computer equipment and medium for a consumer machine. Background Art

[0002] In the face of data storage and query challenges brought about by the growth of the company's business, an incremental data synchronization solution based on Kafka is generally adopted. The incremental data synchronization solution of Kafka mainly includes: first, listen to the binlog logs sent by a data source, usually a relational database. Then set up a Kafka cluster, which receives and consumes business data and ensures that it can handle the expected data volume and throughput. Among them, the received binlog logs can be converted into the required data format in the consumer application, and the logic of assembling wide tables can be added to write the converted data into the new data storage system, so as to achieve customized incremental data synchronization and ensure that the new system can handle the growing business data while maintaining the response speed of queries.

[0003] However, the inventors realized that since the Kafka client can only be opened and consumed by a single consumer machine, if the consumer machine unexpectedly crashes or the program is unavailable, other consumer machines will not be able to continue consuming the Kafka message, which can easily cause incremental data synchronization delays and lead to business uncontrollability. Summary of the invention

[0004] In view of the above-mentioned shortcomings or disadvantages, the present application provides a method, device, computer equipment and medium for disaster recovery switching of a consumer machine. By implementing the solution, it is ensured that business data can only be consumed by a single consumer machine in a distributed environment. When a problem occurs, a disaster recovery switching operation is performed to avoid the problem of incremental data synchronization delay, thereby ensuring the normal operation of the business.

[0005] According to a first aspect, the present application provides a method for disaster recovery switching of a consumer machine, the method being applied to a distributed data consumption system including multiple consumer machines, comprising:

[0006] When it is detected that the heartbeat of the first consumer machine stops for a period of time that reaches a preset heartbeat period, the business data on the first consumer machine is recovered, where the first consumer machine is a consumer machine that is currently consuming business data;

[0007] When it is detected that the second consumer machine issues an exclusive lock, the service data is sent to the second consumer machine alone to complete the switching operation of the consumer machine; the second consumer machine is the first consumer machine among the remaining multiple consumer machines to issue an exclusive lock to the Redis (RemoteDictionary Server, remote dictionary service) end;

[0008] Among them, when the first consumer machine crashes or an internal program fails, the first consumer machine will stop sending heartbeats.

[0009] In some embodiments, when starting a business project of the distributed data consumption system, the method further includes:

[0010] Randomly select a consumer machine in the distributed data consumption system as the first consumer machine;

[0011] Instruct the first consumer machine to obtain the Kafka configuration identifier set from the Redis end through subscription;

[0012] Among them, after obtaining the Kafka configuration identifier set, the first consumer machine traverses the Kafka configuration identifier set, extracts one of the target Kafka configuration identifiers to determine the value of the exclusive lock, and sends the exclusive lock to the Redis end.

[0013] In some embodiments, each consumer machine is originally set with an empty flag. When the empty flag of a consumer machine is assigned a specific Kafka configuration flag, the consumer machine sends a heartbeat to the Redis end. The method further includes:

[0014] When monitoring that the first consumer machine sends an exclusive lock to the Redis end, send the consumer configuration information corresponding to the target Kafka configuration identifier to the first consumer machine, where the consumer configuration information includes the Kafka server address, message set key, and client consumption mode information;

[0015] The Kafka client of the first consumer machine is started to perform consumption business data operations, and the empty identifier of the first consumer machine is assigned the target Kafka configuration identifier.

[0016] In some embodiments, starting a Kafka client of a first consumer machine to perform a business data operation includes:

[0017] Start the first and second sub-threads of the Kafka client;

[0018] Receive business data through the first child thread and store the business data in a blocking queue;

[0019] The business data is taken out from the blocking queue through the second sub-thread, and the corresponding business code is executed according to the business data.

[0020] In some embodiments, after the empty identifier of the first consumer machine is assigned the target Kafka configuration identifier, the first consumer machine sends a heartbeat to the Redis end according to the set heartbeat period;

[0021] Among them, a single heartbeat renewal period is greater than the heartbeat cycle; if the heartbeat renewal of the first consumer machine is successful, then within the current heartbeat renewal period, the first consumer machine exclusively occupies the target Kafka configuration identifier.

[0022] In some embodiments, retrieving the service data on the first consumer machine includes:

[0023] The target Kafka configuration identifier of the first consumer machine is set to a null identifier to stop the consumption business data operation of the Kafka client.

[0024] In some embodiments, the method further comprises:

[0025] Set the target Kafka configuration identifier of the first consumer machine and the current identifiers of the remaining consumer machines to null identifiers to stop all consumer business data operations in the distributed data consumer system;

[0026] If no consumer machine sends a heartbeat during the current heartbeat renewal period, then during the next heartbeat renewal period, if a third consumer machine is detected to send an exclusive lock to the Redis end, business data will be sent to the third consumer machine alone to complete the switching operation of the consumer machine.

[0027] According to a second aspect, the present application provides a disaster recovery switching device for a consumer machine, the disaster recovery switching device being applied to a distributed data consumption system including multiple consumer machines, including:

[0028] A business data recovery module is used to recover business data on the first consumer machine when it is detected that the heartbeat of the first consumer machine stops for a period of time that reaches a preset heartbeat period. The first consumer machine is a consumer machine that is currently consuming business data.

[0029] The business data switching module is used to send business data to the second consumer machine alone to complete the switching operation of the consumer machine when it is monitored that the second consumer machine issues an exclusive lock; the second consumer machine is the first consumer machine to issue an exclusive lock to the Redis end among the remaining multiple consumer machines;

[0030] Among them, when the first consumer machine crashes or an internal program fails, the first consumer machine will stop sending heartbeats.

[0031] According to a third aspect, the present application provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the disaster recovery switching method for a consumer machine in any one of the above embodiments are implemented.

[0032] According to a fourth aspect, the present application provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the disaster recovery switching method for a consumer machine in any one of the above-mentioned embodiments are implemented.

[0033] The method of the above embodiment can be applied to a monitoring server that monitors the heartbeat of a consumer machine. When the monitoring server receives a project startup instruction, it will randomly select a consumer machine in the distributed data consumption system as the first consumer machine, and then let the first consumer machine obtain the Kafka configuration identifier set from the Redis end through a subscription method. Among them, after obtaining the Kafka configuration identifier set, the first consumer machine will traverse the Kafka configuration identifier set, extract one of the target Kafka configuration identifiers to determine the value of the exclusive lock, and send an exclusive lock to the Redis end, so that the business data can only be consumed by the first consumer machine. When the monitoring server monitors that the heartbeat of the first consumer machine stops for a period longer than the heartbeat renewal period, it indicates that the first consumer machine has crashed or the internal program has an abnormality, and the business data on the first consumer machine is recovered. At this time, consumer machines other than the first consumer machine can send exclusive locks to the Redis end to strive for the opportunity to consume business data. Finally, when the monitoring server monitors that the second consumer machine has issued an exclusive lock, it sends business data to the second consumer machine alone to complete the switching operation of the consumer machine. In this way, it is ensured that business data can only be consumed by a single consumer machine in a distributed environment. When a problem occurs, a disaster recovery switching operation is performed to avoid the problem of incremental data synchronization delay, thus ensuring the normal operation of the business. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 A schematic flow chart of a method for disaster recovery switching of a consumer machine in one or more embodiments of the present application;

[0035] Figure 2 This is an example diagram of a distributed data consumption system monitored by a heartbeat mechanism in one or more embodiments of the present application;

[0036] Figure 3 A flowchart of a method for initially determining a consumer machine in one or more embodiments of the present application;

[0037] Figure 4 This is an example diagram of business data switching of a distributed data consumption system in one or more embodiments of the present application;

[0038] Figure 5 A flowchart of a consumption identification configuration method in one or more embodiments of the present application;

[0039] Figure 6A flowchart of a method for starting a Kafka client to perform business data operations in one or more embodiments of the present application;

[0040] Figure 7 An exemplary diagram of an active switching method of a consumer machine in one or more embodiments of the present application;

[0041] Figure 8 A schematic diagram of the structure of a disaster recovery switching device of a consumer machine in one or more embodiments of the present application;

[0042] Fig. 9 A schematic diagram of the structure of a computer device in one or more embodiments of the present application. DETAILED DESCRIPTION

[0043] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0044] According to a first aspect, the present application provides a method for disaster recovery switching of a consumer machine, the method being applied to a distributed data consumption system including multiple consumer machines, such as Figure 1 As shown, these consumer machines may be independent servers for processing business data, and the method includes:

[0045] S110: when it is detected that the heartbeat of the first consumer machine stops for a period of time that reaches a preset heartbeat period, the service data on the first consumer machine is recovered, where the first consumer machine is a consumer machine that is currently consuming service data;

[0046] Among them, when the first consumer machine crashes or an internal program fails, the first consumer machine will stop sending heartbeats.

[0047] Specifically, the above disaster recovery switching method can be applied to a monitoring server that monitors the heartbeat of a consumer machine. When the monitoring server receives a project startup instruction, it randomly selects a consumer machine in the distributed data consumer system as the first consumer machine.

[0048] In some embodiments, after the empty identifier of the first consumer machine is assigned the target Kafka configuration identifier, the first consumer machine sends a heartbeat to the Redis end according to the set heartbeat period;

[0049] Among them, a single heartbeat renewal period is greater than the heartbeat cycle; if the heartbeat renewal of the first consumer machine is successful, then within the current heartbeat renewal period, the first consumer machine exclusively occupies the target Kafka configuration identifier.

[0050] For example, Figure 2 As shown, after grabbing the target Kafka configuration identifier, the first consumer machine can send a heartbeat to the Redis end every 15 seconds. At this time, the heartbeat renewal period is equal to 4 times the heartbeat period, so that the first consumer machine has exclusive use of the target Kafka configuration identifier within 60 seconds of the heartbeat renewal period. If the first consumer machine is still processing business data within the next 60 seconds, or has grabbed a new Kafka configuration identifier, it will have exclusive use of the new Kafka configuration identifier within the next 60 seconds.

[0051] In some embodiments, when starting a business project of a distributed data consumption system, such as Figure 3 As shown, the method also includes:

[0052] S210: randomly selecting a consumer machine in the distributed data consumption system as a first consumer machine;

[0053] For example, Figure 4 As shown, the monitoring server can first randomly select consumer machine 1 from consumer machines 1-3 as the consumer machine of the business data. At this time, consumer machine 1 serves as the host, and consumer machines 1 and 2 serve as disaster recovery machines.

[0054] S220: Instruct the first consumer machine to obtain the Kafka configuration identifier set from the Redis end through a subscription method;

[0055] Among them, after obtaining the Kafka configuration identifier set, the first consumer machine traverses the Kafka configuration identifier set, extracts one of the target Kafka configuration identifiers to determine the value of the exclusive lock, and sends the exclusive lock to the Redis end.

[0056] For example, Figure 3 As shown in the figure, after the first consumer machine determines the target Kafka configuration identifier, the first consumer machine first uses the SETNX command to create a key offset on the Redis side, and the message offset of the target Kafka configuration identifier can be used as part of the offset, thereby determining the value of the exclusive lock. Then, when the SETNX command returns 1 to the monitoring server, it indicates that the Redis side has obtained the exclusive lock issued by the first consumer machine. In this way, it is ensured that business data can only be consumed by a single consumer machine in a distributed environment, which improves the accuracy and effectiveness of business data consumption.

[0057] In some embodiments, each consumer machine is originally set with an empty flag. When the empty flag of a consumer machine is assigned a specific Kafka configuration flag, the consumer machine sends a heartbeat to the Redis end; Figure 5 As shown, the method also includes:

[0058] S310: When monitoring that the first consumer machine sends an exclusive lock to the Redis end, send consumption configuration information corresponding to the target Kafka configuration identifier to the first consumer machine, where the consumption configuration information includes the Kafka server address, the message set key, and the client consumption mode information;

[0059] Specifically, the message set key may be a Kafka topic or partition information, and the client consumption mode may be automatic offset submission, manual submission, etc. The monitoring server may implement a monitoring mechanism for the first consumer machine to send an exclusive lock to the Redis end through a message queue, a REST API call, or other communication mechanisms.

[0060] S320: Start the Kafka client of the first consumer machine to perform a consumption business data operation, and assign a target Kafka configuration identifier to the empty identifier of the first consumer machine.

[0061] Specifically, after receiving the Kafka consumption configuration information, the first consumer machine will configure its own Kafka client according to this information. Then the first consumer machine starts the Kafka client using the provided configuration information and starts listening to the specified Kafka topic or partition. In addition, the first consumer machine consumes messages according to the specified consumption method (such as batch processing, stream processing, etc.) and executes the corresponding business logic. Finally, the first consumer machine can implement the update step from the empty identifier to the target Kafka configuration identifier by updating the local configuration file, database record or other state management mechanism. Generally, once the first consumer machine completes the consumption task or needs to stop consumption, it will release the exclusive lock on the Redis side and update the status or stop the Kafka client as needed. In this way, it is ensured that the first consumer machine can correctly parse and apply the received configuration information and establish a stable connection with the Kafka cluster.

[0062] In some embodiments, the Kafka client of the first consumer machine is started to perform business data operations, such as Figure 6 As shown, the method includes:

[0063] S410: Start the first subthread and the second subthread of the Kafka client;

[0064] Specifically, the first sub-thread is used to communicate with the Kafka cluster and receive business data from Kafka. The second sub-thread is used to process business data and execute business logic, that is, the producer is the first sub-thread and the consumer is the second sub-thread.

[0065] S420: receiving business data through the first sub-thread, and storing the business data in a blocking queue;

[0066] Specifically, the business data received by the first child thread from Kafka can be messages from different topics, and these business data are put into a blocking queue. The blocking queue is a thread-safe queue. When the blocking queue is empty, the operation of taking data out of the queue by the second child thread will be blocked until available data is taken out from the blocking queue.

[0067] S430: Taking out the business data from the blocking queue through the second sub-thread, and executing the corresponding business code according to the business data.

[0068] Specifically, when the monitoring server continuously takes out data from the blocking queue through the second child thread, the monitoring server will execute the corresponding business logic according to each currently taken out data. When performing business data operations, the blocking queue can be used to effectively coordinate the work of the two threads and avoid data competition and condition competition problems.

[0069] S120: When it is detected that the second consumer machine issues an exclusive lock, business data is sent to the second consumer machine alone to complete the switching operation of the consumer machine; the second consumer machine is the first consumer machine among the remaining multiple consumer machines to issue an exclusive lock to the Redis end;

[0070] In this way, it is ensured that business data can only be consumed by a single consumer machine in a distributed environment. When a problem occurs, a disaster recovery switching operation is performed to avoid the problem of incremental data synchronization delay, thus ensuring the normal operation of the business.

[0071] In some embodiments, retrieving the service data on the first consumer machine includes:

[0072] The target Kafka configuration identifier of the first consumer machine is set to a null identifier to stop the consumption business data operation of the Kafka client.

[0073] Specifically, the monitoring server can suspend the Kafka consumer service running on the first consumer machine by sending a stop signal or using a service management tool. Then, the target Kafka configuration identifier on the first consumer machine is set to null by modifying a configuration file or an environment variable, so that the Kafka client of the first consumer machine stops consuming business data.

[0074] For example, when using an environment variable to set the Kafka configuration identifier, the Kafka configuration identifier can be set to an empty string or the environment variable can be deleted. In this way, the business data being consumed on the consuming machine can be effectively recovered, avoiding the problem of incremental data synchronization delay.

[0075] In some embodiments, Figure 7 As shown, the method also includes:

[0076] S510: setting the target Kafka configuration identifier of the first consumer machine and the current identifiers of the remaining consumer machines to empty identifiers to stop all consumer business data operations in the distributed data consumption system;

[0077] Specifically, the monitoring server can perform an active business data switching operation, that is, set the target Kafka configuration identifier of the first consumer machine to empty, and set the Kafka configuration identifiers of the remaining consumer machines to empty, so that all consumer machines stop consuming business data, and all consumer machines have the opportunity to send an exclusive lock to the Redis end to strive for a new Kafka configuration identifier.

[0078] S520: If no consumer machine sends a heartbeat during the current heartbeat renewal period, then during the next heartbeat renewal period, if a third consumer machine is detected to send an exclusive lock to the Redis end, business data is sent to the third consumer machine alone to complete the switching operation of the consumer machine.

[0079] For example, Figure 2 As shown, after all consuming machines stop consuming business data,

[0080] If the monitoring server does not send a heartbeat from any consumer machine within the current 60-second heartbeat renewal period, it needs to send business data to the third consumer machine after entering the next 60-second heartbeat renewal period. In this way, consumption confusion caused by different Kafka configuration identifiers within the same heartbeat renewal period is avoided, ensuring the continuity of business data consumption.

[0081] It should be noted that, with respect to the various steps included in the disaster recovery switching method for a consumer machine provided in any of the above embodiments, unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and these steps can be executed in other orders. Moreover, at least a portion of these steps can include multiple sub-steps or multiple stages, and these sub-steps or stages do not necessarily have to be executed at the same time, but can be executed at different times, and the execution order of these sub-steps or stages does not have to be sequential, but can be executed in turn or alternately with other steps or at least a portion of the sub-steps or stages of other steps.

[0082] According to a second aspect, the present application provides a disaster recovery switching device for a consumer machine, such as Figure 8 As shown, the disaster recovery switching device is applied to a distributed data consumption system including multiple consumption machines, and the disaster recovery switching device includes:

[0083] The service data recovery module 120 is used to recover the service data on the first consumer machine when it is detected that the heartbeat of the first consumer machine stops for a preset heartbeat period, and the first consumer machine is a consumer machine that is currently consuming the service data;

[0084] The business data switching module 130 is used to send business data to the second consumer machine alone to complete the switching operation of the consumer machine when it is monitored that the second consumer machine issues an exclusive lock; the second consumer machine is the first consumer machine to issue an exclusive lock to the Redis end among the remaining multiple consumer machines;

[0085] Among them, when the first consumer machine crashes or an internal program fails, the first consumer machine will stop sending heartbeats.

[0086] In some embodiments, when starting a business project of a distributed data consumption system, the above-mentioned disaster recovery switching device further includes a Kafka identification configuration module 110, which is used to randomly select a consumption machine in the distributed data consumption system as a first consumption machine;

[0087] Instruct the first consumer machine to obtain the Kafka configuration identifier set from the Redis end through subscription;

[0088] Among them, after obtaining the Kafka configuration identifier set, the first consumer machine traverses the Kafka configuration identifier set, extracts one of the target Kafka configuration identifiers to determine the value of the exclusive lock, and sends the exclusive lock to the Redis end.

[0089] In some embodiments, the Kafka identifier configuration module 110 is further used to send consumption configuration information corresponding to the target Kafka configuration identifier to the first consumer machine when monitoring that the first consumer machine sends an exclusive lock to the Redis end, and the consumption configuration information includes the Kafka server address, the message set key and the client consumption mode information;

[0090] The Kafka client of the first consumer machine is started to perform consumption business data operations, and the empty identifier of the first consumer machine is assigned the target Kafka configuration identifier.

[0091] In some embodiments, the Kafka identification configuration module 110 is further used to start the first sub-thread and the second sub-thread of the Kafka client;

[0092] Receive business data through the first child thread and store the business data in a blocking queue;

[0093] The business data is taken out from the blocking queue through the second sub-thread, and the corresponding business code is executed according to the business data.

[0094] In some embodiments, the business data recovery module 120 is further used to set the target Kafka configuration identifier of the first consumer machine to a null identifier to stop the Kafka client's business data consumption operation.

[0095] The business data switching module 130 is further used to set the target Kafka configuration identifier of the first consumer machine and the current identifiers of the remaining consumer machines to empty identifiers to stop all consumer business data operations in the distributed data consumption system;

[0096] If no consumer machine sends a heartbeat during the current heartbeat renewal period, then during the next heartbeat renewal period, if a third consumer machine is detected to send an exclusive lock to the Redis end, business data will be sent to the third consumer machine alone to complete the switching operation of the consumer machine.

[0097] For the specific limitations of the disaster recovery switching device applicable to consumer machines, please refer to the limitations of the disaster recovery switching method applicable to consumer machines in the above text, which will not be repeated here. Each module in the above-mentioned current limiting device applicable to business access can be implemented in whole or in part through software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0098] According to a third aspect, the present application provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the disaster recovery switching method for a consumer machine in any one of the above embodiments are implemented.

[0099] According to a fourth aspect, the present application provides a computer device, such as Fig. 9 As shown, it includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the disaster recovery switching method of a consumer machine in any one of the above embodiments are implemented.

[0100] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Fig. 9As shown. The computer device includes a processor, a memory, a network interface and a database connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store business data. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a disaster recovery switching method for a consumer machine is implemented.

[0101] Among them, any reference to memory, storage, database or other media used in the embodiments provided in the present application may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (RamCUs), direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0102] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0103] The above embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

[0104] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the statement "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

Claims

1. A method for disaster recovery switching of a consumer machine, characterized in that: The method is applied to a distributed data consumption system including multiple consumption machines, and the method comprises: When it is detected that the heartbeat of the first consumer machine stops for a period of time that reaches a preset heartbeat period, the service data on the first consumer machine is retrieved, where the first consumer machine is a consumer machine that is currently consuming service data; When it is monitored that the second consumer machine issues an exclusive lock, the business data is sent to the second consumer machine alone to complete the switching operation of the consumer machine; the second consumer machine is the first consumer machine among the remaining multiple consumer machines to issue an exclusive lock to the remote dictionary service Redis end; When the first consumer machine crashes or an internal program fails, the first consumer machine will stop sending heartbeats.

2. The method according to claim 1, characterized in that When starting a business project of the distributed data consumption system, the method further includes: Randomly selecting a consumption machine in the distributed data consumption system as the first consumption machine; Instruct the first consumer machine to obtain the Kafka configuration identifier set from the Redis end by subscription; Among them, after obtaining the Kafka configuration identifier set, the first consumer machine traverses the Kafka configuration identifier set, extracts one of the target Kafka configuration identifiers to determine the value of the exclusive lock, and sends the exclusive lock to the Redis end.

3. The method according to claim 2, characterized in that Each of the consumer machines is originally set with an empty flag. When the empty flag of a consumer machine is assigned a specific Kafka configuration flag, the consumer machine sends a heartbeat to the Redis end. The method further includes: When monitoring that the first consumer machine sends an exclusive lock to the Redis end, send consumption configuration information corresponding to the target Kafka configuration identifier to the first consumer machine, the consumption configuration information including the Kafka server address, message set key and client consumption mode information; The Kafka client of the first consumer machine is started to perform a consumption business data operation, and the empty identifier of the first consumer machine is assigned the target Kafka configuration identifier.

4. The method according to claim 3, characterized in that Start the Kafka client of the first consumer machine to perform business data operations, including: Start the first subthread and the second subthread of the Kafka client; Receive the service data through the first sub-thread, and store the service data in a blocking queue; The business data is taken out from the blocking queue through the second sub-thread, and the corresponding business code is executed according to the business data.

5. The method according to claim 3, characterized in that: After assigning the target Kafka configuration identifier to the empty identifier of the first consumer machine, the first consumer machine will send a heartbeat to the Redis end according to the set heartbeat cycle; Among them, a single heartbeat renewal period is greater than the heartbeat period; if the heartbeat renewal of the first consumer machine is successful, then within the current heartbeat renewal period, the first consumer machine exclusively occupies the target Kafka configuration identifier.

6. The method according to claim 3, characterized in that Recovering the service data on the first consumer machine includes: The target Kafka configuration identifier of the first consumer machine is set to a null identifier to stop the consumption business data operation of the Kafka client.

7. The method according to claim 6, characterized in that The method further comprises: Setting the target Kafka configuration identifier of the first consumer machine and the current identifiers of the remaining consumer machines to empty identifiers to stop all consumer business data operations in the distributed data consumption system; If no consumer machine sends a heartbeat during the current heartbeat renewal period, then during the next heartbeat renewal period, if a third consumer machine is detected to send the exclusive lock to the Redis end, the business data is sent to the third consumer machine alone to complete the switching operation of the consumer machine.

8. A disaster recovery switching device for a consumer machine, characterized in that: The disaster recovery switching device is applied to a distributed data consumption system including multiple consumption machines, including: A service data recovery module is used to recover service data on a first consumer machine when it is detected that the heartbeat of the first consumer machine stops for a preset heartbeat period, wherein the first consumer machine is a consumer machine that is currently consuming service data; The business data switching module is used to send the business data to the second consumer machine alone to complete the switching operation of the consumer machine when it is monitored that the second consumer machine issues an exclusive lock; the second consumer machine is a consumer machine that first issues an exclusive lock to the remote dictionary service Redis end among the remaining multiple consumer machines; When the first consumer machine crashes or an internal program fails, the first consumer machine will stop sending heartbeats.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

Images