Fault node processing method and device of distributed table system

Abstract

The invention provides a fault node processing method and device for a distributed table system, and relates to the technical field of computers, in particular to the technical field of distributed data storage. The distributed table system comprises a main node and a plurality of data nodes, the plurality of data nodes bear fragment copies of a table, the main fragment copies provide read-write service for the outside, and the method comprises the following steps: carrying out heartbeat check on the plurality of data nodes; detecting own network connectivity in response to the fact that abnormal data nodes exist based on heartbeat examination; in response to the fact that the own network is determined to be normal based on connectivity detection, selecting a main fragment copy borne by the abnormal data node; detecting whether bottom file lock resources of the main fragment copy are released or not; and in response to determining that the underlying file lock resource is released, sending a master switching instruction to a data node where a slave fragment copy corresponding to the master fragment copy is located, so as to switch the slave fragment copy into a new master fragment copy, and removing an abnormal data node from the distributed table system.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of computer, and particularly relates to the technical field of distributed data storage. BACKGROUND

[0002] With the popularity of big data applications, distributed table storage systems have become the key infrastructure supporting core businesses such as Internet search and maps due to their high scalability and high availability. They achieve distributed storage and efficient read and write of massive data through cluster deployment, meeting the core needs of data management for businesses.

[0003] The mainstream distributed table storage system adopts a "master node - data node" architecture. The master node is responsible for maintaining cluster metadata and configuring primary and backup replicas to ensure its own availability. After horizontal sharding of table data, multiple replicas are distributed in data nodes, and only the primary shard replica provides read and write services to the outside. The data nodes are managed by the master node, and the master node applies for and maintains the lease through a heartbeat mechanism. Holding a valid lease is a prerequisite for providing services. When the master node detects a heartbeat loss or lease timeout, it will remove the data node from the available queue and select a new master for the corresponding primary shard. When the data node lease expires and cannot be renewed, it will also actively switch the primary shard to a slave. SUMMARY

[0004] Embodiments of the present disclosure provide a fault node processing method, device, equipment, storage medium and program product of a distributed table system.

[0005] In a first aspect, a fault node processing method of a distributed table system is provided. The method includes: performing feature extraction on a radar echo signal of a user to generate a radar echo feature; performing feature extraction on an audio signal of an environment in which the user is located to generate an audio feature; inputting the radar echo feature and the audio feature into a multi-modal fusion model to output focus state information of the user.

[0006] In a second aspect, a fault node processing device of a distributed table system is provided. The device includes: a first extraction module configured to perform feature extraction on a radar echo signal of a user to generate a radar echo feature; a second extraction module configured to perform feature extraction on an audio signal of an environment in which the user is located to generate an audio feature; and an evaluation module configured to input the radar echo feature and the audio feature into a multi-modal fusion model to output focus state information of the user.

[0007] In a third aspect, an electronic device is provided. The electronic device includes at least one processor and a memory communicatively connected to the at least one processor. The memory stores instructions executable by the at least one processor. The instructions are executed by the at least one processor to enable the at least one processor to perform the method described in the first aspect.

[0008] In a fourth aspect, the embodiments of the present disclosure provide a non-transitory computer-readable storage medium storing computer instructions, the computer instructions being used for causing a computer to execute the method according to the first aspect.

[0009] In a fifth aspect, the embodiments of the present disclosure provide a computer program product comprising a computer program, the computer program being used for implementing the method according to the first aspect when executed by a processor.

[0010] The key or important features of the embodiments of the present disclosure are not used to limit the scope of the present disclosure. Other features of the present disclosure will become apparent from the following description. BRIEF DESCRIPTION OF DRAWINGS

[0011] Other features, objects, and advantages of the present disclosure will become more apparent from the following detailed description of non-limiting embodiments thereof, read in conjunction with the accompanying drawings. The drawings are provided for the purpose of better understanding the present disclosure and are not to be construed as limiting the present disclosure. Among others: Figure 1 is a flowchart of one embodiment of a method for handling a failed node of a distributed table system according to the present disclosure; Figure 2 is a flowchart of another embodiment of a method for handling a failed node of a distributed table system according to the present disclosure; Figure 3 is a flowchart of a process for lease management and master-slave switching of a master node and a data node of a distributed table system; Figure 4 is a structural schematic diagram of one embodiment of an apparatus for handling a failed node of a distributed table system according to the present disclosure; Figure 5 is a structural schematic diagram of another embodiment of an apparatus for handling a failed node of a distributed table system according to the present disclosure; Figure 6 is a block diagram of an electronic device for implementing the method for handling a failed node of a distributed table system according to the embodiments of the present disclosure. DETAILED DESCRIPTION

[0012] Exemplary embodiments of the present disclosure are described below with reference to the accompanying drawings, which include various details of the embodiments of the present disclosure to help in understanding them. These should be considered in their context only. Thus, those of ordinary skill in the art will recognize the various changes and modifications of the embodiments described herein, without departing from the scope and spirit of the present disclosure. Also, descriptions of well-known functions and structures are omitted in the following description for clarity and conciseness.

[0013] It should be noted that the embodiments in the present disclosure and the features in the embodiments can be combined with each other in the case of no conflict. The present disclosure will be described in detail below with reference to the drawings and in combination with the embodiments.

[0014] Figure 1 Flow 100 of one embodiment of a fault node processing method of a distributed table system according to the present disclosure is shown. The fault node processing method of the distributed table system comprises the following steps: Step 101, performing heartbeat checking on the plurality of data nodes.

[0015] In this embodiment, the master node can perform heartbeat checking on the plurality of data nodes.

[0016] The distributed table system can include a master node and a plurality of data nodes. The plurality of data nodes can carry a shard copy of a table, and the master shard copy provides read and write services externally. The master node, as the core management node of the distributed table system, can be responsible for maintaining the meta information of the entire cluster, including but not limited to: table shard distribution, data node state, master-slave shard mapping relationship, and other key contents. The master node can perform periodic heartbeat checking on all data nodes in the cluster to keep abreast of the running state of each data node. The heartbeat mechanism can be a core communication way between the data nodes and the master node. The data nodes can send heartbeat messages to the master node at a preset period (such as seconds) to achieve state synchronization and lease renewal.

[0017] In some embodiments, the specific way in which the master node performs heartbeat checking can include: determining whether the heartbeat message sent by the plurality of data nodes at the preset period is received within a preset heartbeat timeout threshold, the heartbeat message can be used to notify that the corresponding data node is in a normal running state; and / or determining whether the heartbeat renewal information sent by the plurality of data nodes is received before the lease expires, the heartbeat renewal information can be used to apply for extension of the lease validity period.

[0018] The lease is a service permission certificate granted by the master node to the data node, and holding a valid lease is a prerequisite for the data node to provide services externally (especially master shard read and write services), and the data node must complete the renewal through the heartbeat message before the lease expires, otherwise the lease will be invalid. The master node can maintain a heartbeat record table and a lease expiration time for each data node, and through the verification of the above two ways, it can preliminarily judge whether the data node is in a normal communication state.

[0019] Step 102, in response to determining that there is an abnormal data node based on the heartbeat checking, detecting the network connectivity of itself.

[0020] In this embodiment, if it is determined that there is an abnormal data node based on the heartbeat checking, the master node can detect the network connectivity of itself.

[0021] If the master node finds that a data node does not send heartbeat packets within a preset heartbeat timeout threshold or does not complete renewal before the lease expires based on the heartbeat check result, the master node determines that the data node is an abnormal data node. At this time, the master node does not directly determine that the data node is faulty, but first starts a network connectivity detection of the master node itself to exclude the possibility that the abnormality is caused by network problems of the master node itself, so as to avoid misjudgment of the data node state due to network failure of the master node itself.

[0022] In some embodiments, the detection process of the master node on the network connectivity of the master node itself can include: First, a verification node is selected from a preset network verification target list.

[0023] The network verification target list can be composed of stable network nodes in the cluster, such as a cluster gateway, a core switch, a backup master node, or a preset reference data node, to ensure that the detection result can truly reflect the connectivity state of the master node and the cluster network, rather than the reachability of a single node.

[0024] Second, an Internet Control Message Protocol echo request packet is sent to the verification node through a ping mechanism, and a response is waited for within a preset detection timeout threshold (such as 1-2 seconds, optimized for the internal network characteristics of a distributed cluster).

[0025] Third, if Internet Control Message Protocol echo response packets returned by more than a preset number of verification nodes are received (such as 2 or more valid responses when 3 verification nodes are selected), it is determined that the network connectivity of the master node itself is normal; if Internet Control Message Protocol echo response packets returned by no more than a preset number of verification nodes are received, it is determined that the network connectivity of the master node itself is abnormal.

[0026] Using the multi-node verification and threshold judgment method, network state misjudgment caused by failure of a single verification node can be avoided, and the reliability of the detection result can be improved.

[0027] In some embodiments, in response to determining that the network is abnormal based on the connectivity detection, the master node can stop processing the abnormal data node. At this time, the master node determines that the root cause of the abnormality is network failure of the master node itself, rather than actual failure of the data node. If subsequent master switching, node removal, and other operations are continued to be performed, a large number of invalid operations and cluster chaos will be caused. Therefore, the master node only records an abnormal log and terminates all subsequent processing processes for the abnormal data node, and re-performs state verification after the network of the master node itself is restored to normal.

[0028] Step 103, in response to determining that the network is normal based on the connectivity detection, a master shard replica carried by the abnormal data node is selected.

[0029] In this embodiment, if the master node determines that its own network is normal based on connectivity detection, the master node can select the primary shard replica carried by the abnormal data node.

[0030] If the master node confirms its network is normal through network connectivity testing, it rules out communication anomalies caused by its own network factors and determines that the anomaly originates from the data node side or the link between the data node and the master node. At this point, the master node can further verify the actual status of the primary shard replicas on the abnormal data node by randomly selecting at least one primary shard replica from all the primary shard replicas carried by that abnormal data node. Random selection avoids over-reliance on a single primary shard replica, while balancing detection efficiency and result representativeness, eliminating the need to probe all primary shard replicas one by one, thus saving system resources.

[0031] Step 104: Detect whether the underlying file lock resources of the primary shard replica have been released.

[0032] In this embodiment, the master node can detect whether the underlying file lock resources of the primary shard replica have been released.

[0033] After the primary node selects a primary shard replica, it can probe whether the underlying file lock resource corresponding to that primary shard replica has been released. The underlying file lock is the core resource guarantee for the primary shard replica to provide services. The primary shard replica holds the underlying file lock when providing read and write services to prevent data inconsistency. If a data node does fail, the primary shard replica it hosts will release the underlying file lock; if the data node only experiences communication problems but its service is normal, the primary shard replica will continue to hold the underlying file lock. Therefore, by probing the release status of the file lock resource, it is possible to accurately determine whether the primary shard replica is still in a valid service state, providing crucial information for subsequent primary switchover decisions.

[0034] Step 105: In response to determining that the underlying file lock resources have been released, a master switch instruction is sent to the data node where the slave shard replica is located, which is used to switch the slave shard replica to the new master shard replica and remove the abnormal data node from the distributed table system.

[0035] In this embodiment, if it is determined that the underlying file lock resources have been released, the master node can send a master switch instruction to the data node where the slave shard replica is located, which is used to switch the slave shard replica to the new master shard replica and remove the abnormal data node from the distributed table system.

[0036] If the master node detects that the underlying file lock resources of the primary shard replica have been released, it indicates that the primary shard replica has stopped providing service, and the data node is highly likely in a state of actual failure. At this time, the master node can execute a fault recovery procedure, sending a switch master command to the data node containing the corresponding secondary shard replica, explicitly specifying that the secondary shard replica should be switched to the new primary shard replica, ensuring uninterrupted read and write services for that shard. Simultaneously, the abnormal data node is removed from the available data node queue of the distributed table system to prevent subsequent new service requests from being assigned to this failed node.

[0037] In some embodiments, in response to determining that the underlying file lock resource has not been released, the master node can pause the switch to a new master shard replica and continuously monitor the release status of the underlying file lock resource. This indicates that although the abnormal data node is communicating abnormally with the master node, its master shard replica is still holding the file lock and providing services normally. Forcing a switch would result in the old master replica and the new master replica coexisting, causing data inconsistency and unnecessary resource consumption. Therefore, the master node temporarily refrains from performing the switch operation, while continuously monitoring the file lock resource status. The switch process is triggered only after the file lock is released, ensuring the accuracy of the decision.

[0038] The fault node handling method for distributed table systems provided in this disclosure replaces the simple mechanism of "single heartbeat / lease determination" in traditional distributed table systems with a three-layer verification logic of "heartbeat check - self-network verification - primary shard file lock detection". This effectively solves the system jitter problem in special scenarios such as primary node network failure and data node communication abnormalities but normal service. The primary node eliminates its own fault factors through self-network verification and verifies the actual status of the primary shard through file lock detection, avoiding blind primary switching and node removal operations, reducing unnecessary resource consumption, and ensuring the service continuity and stability of the distributed table system. It is especially suitable for core business scenarios with extremely high availability requirements, such as search and maps.

[0039] Figure 2 A flow 200 of another embodiment of a fault node handling method for a distributed table system according to the present disclosure is shown. The fault node handling method for the distributed table system includes the following steps: Step 201: Check the heartbeat communication status between itself and the master node.

[0040] In this embodiment, the data node can check its own heartbeat communication status with the master node.

[0041] This distributed table system can include a master node and multiple data nodes. The multiple data nodes carry sharded replicas of the table, and the master shard replica provides read and write services to the outside world. As the carriers of table shard replicas, data nodes need to periodically communicate with the master node through heartbeats to maintain the validity of leases and inform the master node of their own operating status. Therefore, data nodes can proactively perform self-checks on their own heartbeat communication status with the master node to promptly detect communication anomalies.

[0042] In some embodiments, the method by which a data node checks the heartbeat communication status may include: determining whether it has sent a heartbeat message to the master node according to a preset period, the heartbeat message being used to notify itself that it is in normal operating condition; and / or determining whether it has sent a heartbeat renewal message to the master node before the lease expires, the heartbeat renewal message being used to request an extension of the lease validity period; and / or determining whether it has received feedback information from the master node regarding the heartbeat message or heartbeat renewal message within a preset response threshold.

[0043] The three inspection methods mentioned above comprehensively cover various communication anomaly scenarios, such as data nodes not sending heartbeats, not completing lease renewals, and sending heartbeats but not receiving feedback. This ensures that data nodes can accurately perceive the communication status with the master node and avoid anomalies that may be missed due to a single-dimensional inspection.

[0044] Step 202: In response to determining an abnormal heartbeat based on the heartbeat check, check whether it can normally access the underlying file lock resources.

[0045] In this embodiment, if a heartbeat abnormality is determined based on the heartbeat check, the data node can check whether it is accessing the underlying file lock resources normally.

[0046] If a data node determines through the above checks that its heartbeat communication with the master node is abnormal, it will not directly perform a master-shard switchover. Instead, it will first check whether it can access the underlying file system normally. The underlying file system is the foundation upon which the data node carries sharded replicas and provides read and write services. If the data node can access the underlying file system normally, it means that its core service capabilities are unaffected, and the communication abnormality is only a link problem. If it cannot access the underlying file system normally, it means that the data node itself has a fault and needs to execute the fault handling procedure. This self-check step is crucial for data nodes to avoid blindly switching masters and is also a core component of the anti-jitter solution.

[0047] Step 203: In response to the determination that the underlying file lock resources cannot be accessed normally, the primary shard replica it carries is switched to a secondary shard replica.

[0048] In this embodiment, if it is determined that the underlying file lock resources cannot be accessed normally, the data node can switch its own primary shard replica to a secondary shard replica.

[0049] If a data node determines that it cannot access the underlying file system normally, it means that it has lost the basic conditions for providing read and write services. Continuing to hold the primary shard replica will lead to data read and write failures or data inconsistency. At this time, the data node can proactively switch all the primary shard replicas it hosts to secondary shard replicas and stop providing read and write services to the outside world. This avoids affecting the cluster data consistency due to its own failure, and at the same time provides the premise for the subsequent primary node to reassign a new primary shard replica.

[0050] In some embodiments, in response to determining that access to the underlying file lock resources is normal, a data node can maintain the service state of its primary shard replica and continue to provide read and write services. In this case, the data node determines that the anomaly with the master node is only a communication link anomaly, and its core service capabilities are normal. Blindly switching masters would lead to cluster shard service interruption and resource waste. Therefore, the data node maintains the service state of its primary shard replica and continues to provide read and write services until the lease completely expires or it receives a master-switching instruction from the master node, ensuring the continuity of business services.

[0051] The fault node handling method for a distributed table system provided in this disclosure supplements anti-jitter logic from the perspective of data nodes. Through dual verification of "heartbeat communication check - underlying file system self-check," it achieves accurate differentiation between communication anomalies and service anomalies. In scenarios where only communication is abnormal but service is normal, data nodes continue to provide services, avoiding unnecessary master-slave switching and reducing cluster sharding fluctuations and resource consumption. In scenarios where the data node itself is indeed faulty, it proactively switches to the master, ensuring data consistency. This synergy with the handling method on the master node side comprehensively improves the anti-jitter capability and stability of the distributed table system.

[0052] Figure 3 The diagram illustrates the lease management and shard master-slave switching process of the master node and data nodes in a distributed table system. The diagram clearly shows the interaction between the client 301, the master node 302, and the data node 303 in the distributed table system, as well as the core process logic of lease management, heartbeat communication, and shard master-slave switching.

[0053] Client 301, acting as the initiator of data read / write requests, sends a request to master node 302 through the metadata query interface. After obtaining the location information of the data node 303 where the primary shard replica of the target data is located, it sends read / write commands to the corresponding data node 303 through the data read / write request interface. Master node 302, as the cluster control core, contains primary and backup replicas to ensure its high availability. It also receives heartbeat messages from data nodes 303 through the heartbeat collection module, manages the lease status of each data node 303 through the lease timeout check module, and is responsible for maintaining metadata. The data (including table sharding distribution, master-slave sharding mapping, etc.) can transmit sharding switching instructions to data nodes 303 through the master switch instruction sending interface; multiple data nodes 303 carry master-slave sharded replicas of the table, where the master sharded replica provides read and write services to the outside world, and the slave sharded replica serves as a redundant backup. The master and slave sharded replicas share the storage resources of the underlying distributed file system 304. The data node 303 ensures the read and write uniqueness of the master sharded replica by holding the underlying storage lock. At the same time, it has a built-in lease timeout check module and a self-service status determination module, and can autonomously perform master-slave sharding switching operations.

[0054] This diagram fully illustrates the core interaction flow of the disclosed technical solution: Master node 302 senses the status of data node 303 through heartbeat collection and lease timeout checks; data node 303 maintains its service qualification through heartbeat sending and lease renewal. When master node 302 detects an anomaly in data node 303's heartbeat / lease, it first eliminates its own fault through network connectivity checks (such as ping mechanisms), and then performs underlying file lock detection on the primary shard replica of the abnormal data node 303. When data node 303 detects an anomaly in communication with the master node, it performs an underlying storage access self-check. Both parties perform differentiated operations based on their respective verification results: if master node 302 detects that the file lock has been released, it issues a master switch command; otherwise, it suspends operations. If data node 303 can access the underlying storage normally, it maintains the primary shard service; if it cannot access it, it actively switches to slave. Ultimately, this achieves stable cluster operation and fault tolerance, intuitively demonstrating the core anti-jitter design of "two-way verification + differentiated decision-making."

[0055] Further reference Figure 4 As an implementation of the methods shown in the above figures, this disclosure provides an embodiment of a fault node processing device for a distributed table system. This device embodiment is similar to... Figure 1 Corresponding to the method embodiments shown, this device can be specifically applied to various electronic devices.

[0056] like Figure 4As shown, the distributed table system in this embodiment may include a master node and multiple data nodes. The multiple data nodes carry sharded replicas of the table, and the master shard replica provides read and write services to the outside world. The fault node handling device 400 of the distributed table system may include: a heartbeat check module 401, a network detection module 402, a master shard selection module 403, a resource detection module 404, and a command sending module 405. The system includes a heartbeat check module 401 configured to perform heartbeat checks on multiple data nodes; a network detection module 402 configured to detect its own network connectivity in response to the heartbeat check determining the presence of an abnormal data node; a primary shard selection module 403 configured to select a primary shard replica carried by an abnormal data node in response to the connectivity detection determining that its own network is normal; a resource detection module 404 configured to detect whether the underlying file lock resources of the primary shard replica have been released; and an instruction sending module 405 configured to send a master switch instruction to the data node where the slave shard replica corresponding to the primary shard replica is located in response to the determination that the underlying file lock resources have been released, in order to switch the slave shard replica to a new primary shard replica and remove the abnormal data node from the distributed table system.

[0057] In this embodiment, the specific processing and technical effects of the heartbeat check module 401, network detection module 402, master shard selection module 403, resource detection module 404, and command sending module 405 in the fault node processing device 400 of the distributed table system can be found in the following references. Figure 1 The relevant descriptions of steps 101-105 in the corresponding embodiments will not be repeated here.

[0058] In some optional implementations of this embodiment, the heartbeat check module 401 is further configured to: determine whether multiple heartbeat messages sent by multiple data nodes according to a preset period are received within a preset heartbeat timeout threshold, the heartbeat messages being used to notify the corresponding data nodes that they are in normal operating condition; and / or determine whether multiple heartbeat renewal information sent by multiple data nodes is received before the lease expires, the heartbeat renewal information being used to apply for an extension of the lease validity period.

[0059] In some optional implementations of this embodiment, the network detection module 402 is further configured to: select verification nodes from a preset network verification target list; send Internet Control Message Protocol (ICP) echo request messages to the verification nodes via a ping mechanism, and wait for a response within a preset detection timeout threshold; if more than a preset number of ICP echo response messages are received from verification nodes, it is determined that its own network connectivity is normal; if no more than a preset number of ICP echo response messages are received from verification nodes, it is determined that its own network connectivity is abnormal.

[0060] In some optional implementations of this embodiment, the fault node processing device 400 of the distributed table system further includes: a switching pause module, configured to pause the switching of the primary shard replica to a new primary shard replica in response to determining that the underlying file lock resource has not been released, and to continuously monitor the release status of the underlying file lock resource.

[0061] In some optional implementations of this embodiment, the fault node processing device 400 of the distributed table system further includes: a processing stop module, configured to stop processing the abnormal data node in response to determining that its own network is abnormal based on connectivity detection.

[0062] Further reference Figure 5 As an implementation of the methods shown in the above figures, this disclosure provides an embodiment of a fault node processing device for a distributed table system. This device embodiment is similar to... Figure 2 Corresponding to the method embodiments shown, this device can be specifically applied to various electronic devices.

[0063] like Figure 5 As shown, the distributed table system in this embodiment may include a master node and multiple data nodes. The multiple data nodes carry sharded replicas of the table, and the master shard replica provides read and write services to the outside world. The fault node handling device 500 of the distributed table system may include: a heartbeat check module 501, a resource check module 502, and a shard switching module 503. The heartbeat check module 501 is configured to check its own heartbeat communication status with the master node; the resource check module 502 is configured to check whether it can normally access the underlying file lock resources in response to determining an abnormal heartbeat based on the heartbeat check; the shard switching module 503 is configured to switch its own master shard replica to a slave shard replica in response to determining that it cannot normally access the underlying file lock resources.

[0064] In this embodiment, the specific processing of the heartbeat check module 501, resource check module 502, and shard switching module 503 in the fault node processing device 500 of the distributed table system, and the resulting technical effects, can be found in the following references: Figure 2 The relevant descriptions of steps 201-203 in the corresponding embodiments will not be repeated here.

[0065] In some optional implementations of this embodiment, the heartbeat check module 501 is further configured to: determine whether a heartbeat message has been sent to the master node according to a preset period, the heartbeat message being used to notify itself that it is in normal operating condition; and / or determine whether a heartbeat renewal message has been sent to the master node before the lease expires, the heartbeat renewal message being used to apply for an extension of the lease validity period; and / or determine whether feedback information from the master node regarding the heartbeat message or heartbeat renewal message has been received within a preset response threshold.

[0066] In some optional implementations of this embodiment, the fault node processing device 500 of the distributed table system further includes: a state maintenance module, configured to maintain the service state of the primary shard replica it carries and continue to provide read and write services to the outside world in response to determining that normal access to the underlying file lock resources is possible.

[0067] This disclosure provides a distributed table system, which may include a master node and multiple data nodes. The master node can execute... Figure 1 The described method allows data nodes to execute... Figure 2 The method described.

[0068] The acquisition, storage, and application of user personal information involved in the technical solution disclosed herein comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0069] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0070] Figure 6 A schematic block diagram of an example electronic device 600 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0071] like Figure 6 As shown, device 600 includes a computing unit 601, which can perform various appropriate actions and processes based on a computer program stored in read-only memory (ROM) 602 or a computer program loaded into random access memory (RAM) 603 from storage unit 608. RAM 603 may also store various programs and data required for the operation of device 600. The computing unit 601, ROM 602, and RAM 603 are interconnected via bus 604. Input / output (I / O) interface 605 is also connected to bus 604.

[0072] Multiple components in device 600 are connected to I / O interface 605, including: input unit 606, such as keyboard, mouse, etc.; output unit 607, such as various types of monitors, speakers, etc.; storage unit 608, such as disk, optical disk, etc.; and communication unit 609, such as network card, modem, wireless transceiver, etc. Communication unit 609 allows device 600 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0073] The computing unit 601 can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 601 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 601 performs the various methods and processes described above, such as the fault node handling method for a distributed table system. For example, in some embodiments, the fault node handling method for a distributed table system can be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 608. In some embodiments, part or all of the computer program can be loaded and / or installed on device 600 via ROM 602 and / or communication unit 609. When the computer program is loaded into RAM 603 and executed by the computing unit 601, one or more steps of the fault node handling method for a distributed table system described above can be performed. Alternatively, in other embodiments, the computing unit 601 may be configured to perform a fault node handling method for a distributed table system by any other suitable means (e.g., by means of firmware).

[0074] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0075] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0076] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0077] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0078] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0079] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, distributed system servers, or servers incorporating blockchain technology.

[0080] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution provided in this disclosure can be achieved, and this is not limited herein.

[0081] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A method for handling faulty nodes in a distributed table system, the distributed table system comprising a master node and multiple data nodes, the multiple data nodes carrying sharded replicas of the table, and the master shard replica providing read and write services to the outside world, the method comprising: Perform heartbeat checks on the multiple data nodes; In response to the detection of abnormal data nodes based on heartbeat checks, the system checks its own network connectivity. In response to determining that its own network is normal based on connectivity detection, the primary shard replica carried by the abnormal data node is selected; Detect whether the underlying file lock resources of the primary shard replica have been released; In response to determining that the underlying file lock resource has been released, a switch master instruction is sent to the data node where the slave shard replica corresponding to the primary shard replica is located, to switch the slave shard replica to the new primary shard replica, and to remove the abnormal data node from the distributed table system.

2. The method according to claim 1, wherein, The heartbeat check of the plurality of data nodes includes: Determine whether a heartbeat message sent by the plurality of data nodes according to a preset period is received within a preset heartbeat timeout threshold, wherein the heartbeat message is used to notify the corresponding data node that it is in normal operating condition; and / or Determine whether heartbeat renewal information is received from the multiple data nodes before the lease expires, and the heartbeat renewal information is used to apply for an extension of the lease validity period.

3. The method according to claim 1, wherein, The detection of its own network connectivity includes: Select a verification node from the preset list of network verification targets; The system sends an Internet Control Message Protocol echo request message to the verification node via the ping mechanism and waits for a response within a preset probe timeout threshold. If more than a preset number of Internet Control Message Protocol echo response messages are received from verification nodes, it is determined that the network connectivity is normal. If no more than a preset number of verification nodes return Internet Control Message Protocol echo response messages, it is determined that its own network connectivity is abnormal.

4. The method according to claim 1, wherein, The method further includes: In response to the determination that the underlying file lock resource has not been released, the switching of the primary shard replica to a new primary shard replica is paused, and the release status of the underlying file lock resource is continuously monitored.

5. The method according to claim 1, wherein, The method further includes: In response to determining that its own network is abnormal based on connectivity detection, the processing of the abnormal data node is stopped.

6. A method for handling faulty nodes in a distributed table system, the distributed table system comprising a master node and multiple data nodes, the multiple data nodes carrying sharded replicas of the table, and the master shard replica providing read and write services to the outside world, the method comprising: Check the heartbeat communication status between itself and the master node; In response to heartbeat abnormality detected by heartbeat check, check whether it can access the underlying file lock resources normally; In response to the determination that the underlying file lock resource cannot be accessed normally, the primary shard replica it carries is switched to a secondary shard replica.

7. The method according to claim 6, wherein, The process of checking the heartbeat communication status between itself and the master node includes: Determine whether a heartbeat message has been sent to the master node according to a preset period, the heartbeat message being used to notify itself that it is in normal operating condition; and / or Determine whether a heartbeat renewal message has been sent to the master node before the lease expires, the heartbeat renewal message being used to request an extension of the lease validity period; and / or Determine whether feedback information regarding the heartbeat message or the heartbeat renewal information is received from the master node within a preset response threshold.

8. The method according to claim 6, wherein, The method further includes: In response to the confirmation that normal access to the underlying file lock resources is possible, the service status of the primary shard replica it carries is maintained, and read and write services are continued to be provided to the outside world.

9. A fault node handling device for a distributed table system, the distributed table system including a master node and multiple data nodes, the multiple data nodes carrying sharded replicas of the table, and the master shard replica providing read and write services to the outside world, the device comprising: The heartbeat check module is configured to perform heartbeat checks on the plurality of data nodes; The network detection module is configured to detect its own network connectivity in response to the detection of abnormal data nodes based on heartbeat checks; The primary shard selection module is configured to select the primary shard replica carried by the abnormal data node in response to determining that its own network is normal based on connectivity detection; The resource detection module is configured to detect whether the underlying file lock resources of the primary shard replica have been released; The instruction sending module is configured to, in response to determining that the underlying file lock resource has been released, send a switch instruction to the data node where the slave shard replica corresponding to the primary shard replica is located, for switching the slave shard replica to a new primary shard replica and removing the abnormal data node from the distributed table system.

10. A fault node handling device for a distributed table system, the distributed table system including a master node and multiple data nodes, the multiple data nodes carrying sharded replicas of the table, and the master shard replica providing read and write services to the outside world, the device comprising: The heartbeat check module is configured to check its own heartbeat communication status with the master node; The resource check module is configured to check whether it can access the underlying file lock resources normally in response to a heartbeat abnormality determined by a heartbeat check. The shard switching module is configured to switch its own primary shard replica to a secondary shard replica in response to determining that it cannot access the underlying file lock resource normally.

11. A distributed table system, comprising a master node and multiple data nodes, wherein the master node performs the method of any one of claims 1-5, and the data nodes perform the method of any one of claims 6-8.

12. An electronic device, comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, which, when executed by the at least one processor, enables the at least one processor to perform the method of any one of claims 1-5 or 6-8.

13. A non-transitory computer-readable storage medium storing computer instructions for causing the computer to perform the method of any one of claims 1-5 or 6-8.

14. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1-5 or 6-8.

Images