Node Expansion System and Method

Abstract

The present application provides a node expansion system and method. The system includes: a management node for responding to a node expansion instruction and switching the node states of a target number of standby nodes to a preparatory state; a standby node for, when in the preparatory state, obtaining a plurality of first messages written by a master node, and during the process of obtaining the plurality of first messages, caching a second message corresponding to a message writing request when receiving the message writing request; when there is a first message consistent with the second message among the plurality of first messages, releasing the second message, and when the remaining message quantity cached by the standby node is lower than a first quantity threshold, sending a first notification message to the management node; the management node is further used for switching the standby node that sends the first notification message from the preparatory state to an online state to serve as the master node for message writing; through the present application, the node expansion process can be made imperceptible and the stability of node expansion can be improved.

Description

Technical Field

[0001] This application relates to the field of information processing technologies, and in particular, to a node expansion system and method. Background Art

[0002] In related technologies, when expanding the main node in a distributed system (such as a distributed storage and computing service system) (for example, if the original number of main nodes is N and it needs to be expanded to 2N), it is necessary to first prepare new node devices, then perform data backup, and migrate the data after backup to the prepared node devices one by one to achieve expansion.

[0003] However, in this way of expanding the main node, if there is network lag, it is very likely that data backup and migration will fail, resulting in expansion failure. Moreover, during the expansion process, due to the need to perform data backup and migration and other operations between each node (including the original main node and the newly prepared nodes), the message writing requests on the business side cannot be completed normally, affecting normal use. Summary of the Invention

[0004] Embodiments of this application provide a node expansion system, method, device, equipment, storage medium, and computer program product, which can achieve a seamless node expansion process and improve the stability of node expansion.

[0005] The technical solution of the embodiments of this application is implemented as follows:

[0006] Embodiments of this application also provide a node expansion system, including: a management node, a main node, and at least one standby node, where

[0007] The management node is configured to, in response to a node expansion instruction, switch the node states of a target number of standby nodes to a standby state;

[0008] The main node is configured to perform message writing when receiving a message writing request;

[0009] The standby node is configured to, when in the standby state, obtain a plurality of first messages written by the main node, and during the process of obtaining the plurality of first messages, when receiving a message writing request, cache the second message corresponding to the message writing request;

[0010] When there is a first message among the plurality of first messages that is the same as the second message, release the second message, and when the remaining number of messages cached by the standby node is lower than a first quantity threshold, send a first notification message to the management node;

[0011] The management node is further configured to, when receiving the first notification message, switch the standby node that sends the first notification message from the standby state to the online state to write messages as the primary node.

[0012] An embodiment of the present application further provides a node expansion method, which is applied to a node expansion system including a management node, a primary node, and at least one standby node; the method includes:

[0013] A standby node in the standby state obtains a plurality of first messages written by the primary node. The standby node in the standby state is obtained by the management node in response to a node expansion instruction to switch the states of a target number of standby nodes.

[0014] During the process of obtaining the plurality of first messages, when a message write request is received, cache the second message corresponding to the message write request.

[0015] When there is a first message in the plurality of first messages that is the same as the second message, release the second message, and when the remaining number of messages cached by the standby node is lower than a first quantity threshold, send a first notification message to the management node, so that

[0016] The management node, based on the first notification message, switches the standby node that sends the first notification message from the standby state to the online state to write messages as the primary node.

[0017] An embodiment of the present application further provides a node expansion device, which is applied to a node expansion system including a management node, a primary node, and at least one standby node; the device includes:

[0018] An obtaining module, configured to enable a standby node in the standby state to obtain a plurality of first messages written by the primary node. The standby node in the standby state is obtained by the management node in response to a node expansion instruction to switch the states of a target number of standby nodes.

[0019] A caching module, configured to cache the second message corresponding to the message write request when a message write request is received during the process of obtaining the plurality of first messages.

[0020] A sending module, configured to release the second message when there is a first message in the plurality of first messages that is the same as the second message, and send a first notification message to the management node when the remaining number of messages cached by the standby node is lower than a first quantity threshold, so that the management node, based on the first notification message, switches the standby node that sends the first notification message from the standby state to the online state to write messages as the primary node.

[0021] An embodiment of the present application further provides an electronic device, including:

[0022] A memory for storing executable instructions;

[0023] A processor, when executing the executable instructions stored in the memory, implements the node expansion method provided by the embodiment of the present application.

[0024] An embodiment of the present application further provides a computer-readable storage medium storing executable instructions, which, when executed by a processor, implement the node expansion method provided by the embodiment of the present application.

[0025] An embodiment of the present application further provides a computer program product, including a computer program or instructions, which, when executed by a processor, implement the node expansion method provided by the embodiment of the present application.

[0026] The embodiment of the present application has the following beneficial effects:

[0027] Applying the embodiment of the present application, when the management node receives a node expansion instruction, it controls a target number of standby nodes to be in a standby state. The primary node writes messages based on a message write request. The standby nodes in the standby state obtain multiple first messages written by the primary node, and during the process of obtaining the multiple first messages, cache the second messages corresponding to the received message write requests. When a standby node determines that there is a first message in the obtained first messages that is the same as the second message, it releases the second message, and when the remaining number of messages cached by the standby node is lower than a first quantity threshold, it sends a first notification message to the management node. Based on the received first notification message, the management node switches the standby node that sent the first notification message from the standby state to the online state, so that the standby node can write messages as the primary node, thereby realizing the expansion of the primary node.

[0028] Here, during the entire primary node expansion process, the message write requests of the primary node are uninterrupted and normal writing is maintained. The node expansion process is imperceptible to the business side. And the standby nodes obtain the first messages after the primary node writes the first messages corresponding to the message write requests, realizing message synchronization of the standby nodes relative to the primary node. When there is a message in the obtained first messages that is the same as the second message cached for the message write request, the second message is released, and based on whether the remaining quantity of the second message is lower than the quantity threshold (if it is lower, it means that the message synchronization of the standby node relative to the primary node is completed and it can be used as the primary node) to determine whether the standby node can be used as the primary node, avoiding the problem of expansion failure caused by data backup and migration failure, and improving the stability and availability of node expansion. Description of the Drawings

[0029] Figure 1It is a schematic architecture diagram of the node expansion system 100 provided by an embodiment of the present application;

[0030] Figure 2 It is a schematic structural diagram of the electronic device 500 for implementing the node expansion method provided by an embodiment of the present application;

[0031] Figure 3 It is a schematic diagram of the interaction process between nodes in the node expansion system provided by an embodiment of the present application;

[0032] Figure 4 It is a schematic flowchart of the node expansion method provided by an embodiment of the present application;

[0033] Figure 5 It is a schematic architecture diagram of the distributed computing and storage system provided by an embodiment of the present application;

[0034] Figure 6 It is a schematic diagram of state 1 in the main node expansion process provided by an embodiment of the present application;

[0035] Figure 7 It is a schematic diagram of state 2 in the main node expansion process provided by an embodiment of the present application;

[0036] Figure 8 It is a schematic diagram of state 3 in the main node expansion process provided by an embodiment of the present application;

[0037] Figure 9 It is a schematic diagram of state 4 in the main node expansion process provided by an embodiment of the present application;

[0038] Figure 10 It is a schematic diagram of state 5 in the main node expansion process provided by an embodiment of the present application;

[0039] Figure 11 It is a schematic diagram of state 6 in the main node expansion process provided by an embodiment of the present application. Detailed implementation manners

[0040] In order to make the objectives, technical solutions, and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings. The described embodiments should not be regarded as limitations on the present application. All other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the scope of protection of the present application.

[0041] In the following description, reference is made to "some embodiments", which describe a subset of all possible embodiments. However, it can be understood that "some embodiments" can be the same subset or different subsets of all possible embodiments, and can be combined with each other without conflict.

[0042] In the following description, the terms "first", "second", and "third" only distinguish similar objects and do not represent a specific order for the objects. It can be understood that "first", "second", and "third" can be interchanged in a specific order or sequence when permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0043] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those of ordinary skill in the technical field to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0044] Before further elaborating on the embodiments of this application, the nouns and terms involved in the embodiments of this application are described. The nouns and terms involved in the embodiments of this application are subject to the following explanations.

[0045] 1) Responsive to, which is used to represent the conditions or states on which the executed operations depend. When the dependent conditions or states are met, one or more executed operations can be real-time or can have a set delay; without special instructions, there is no restriction on the execution order of the multiple executed operations.

[0046] Based on the above explanations of the nouns and terms involved in the embodiments of this application, the node expansion system provided by the embodiments of this application is described below. See Figure 1 , Figure 1 FIG. is a schematic architecture diagram of the node expansion system 10 provided by the embodiments of this application. The node expansion system includes: a management node, a master node, and at least one standby node. To support an exemplary application, the terminal 100 is connected to the management node 300 through the network 200, and each node device is also connected through the network. For example, the management node 300, the master node 400, and the standby nodes (the standby node 500-1 and the standby node 500-2 are exemplarily shown) are also connected through the network. The network 200 can be a wide area network or a local area network, or a combination of the two, and uses wireless or wired links to achieve data transmission.

[0047] The terminal 100 is configured to send a node expansion instruction to the management node 300 in response to a node expansion operation triggered by a user;

[0048] The management node 300 is configured to switch the node states of a target number of standby nodes to a standby state in response to the node expansion instruction;

[0049] The master node 400 is configured to perform message writing when receiving a message writing request;

[0050] A standby node (such as standby node 500-1) is used to obtain multiple first messages written by the master node when in a standby state. During the process of obtaining the multiple first messages, when a message writing request is received, the second message corresponding to the message writing request is cached; when there is a first message in the multiple first messages that is the same as the second message, the second message is released, and when the remaining number of messages cached by the standby node is lower than the first quantity threshold, a first notification message is sent to the management node; wherein, the first notification message is used to indicate that the standby node has the ability to be switched to an online state.

[0051] The management node 303 is further configured to, when receiving the first notification message, switch the standby node that sends the first notification message from the standby state to the online state to act as the master node for message writing.

[0052] In practical applications, each node device (such as the management node 300, the standby node, or the master node 400) can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The terminal 100 can be a smart phone, a tablet computer, a notebook computer, a desktop computer, a smart speaker, a smart TV, a smart watch, etc., but is not limited thereto. The terminal and the node device can be directly or indirectly connected through wired or wireless communication methods, and this application does not make any restrictions here.

[0053] See Figure 2 , Figure 2 FIG. is a schematic structural diagram of an electronic device 600 for implementing the node expansion method provided by an embodiment of the present application. In practical applications, the electronic device 600 can be Figure 1 each of the node devices shown, taking the electronic device 600 as Figure 1 the standby node shown as an example to illustrate the electronic device for implementing the node expansion method of the embodiment of the present application. The electronic device 600 provided by the embodiment of the present application includes: at least one processor 610, a memory 650, at least one network interface 620, and a user interface 630. Each component in the electronic device 600 is coupled together through a bus system 640. It can be understood that the bus system 640 is used to realize the connection and communication between these components. The bus system 640 includes, in addition to the data bus, a power bus, a control bus, and a status signal bus. However, for the sake of clear illustration, in Figure 2 all kinds of buses are labeled as the bus system 640.

[0054] The processor 610 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., where the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0055] The user interface 630 includes one or more output devices 631 that enable presentation of media content, including one or more speakers and / or one or more visual display screens. The user interface 630 also includes one or more input devices 632, including user interface components that facilitate user input, such as a keyboard, mouse, microphone, touch screen display, camera, other input buttons and controls.

[0056] The memory 650 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state memory, hard drives, optical drives, etc. The memory 650 may optionally include one or more storage devices that are physically remote from the processor 610.

[0057] The memory 650 includes volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory may be a read-only memory (ROM), and the volatile memory may be a random access memory (RAM). The memory 650 described in the embodiments of the present application is intended to include any suitable type of memory.

[0058] In some embodiments, the memory 650 can store data to support various operations, examples of which include programs, modules, and data structures, or a subset or superset thereof, as exemplified below.

[0059] Operating system 651, including system programs for processing various basic system services and performing hardware-related tasks, such as the framework layer, core library layer, and driver layer, which are used to implement various basic services and process hardware-based tasks;

[0060] A network communication module 652 for reaching other computing devices via one or more (wired or wireless) network interfaces 620 , exemplary network interfaces 620 including Bluetooth, WiFi, and USB;

[0061] A presentation module 653 for enabling presentation of information (e.g., a user interface for operating a peripheral device and displaying content and information) via one or more output devices 631 associated with the user interface 630 (e.g., a display screen, a speaker, etc.);

[0062] An input processing module 654 for detecting and translating one or more user inputs or interactions from one of one or more input devices 632.

[0063] In some embodiments, the node expansion device provided in the embodiments of the present application may be implemented in software. Figure 2 Shown is a node expansion device 655 stored in the memory 650, which may be software in the form of a program and a plug-in, etc., including the following software modules: an acquisition module 6551, a cache module 6552, and a sending module 6553. These modules are logical, and thus can be combined arbitrarily or further split according to the functions to be implemented. The functions of each module will be described below.

[0064] In other embodiments, the node expansion device provided in the embodiments of the present application may be implemented in a combination of software and hardware. As an example, the node expansion device provided in the embodiments of the present application may be a processor in the form of a hardware decoding processor, which is programmed to execute the node expansion method provided in the embodiments of the present application. For example, the processor in the form of a hardware decoding processor may employ one or more application specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field programmable gate arrays (FPGAs), or other electronic components.

[0065] In some embodiments, the standby node may implement the node expansion method provided in the embodiments of the present application by running a computer program. For example, the computer program may be a native program or software module in an operating system; it may be a native application (APP). In short, the above computer program may be any form of application program, module, or plug-in.

[0066] Next, a detailed description will be given of the node expansion system provided in the embodiments of the present application. The node expansion system provided in the embodiments of the present application includes: a management node, a primary node, and at least one standby node. Refer to Figure 3 ,Figure 3 It is a schematic diagram of the interaction process among nodes in the node expansion system provided by an embodiment of the present application. The interaction process among nodes in the node expansion system provided by an embodiment of the present application includes:

[0067] Step 101: In response to a node expansion instruction, the management node switches the node states of the target number of standby nodes to the preparatory state.

[0068] Here, in practical applications, the node expansion system can be a storage computing service system for carrying read and write requests from the business side (such as message requests, information storage, etc.). For example, it responds to write requests to write messages and responds to read requests to return corresponding data information. In actual implementation, the above-mentioned read and write requests are mainly carried out by node devices in the system, and the node devices mostly exist based on the architecture of a master node + standby node. In an embodiment of the present application, there is a corresponding standby node for each master node, and the standby node is equivalent to the mirror node of the corresponding master node to achieve data synchronization between the master node and the standby node. Based on this, the read and write requests in the embodiment of the present application are separated, that is, messages are written through the master node to respond to write requests, and messages are read through the standby node to respond to read requests. In this way, the master node does not need to carry read requests. When there is a need to expand the master node, while expanding the master node, the processing of read requests is not affected, making the read requests uninterrupted. For the business side and users, the effect of no perception of the read request service during the expansion process of the master node is achieved.

[0069] It should be noted that the standby nodes in a mirror relationship with the master node here are other standby nodes in the system except for the above-mentioned target number of standby nodes. The above-mentioned target number of standby nodes refers to specific standby nodes. For example, if the target number is 4 standby nodes A, B, C, and D, then the subsequent target number of standby nodes specifically refers to these standby nodes A, B, C, and D.

[0070] Here, the node expansion system provided by an embodiment of the present application is applied to the expansion of the master node in the system. Next, the node expansion system provided by an embodiment of the present application will be described in detail based on the expansion process of the master node.

[0071] In some embodiments, the node expansion system further includes a management node, which is mainly responsible for managing the node status of each node device, the selection of the master node, heartbeat maintenance, and so on. In the embodiments of the present application, when the user has a demand for expanding the master node, for example, when the master node service is overloaded and the master node needs to be expanded, the user can trigger a node expansion instruction for the master node. This node expansion instruction acts on the management node, and the expansion of the master node is achieved through the management node. When the management node receives this node expansion instruction, it searches for standby nodes that can be used for expanding the master node in the system, so as to expand the master node by converting the standby nodes into master nodes. In practical applications, the node expansion instruction may also include the number of standby nodes required for expanding the master node, or the number of nodes to be reached by the master node after expansion, etc., so that the management node can achieve the master node expansion task corresponding to the node expansion instruction by controlling the corresponding number of standby nodes.

[0072] When the management node determines that a target number of standby nodes are needed to expand the master node, the management node switches the node status of the target number of standby nodes to the preparatory state, that is, the preparatory state as the master node. In actual implementation, the management node can send a control instruction for instructing the corresponding standby nodes to switch their node status to the preparatory state. When the standby nodes receive this control instruction, they control their own node status to be switched from the current state (such as the offline state) to the preparatory state, so as to wait for subsequent operations to achieve the conversion from the standby nodes to the master node and achieve the purpose of expanding the master node. For example, if the node expansion instruction indicates that the number of nodes after expanding the master node is 2, and the original number of master nodes is 1, at this time, if the original master node needs to be taken offline after expansion, the target number of standby nodes required for expansion is 2; if the original master node does not need to be taken offline after expansion, the target number of standby nodes required for expansion is 1.

[0073] It should be noted that the target number of standby nodes refers to specific standby nodes. For example, if the target number of standby nodes is 4, namely standby nodes A, B, C, and D, then the subsequent target number of standby nodes specifically refers to these standby nodes A, B, C, and D.

[0074] In some embodiments, during the process of expanding the above standby node to a primary node, before the standby node enters the standby state, it may also be in a trial state, so as to switch to the standby state after completing the target operation in the trial state. Based on this, the management node is further configured to, in response to a node expansion instruction, switch the node states of a target number of standby nodes to the trial state; the standby node is further configured to, when in the trial state, obtain a plurality of first messages written by the primary node, and during the process of obtaining the plurality of first messages, when it is determined that the number of unobtained first messages is lower than a second quantity threshold, send a second notification message to the management node; the management node is further configured to receive the second notification message and switch the standby node that sent the second notification message from the trial state to the standby state. Wherein, the second notification message is used to indicate that the standby node in the trial state has the ability to be switched to the standby state;

[0075] Here, before controlling a target number of standby nodes to be in the standby state, the management node may also, in response to a node expansion instruction, switch the node states of a target number of standby nodes to the trial state. Specifically, the management node may send a control instruction for instructing the corresponding standby node to switch its node state to the trial state. When the standby node receives the control instruction, it controls its own node state to switch from the current state (such as the offline state) to the trial state. It should be noted that the target number of standby nodes refers to specific standby nodes. For example, if the target number is 4 standby nodes A, B, C, and D, then the subsequent target number of standby nodes specifically refers to these standby nodes A, B, C, and D.

[0076] Correspondingly, when the standby nodes with the target number are in the trial state, they need to obtain a plurality of first messages written by the primary node (i.e., the original primary node). The plurality of first messages include historical messages written before the node expansion instruction is triggered and messages written in real time after the node expansion instruction is triggered (i.e., during the node expansion process). During the obtaining process, specifically, they are obtained in the order of message writing, that is, in the order of the message writing time points, from the earliest to the latest. In this way, during the primary node expansion process, write requests can be uninterrupted, and for the business side and users, the effect of being unaware of the write request service during the primary node expansion process can be achieved. Here, the standby nodes in the trial state obtain a plurality of first messages written by the primary node (i.e., the original primary node) to catch up on messages, so that the standby nodes can synchronize the messages written in the original primary node to themselves to avoid information loss during the primary node expansion. In actual implementation, the plurality of first messages written by the primary node (i.e., the original primary node) can be stored in the primary node's own storage or in other node devices dedicated to data storage.

[0077] During the process of the standby node obtaining multiple first messages, it determines in real time or periodically the number of remaining first messages that have not been obtained. When it is determined that the number of unobtained first messages is lower than the second quantity threshold, it indicates that the standby node itself is eligible to switch from the current trial state to the standby state. At this time, the standby node sends a second notification message to the management node, and this second notification message is used to indicate that the standby node in the trial state has the ability to be switched to the standby state.

[0078] After receiving the above-mentioned second notification message sent by the standby node, the management node, based on this second notification message, switches the standby node that sent the second notification message from the trial state to the standby state. Here, the standby node can obtain the first messages written by the primary node and write them into itself in the trial state to achieve the synchronization of the messages written in the original primary node; at the same time, the standby node in the trial state does not need to receive message write requests and perform corresponding message caching operations, so that the data synchronization of the primary node can be carried out quickly and efficiently to ensure the smooth progress of the expansion of the standby node to the primary node.

[0079] In some embodiments, the node expansion system further includes a proxy node. Correspondingly, the management node is further configured to send status synchronization information to the proxy node when the standby node undergoes a state switch; the proxy node is configured to receive the status synchronization information and update the node status of the corresponding standby node based on the status synchronization information.

[0080] Here, the system further includes a proxy node. The proxy node is mainly used to send read and write services on the service side (such as message write requests and message read requests) to the primary node or the standby node that bears the read service on the service side. In practical applications, the proxy node can be a node device or a node cluster composed of multiple node devices. The nodes bear the requests of the read and write services on the service side through the consistent hashing algorithm and also transfer the read and write requests on the service side to the primary node or the standby node responsible for completing the read and write through the consistent hashing algorithm.

[0081] Correspondingly, the above-mentioned management node is further configured to synchronize the node status of each node (including the primary node and the standby node) to the above-mentioned proxy node, which is also convenient for the proxy node to transfer the read and write requests on the service side according to the node status of each node (including the primary node and the standby node). Specifically, the management node can perform status synchronization through the heartbeat method. Here, when the standby node undergoes a state switch, the management node will send status synchronization information to the proxy node, and this status synchronization information is used to indicate the node status after the state switch of the corresponding standby node, and this node status can include offline state, trial state, standby state, online state, etc.

[0082] After the proxy node receives the status synchronization information sent by the management node, it updates the node status of the corresponding standby node according to the status synchronization information. For example, the node status of standby node A stored in the proxy node is the offline state. At this time, it receives the status synchronization information sent by the management node and the status synchronization information indicates that standby node A is currently in the preparation state, then it updates the node status of the stored standby node A from the offline state to the preparation state.

[0083] In some embodiments, the proxy node is further configured to reduce the sending rate of the message write request when the status synchronization information indicates that the standby node is in the preparation state.

[0084] Here, in order to better and faster realize the switching of the node status of the standby node (such as switching from the trial state to the preparation state and then to the online state), so as to shorten the release process of the entire master node expansion, a slow write policy can be added to the proxy node. Specifically, when the status synchronization information received by the proxy node indicates that the standby node is in the preparation state, that is, during the master node expansion process, the proxy node reduces the sending rate of the message write request sent to the master node or the standby node. In actual implementation, when it is in a non-expansion process, the sending mode of the message write request of the proxy node is in the normal sending mode. When the status synchronization information indicates that the standby node is in the preparation state, the sending mode of the message write request is controlled to be switched from the normal sending mode to the target sending mode, and the sending rate of the message write request in the target sending mode is lower than that in the normal sending mode.

[0085] In this way, during the master node expansion process, it can still serve the write requests from the external business side, and by adding the slow write policy, it better and faster realizes the switching of the node status of the standby node (such as switching from the trial state to the preparation state and then to the online state), shortening the release process of the entire master node expansion.

[0086] In practical applications, after the master node expansion is completed, the sending rate of the message write request of the proxy node can still be controlled to be restored to the original sending rate, that is, the sending mode of the message write request is controlled to be switched from the target sending mode to the normal sending mode.

[0087] Step 102: When the master node receives a message write request, it performs message writing.

[0088] Here, during the master node expansion process, in the embodiments of the present application, it can still receive the message write requests from the external business side to continue to carry the write service of the business side, realizing uninterrupted write requests. For the business side and users, it realizes the effect that the write request service is unaware during the master node expansion process.

[0089] At this time, if the master node receives a message writing request from the service side, it performs a message writing operation in response to the message writing request and writes the message corresponding to the message writing request. In practical applications, when the master node writes a message, it first writes the message corresponding to the message writing request into the Kafka message sequence, and after completing the writing of the Kafka message sequence, it writes the message corresponding to the message writing request into the local storage.

[0090] In the embodiment of the present application, since the read and write requests are separated, that is, the master node is used to write messages in response to write requests, and the standby node is used to read messages in response to read requests. Each master node has a corresponding standby node, and this standby node is equivalent to the mirror node of the corresponding master node to achieve data synchronization between the master node and the standby node. Specifically, after the master node writes the message corresponding to the message writing request into the Kafka message sequence, the standby node corresponding to the master node (i.e., the mirror node of the master node) reads the message corresponding to the message writing request written into the Kafka message sequence and writes the read message corresponding to the message writing request into the local storage to achieve data synchronization between the master node and its corresponding standby node. In this way, when a message reading request from the service side is received, the read request service can be implemented through this standby node (i.e., the mirror node of the master node), and the corresponding message to be read is returned, realizing the separation of read and write requests, and avoiding the impact on read requests during the expansion of the master node to the greatest extent, making the read requests uninterrupted. For the service side and users, the effect that the read request service is unaware during the expansion of the master node is achieved. It should be noted that the standby node in a mirror relationship with the master node here is other standby nodes in the system except the above-mentioned target number of standby nodes.

[0091] Step 103: When the standby node is in a standby state, it obtains multiple first messages written by the master node. During the process of obtaining the multiple first messages, when a message writing request is received, it caches the second message corresponding to the message writing request.

[0092] Here, when the standby node is in the standby state, it needs to obtain multiple first messages written by the master node (i.e., the original master node). The multiple first messages include historical messages written before the node expansion instruction is triggered, and messages written in real time after the node expansion instruction is triggered (i.e., during the node expansion process). During the acquisition process, specifically, the messages are acquired in the order of writing, that is, according to the writing time points of the messages, in the order from first to last. In this way, it can be ensured that write requests can be uninterrupted during the master node expansion process. For the business side and users, the effect of being unaware of the write request service during the master node expansion process can be achieved. Here, the standby node in the standby state obtains multiple first messages written by the master node (i.e., the original master node) to catch up with the messages, so that the standby node can synchronize the messages written in the original master node to itself to avoid the loss of information during the master node expansion. In actual implementation, the multiple first messages written by the master node (i.e., the original master node) can be stored in the master node's own storage, or can be stored in other node devices specifically used for data storage.

[0093] During the process of the standby node in the standby state obtaining multiple first messages, it will also receive a message write request. At this time, the system is in the dual-write mode, that is, both the original master node and the standby node in the standby state receive the message write request. Here, the original master node needs to respond to the message write request to write the message (i.e., execute the process of first writing the kafka message sequence and then writing to the local storage), that is, step 102 above; while the standby node in the standby state needs to respond to the message write request and does not execute the message write (i.e., does not need to execute the process of first writing the kafka message sequence and then writing to the local storage), but caches the second message corresponding to the message write request in the local memory (such as a set container in the memory).

[0094] In some embodiments, the proxy node is further configured to send a message write request carrying a target identifier to the standby node when receiving a message write instruction and the status synchronization information indicates that the standby node is in the standby state; the standby node is further configured to cache the second message corresponding to the message write request when receiving the message write request carrying the target identifier. Wherein, the target identifier is used to indicate the corresponding standby node to cache the message corresponding to the message write request carrying the target identifier.

[0095] Here, the corresponding standby node is instructed by the target identifier to cache the message corresponding to the first message write request. The proxy node is further configured to send a message write request carrying the target identifier to the standby node when receiving a message write instruction and the status synchronization information indicates that the standby node is in a preparatory state. When the standby node receives the message write request carrying the target identifier, in response to the message write request, it does not perform message writing (i.e., it does not need to execute the process of writing the Kafka message sequence first and then writing to local storage), but caches the second message corresponding to the message write request in the local memory (such as a set container in the memory).

[0096] Step 104: When there is a first message that is the same as the second message among multiple first messages, the standby node releases the second message, and when the remaining number of messages cached by the standby node is lower than the first quantity threshold, it sends a first notification message to the management node.

[0097] Among them, the first notification message is used to indicate that the standby node in the preparatory state has the ability to switch to the online state.

[0098] Here, during the process of the standby node obtaining multiple first messages, it also determines whether there is a first message that is the same as the second message among the multiple first messages obtained. The purpose of executing this step is that since the first messages are obtained in the order of message writing, during the main node expansion process, both the main node (i.e., the original main node) and the standby node in the preparatory state receive the message write request and perform corresponding operations, and the messages written for the same message write request are the same. At this time, it can be determined whether there is a first message that is the same as the second message among the obtained first messages to determine the progress of the standby node in the preparatory state chasing messages relative to the original main node.

[0099] If it is determined that there is a first message that is the same as the second message among multiple first messages, it means that the standby node in the preparatory state has finished reading the historical messages written by the main node before the node expansion instruction is triggered and has now started to read the messages written after the node expansion instruction is triggered. And based on this, it can be determined which second message the read first message is the same as. Since the standby node in the preparatory state has synchronously obtained the message corresponding to the current message write request by obtaining the first message written by the main node, it means that the message synchronization of the standby node relative to the original main node is completed. At this time, the second message cached by itself is released, specifically, it can be achieved by removing the message sequence number item of the second message stored in the set container.

[0100] Meanwhile, during the process of releasing the second message in the cache, the standby node in the preparatory state also needs to determine the remaining number of the second messages after the cached second message is released. When the remaining number of messages cached by the standby node is lower than the first quantity threshold, it indicates that the standby node is eligible to switch from the preparatory state to the online state. At this time, the standby node sends a first notification message to the management node, and the first notification message is used to indicate that the standby node in the preparatory state can switch to the online state.

[0101] Step 105: The management node receives the first notification message and controls the corresponding standby node to switch from the preparatory state to the online state to write messages as the primary node.

[0102] Here, after receiving the above first notification message sent by the standby node in the preparatory state, the management node controls the corresponding standby node to switch from the preparatory state to the online state based on the first notification message. The standby node in the online state can then write messages as the primary node, including writing the messages corresponding to the message write request into the kafka message sequence, and after completing the writing of the kafka message sequence, writing the messages corresponding to the message write request into the local storage.

[0103] In some embodiments, the proxy node is further configured to, when receiving a message write instruction and the number of standby nodes in the online state reaches a third quantity threshold, send a first message write request without carrying a target identifier to the standby nodes; the standby nodes are further configured to, when receiving the first message write request, determine whether there is a target message corresponding to the first message write request in the standby nodes; when it is determined that there is a target message corresponding to the first message write request, not perform the message write for the corresponding first message write request; when it is determined that there is no target message corresponding to the first message write request, perform the message write for the corresponding first message write request.

[0104] Here, when the number of standby nodes in the online state reaches the third quantity threshold and a message write instruction is received, the proxy node sends a first message write request without carrying a target identifier, so that the standby nodes perform normal message write operations, that is, first write to the kafka message queue and then write to the local storage.

[0105] Since the standby node is still obtaining the first message from the original primary node, the first message write request is sent to both the original primary node and the standby node simultaneously, and the message is written in the order of writing to the original primary node first and then to the standby node. Therefore, the standby node may obtain the message corresponding to the first message write request from the original primary node. So when the standby node is writing the message at this time, it first checks whether there is a message corresponding to the first message write request (such as the same message mainkey) locally. If it exists, it does not write the message corresponding to the first message write request and returns a failure notification to the proxy node; if it does not exist, it writes the message corresponding to the first message write request.

[0106] In some embodiments, the standby node is further configured to stop obtaining multiple first messages written by the primary node when the remaining number of cached messages reaches zero.

[0107] Here, the standby node in the online state still needs to obtain the first messages written by the primary node. At the same time, during the process of obtaining multiple first messages, it also determines whether there is a first message among the obtained multiple first messages that is consistent with the cached second message. If it is determined that there is a first message among the multiple first messages that is consistent with the second message, then at this time, it releases the cached second message of itself. At the same time, during the process of releasing the cached second message, the standby node in the standby state also needs to determine the remaining number of the remaining second messages after the cached second message is released. When the remaining number of cached messages reaches zero, it is considered that the messages written in the standby node have caught up with the messages written in the original primary node, that is, the data synchronization is completed, and the standby node stops obtaining multiple first messages written by the primary node.

[0108] At this time, it indicates that the expansion of the primary node has been completed. When the target number of standby nodes is greater than the original number of primary nodes, since the data and traffic of the original primary node are both synchronized to the standby node in the online state, in order to reduce the waste of node resources at this time, the primary node can be taken offline so that it can be used as a standby node when receiving the standby node expansion instruction. In practical applications, after taking the primary node offline, the primary node is in the offline state; at this time, when it is necessary to expand the machine, that is, when it is necessary to expand the standby nodes in the system, the node state of the offline primary node can be switched to the online state so that the primary node that is re-in the online state can be used as a standby node.

[0109] In some embodiments, the target number is greater than the original number of primary nodes; correspondingly, the management node is further configured to take the primary node offline when the standby nodes with the target number are in the online state so that it can be used as a standby node when receiving the standby node expansion instruction.

[0110] Here, when the target number of standby nodes is greater than the original number of primary nodes, if the management node determines that all standby nodes with the target number are in the online state, since the data and traffic of the original primary nodes are synchronized to the standby nodes in the online state, at this time, in order to reduce the waste of node resources, the primary nodes can be taken offline so that they can be used as standby nodes when receiving the standby node expansion instruction.

[0111] In some embodiments, the number of standby nodes is greater than the target number. When the standby nodes with the target number are used as primary nodes to write messages, the other standby nodes except the standby nodes with the target number are mirror nodes of the primary nodes.

[0112] Correspondingly, the standby nodes are used to, when acting as primary nodes, in response to a message write request, write the message write request into the corresponding third message write message queue and write the third message into local storage; the other standby nodes are used to obtain the third message from the message queue and write the third message into local storage; and, when receiving a message read request, return the third message corresponding to the message read request.

[0113] Here, the number of standby nodes in the system is greater than the target number. When the standby nodes with the target number are used as primary nodes to write messages, the other standby nodes except the standby nodes with the target number are mirror nodes of the primary nodes. For example, the number of standby nodes is 4, including standby nodes A, B, C, and D. Among them, the standby nodes with the target number (2) specifically refer to standby nodes A and B. Then, standby nodes C and D are the other standby nodes except the standby nodes with the target number, and these other standby nodes can be used as mirror nodes of the standby nodes A and B that have become primary nodes.

[0114] After the expansion of the main node is completed, the target number of standby nodes act as the main node for message writing. When a standby node acts as the main node, first, the third message of the corresponding message writing request is written into the kafka message sequence. After the writing of the kafka message sequence is completed, the third message of the corresponding message writing request is written into the local storage. At this time, there are still mirror nodes (i.e., other standby nodes) corresponding to the target number of standby nodes in the system, and these mirror nodes can be used to provide message reading services. Specifically, after the main node (i.e., the target number of standby nodes) writes the third message of the corresponding message writing request into the kafka message sequence, other standby nodes corresponding to this main node read the written third message of the corresponding message from the kafka message sequence and write the read third message of the corresponding message into the local storage to achieve data synchronization between the main node and its corresponding other standby nodes. In this way, when a message reading request from the business side is received, the reading request service can be implemented through this standby node (i.e., the mirror node of the main node), and the corresponding message to be read is returned, realizing the separation of read and write requests, and avoiding the impact on read requests during the expansion of the main node to the greatest extent, making the read requests uninterrupted. For the business side and users, the effect that the read request service is unaware during the expansion of the main node is achieved.

[0115] It should be noted that the embodiments of the present application can be applied to various scenarios such as cloud technology, artificial intelligence, intelligent transportation, and assisted driving.

[0116] Applying the above embodiments of the present application, when the management node receives a node expansion instruction, it controls the target number of standby nodes to be in a preparatory state. The main node writes messages based on the message writing request. The standby nodes in the preparatory state obtain multiple first messages written by the main node, and during the process of obtaining the multiple first messages, cache the second messages corresponding to the received message writing requests. When the standby node determines that there is a first message in the obtained first messages that is the same as the second message, it releases the second message, and when the remaining message quantity cached by the standby node is lower than the first quantity threshold, it sends a first notification message to the management node. Based on the received first notification message, the management node switches the standby node that sent the first notification message from the preparatory state to the online state, so that the standby node acts as the main node for message writing, thereby realizing the expansion of the main node.

[0117] Here, during the entire main node expansion process, the message write requests of the main node are uninterrupted and maintain normal writing. The node expansion process is imperceptible to the business side. The standby node obtains the first message after the main node writes the message corresponding to the message write request, realizing message synchronization of the standby node relative to the main node. When there is a message in the obtained first message that is the same as the second message cached for the message write request, the second message is released. Based on whether the remaining quantity of the second message is lower than the quantity threshold (if it is lower, it means that the message synchronization of the standby node relative to the main node is completed and it can be used as the main node), it is determined whether the standby node can be used as the main node, avoiding the problem of expansion failure caused by data backup and migration failures, and improving the stability and availability of node expansion.

[0118] Next, the node expansion method provided by the embodiments of the present application will be described in detail. The node expansion method provided by the embodiments of the present application is applied to a node expansion system including a management node, a main node, and at least one standby node. Refer to Figure 4 , Figure 4 which is a schematic flowchart of the node expansion method provided by the embodiments of the present application. The node expansion method provided by the embodiments of the present application includes:

[0119] Step 201: A standby node in a preparatory state obtains a plurality of first messages written by the main node.

[0120] Among them, the standby node in the preparatory state is obtained by the management node switching the states of a target number of standby nodes in response to a node expansion instruction.

[0121] Here, in practical applications, the node expansion system can be a storage computing service system for carrying read and write requests of the business side (such as message requests, information storage, etc.), such as writing messages in response to write requests and returning corresponding data information in response to read requests. In actual implementation, the above read and write requests are mainly carried by the node devices in the system, and the node devices mostly exist based on the architecture of the main node + standby node. In the embodiments of the present application, each main node corresponds to a corresponding standby node, and the standby node is equivalent to the mirror node of the corresponding main node to achieve data synchronization between the main node and the standby node. Based on this, the read and write requests in the embodiments of the present application are separated, that is, messages are written through the main node to respond to write requests, and messages are read through the standby node to respond to read requests. In this way, the main node does not need to carry read requests. When there is a need to expand the main node, while expanding the main node, the processing of read requests is not affected, making the read requests uninterrupted. For the business side and users, the effect of the read request service being imperceptible during the main node expansion process is achieved.

[0122] It should be noted that the standby nodes in a mirror relationship with the master node here are other standby nodes in the system except for the above-mentioned target number of standby nodes. The above-mentioned target number of standby nodes refers to specific standby nodes. For example, if the target number of standby nodes is 4, namely standby nodes A, B, C, and D, then the subsequent target number of standby nodes specifically refers to these standby nodes A, B, C, and D.

[0123] Here, the node expansion method provided by the embodiments of the present application is applied to the expansion of the master node in the system. Next, the node expansion system provided by the embodiments of the present application will be described in detail based on the expansion process of the master node.

[0124] In some embodiments, the node expansion system further includes a management node, which is mainly responsible for managing the node status of each node device, the selection of the master node, heartbeat maintenance, etc. In the embodiments of the present application, when the user has a need to expand the master node, such as when the master node service is overloaded and the master node needs to be expanded, the user can trigger a node expansion instruction for the master node. This node expansion instruction acts on the management node to achieve the expansion of the master node through the management node. When the management node receives this node expansion instruction, it searches in the system for standby nodes that can be used for expanding the master node, so as to achieve the expansion of the master node by converting the standby nodes into master nodes. In practical applications, the node expansion instruction may also include the number of standby nodes required for expanding the master node, or the number of nodes that the master node is to reach after expansion, etc., so that the management node can control the corresponding number of standby nodes to implement the master node expansion task corresponding to the node expansion instruction.

[0125] When the management node determines that the target number of standby nodes is required to expand the master node, the management node controls the target number of standby nodes to be in a preparatory state, that is, the preparatory state as the master node. In actual implementation, the management node can send a control instruction for instructing the corresponding standby node to switch its node status to the preparatory state. When the standby node receives this control instruction, it controls its own node status to switch from the current state (such as the offline state) to the preparatory state, so as to wait for subsequent operations to achieve the conversion from the standby node to the master node and achieve the purpose of expanding the master node. For example, if the node expansion instruction indicates that the number of nodes after expanding the master node is 2, and the original number of master nodes is 1, then at this time, if the original master node needs to be taken offline after expansion, the target number of standby nodes required for expansion is 2; if the original master node does not need to be taken offline after expansion, the target number of standby nodes required for expansion is 1.

[0126] Step 202: During the process of obtaining multiple first messages, when a message write request is received, cache the second message corresponding to the message write request.

[0127] Here, during the process of expanding the main node, in the embodiments of the present application, it is still possible to receive message write requests from the external service side to continue to carry the write service of the service side, so as to ensure that the write requests are uninterrupted. For the service side and users, the effect that the write request service is unaware during the process of expanding the main node is achieved.

[0128] At this time, if the main node receives a message write request from the service side, it performs a message write operation in response to the message write request and writes the message corresponding to the message write request. In practical applications, when the main node writes a message, it first writes the message corresponding to the message write request into the kafka message sequence, and after completing the writing of the kafka message sequence, it writes the message corresponding to the message write request into the local storage.

[0129] In the embodiments of the present application, since the read and write requests are separated, that is, the main node is used to write messages in response to write requests, and the standby node is used to read messages in response to read requests. Each main node has a corresponding standby node, and this standby node is equivalent to the mirror node of the corresponding main node to achieve data synchronization between the main node and the standby node. Specifically, after the main node writes the message corresponding to the message write request into the kafka message sequence, the standby node corresponding to the main node (i.e., the mirror node of the main node) reads the message corresponding to the message write request written in the kafka message sequence and writes the read message corresponding to the message write request into the local storage to achieve data synchronization between the main node and its corresponding standby node. In this way, when a message read request from the service side is received, the read request service can be implemented through this standby node (i.e., the mirror node of the main node), and the corresponding message to be read is returned, realizing the separation of read and write requests, and maximizing the avoidance of the impact on read requests during the process of expanding the main node, so that the read requests are uninterrupted. For the service side and users, the effect that the read request service is unaware during the process of expanding the main node is achieved. It should be noted that the standby node in a mirror relationship with the main node here is other standby nodes in the system except the above-mentioned target number of standby nodes.

[0130] Step 203: When there is a first message that is the same as the second message among multiple first messages, release the second message, and when the remaining message quantity cached by the standby node is lower than the first quantity threshold, send a first notification message to the management node, so that the management node, based on the first notification message, switches the standby node that sends the first notification message from the standby state to the online state to write messages as the main node.

[0131] Among them, the first notification message is used to indicate that the standby node in the standby state has the ability to be switched to the online state.

[0132] Here, when the standby node is in the preparatory state, it needs to obtain multiple first messages written by the primary node (i.e., the original primary node). The multiple first messages include historical messages written before the node expansion instruction is triggered, and messages written in real time after the node expansion instruction is triggered (i.e., during the node expansion process). During the acquisition process, specifically, the messages are obtained in the order of writing, that is, according to the writing time points of the messages, in the order from first to last. In this way, it can be ensured that write requests can be uninterrupted during the primary node expansion process. For the business side and users, the effect of the write request service being unaware during the primary node expansion process can be achieved. Here, the standby node in the preparatory state obtains multiple first messages written by the primary node (i.e., the original primary node) to catch up with the messages, so that the standby node can synchronize the messages written in the original primary node to itself to avoid information loss during the primary node expansion. In actual implementation, the multiple first messages written by the primary node (i.e., the original primary node) can be stored in the storage of the primary node itself or in other node devices specifically used for data storage.

[0133] During the process of the standby node in the preparatory state obtaining multiple first messages, it will also receive a message write request. At this time, the system is in the dual-write mode, that is, both the original primary node and the standby node in the preparatory state receive the message write request. Here, the original primary node needs to write messages in response to the message write request (i.e., execute the process of writing the kafka message sequence first and then writing to the local storage), that is, step 102 above; while the standby node in the preparatory state needs to respond to the message write request and does not execute message writing (i.e., does not need to execute the process of writing the kafka message sequence first and then writing to the local storage), but caches the second message corresponding to the message write request in the local memory (such as a set container in the memory).

[0134] During the process of the standby node obtaining multiple first messages, it also determines whether there is a first message in the multiple first messages obtained that is the same as the second message. The purpose of performing this step is that since the first messages are obtained in the order of writing when obtaining the first messages, and during the primary node expansion process, both the primary node (i.e., the original primary node) and the standby node in the preparatory state receive the message write request and perform corresponding operations, and the messages written for the same message write request are the same. At this time, it can be determined whether there is a first message in the obtained first messages that is the same as the second message to determine the progress of the standby node in the preparatory state catching up with the messages relative to the original primary node.

[0135] If it is determined that there is a first message among multiple first messages that is consistent with the second message, it indicates that the standby node in the preparatory state has finished reading the historical messages written by the master node before the node expansion instruction was triggered and has now started reading the messages written after the node expansion instruction was triggered. And based on this, it can be determined which second message the read first message is consistent with. Since the standby node in the preparatory state has synchronized and obtained the message corresponding to the current message write request by obtaining the first message written by the master node, at this time, it releases the second message cached in itself. Specifically, it can be achieved by removing the message sequence number item of the second message stored in the set container.

[0136] Meanwhile, during the process of releasing the cached second message, the standby node in the preparatory state also needs to determine the remaining message quantity of the remaining second messages after the cached second message is released. When the remaining message quantity cached by the standby node is lower than the first quantity threshold, it indicates that the standby node is eligible to switch from the preparatory state to the online state. At this time, the standby node sends a first notification message to the management node, and this first notification message is used to indicate that the standby node in the preparatory state has the ability to be switched to the online state.

[0137] After receiving the above first notification message sent by the standby node in the preparatory state, based on this first notification message, the management node controls the corresponding standby node to switch from the preparatory state to the online state. The standby node in the online state can then act as the master node for message writing, including writing the message corresponding to the message write request into the kafka message sequence, and after completing the writing of the kafka message sequence, writing the message corresponding to the message write request into the local storage.

[0138] In practical applications, when the target quantity of standby nodes is greater than the original quantity of master nodes, if the management node determines that all standby nodes with the target quantity are in the online state, since the data and traffic of the original master node are both synchronized to the standby nodes in the online state, at this time, in order to reduce the waste of node resources, the master node can be taken offline to be used as a standby node when receiving the standby node expansion instruction.

[0139] Applying the above embodiments of the present application, when the management node receives a node expansion instruction, it controls a target number of standby nodes to be in a standby state. The primary node writes messages based on the message write request. The standby nodes in the standby state acquire multiple first messages written by the primary node, and during the process of acquiring the multiple first messages, cache the second messages corresponding to the received message write requests. When a standby node determines that there is a first message in the acquired first messages that is the same as the second message, it releases the second message. When the remaining number of messages cached by the standby node is lower than the first quantity threshold, it sends a first notification message to the management node. The management node, based on the received first notification message, switches the standby node that sent the first notification message from the standby state to the online state, so that the standby node writes messages as the primary node, thereby realizing the expansion of the primary node.

[0140] Here, during the entire primary node expansion process, the message write requests of the primary node are uninterrupted and normal writing is maintained. The node expansion process is imperceptible to the business side. And the standby nodes acquire the first messages after the primary node writes the first messages corresponding to the message write requests, realizing the message synchronization of the standby nodes relative to the primary node. When there is a message in the acquired first messages that is the same as the second message cached for the message write request, the second message is released. Based on whether the remaining quantity of the second message is lower than the quantity threshold (if it is lower, it means that the message synchronization of the standby node relative to the primary node is completed and it can be used as the primary node), it is determined whether the standby node can be used as the primary node, avoiding the problem of expansion failure caused by data backup and migration failures, and improving the stability and availability of node expansion.

[0141] Next, the exemplary application of the embodiments of the present application in an actual application scenario will be described.

[0142] Distributed storage computing services (such as Redis, etc.) are usually deployed in the master + slave mode (i.e., the master node + the standby node, and the slave node is equivalent to the mirror of the master node, and the data remains consistent). When expanding the capacity of the master node (for example, if the original master has N nodes and now it needs to be expanded to 2N nodes), it is necessary to first prepare the machines, then make data backups and migrate the data key by key to the prepared machines, and retain the original master node. However, in this master node expansion method, since both read and write requests are completed at the master node, for read requests, they may not be able to be requested at the original master node and need to be routed to the new master node. For write requests, due to the existence of the old and new masters, the data synchronization of the slave is also relatively complex, and the operation and maintenance difficulty is relatively large. At the same time, in the scenario of large values (when a single value is a set or a map), the requirements for the network (such as network bandwidth, network stability, etc.) are very high, and the expansion time is relatively long, and the overall system jitter will be relatively obvious. Once the data migration fails, it may lead to the failure of the final expansion, and the operation and maintenance complexity will increase greatly.

[0143] Based on this, the embodiment of the present application provides a master node expansion method, which can ensure that the read and write requests of the service to the outside are uninterrupted and the business and users are unaware during the expansion of the master node, so as to ensure the high availability of the business and users.

[0144] Before describing the master node expansion method provided by the embodiment of the present application, first, the overall architecture of the distributed computing storage system provided by the embodiment of the present application will be described. Refer to Figure 5 , Figure 5 is the architecture schematic diagram of the distributed computing storage system provided by the embodiment of the present application, and the function descriptions of each module included therein are as follows:

[0145] 1) Business-side read and write service: The call module on the business side, whose main function is to interact the read and write requests on the business side with the storage computing system.

[0146] 2) Proxyserver: The access layer service (i.e., the above-mentioned proxy node), which consists of multiple nodes. The nodes carry the read and write requests of the business-side read and write service through the consistent hash method to pass the read and write requests on the business side to the leafserver.

[0147] 3) configserver: Cluster management service (i.e., the above-mentioned management node), responsible for managing the status of each leafserver node, maintaining heartbeats, selecting the master node, etc. At the same time, it synchronizes cluster information (such as the status change of slave nodes) to the proxyserver via heartbeats.

[0148] 4) leafserver: Includes master (i.e., the primary node) and slave (i.e., the standby node), which is the real computing and storage core module, responsible for processing read and write requests from the proxyserver. In this application, read and write requests are separated. All write requests are carried by the master node of the leafserver, and read requests are carried by the slave nodes. This can avoid the situation where read requests need to request both the old and new master nodes when the master node is scaled out. In practical applications, when the master node processes a write request, it first writes the message to the kafka message queue, and then writes the message to the local storage of the master node. Only if both are successful, it returns success to the proxyserver. Here, there can be multiple lines for Slave node 1 - N, which can be set according to system requirements.

[0149] 5) kafka: Message queue, which exists as a message transfer medium, used to store the messages written by the master node of the leafserver, and then consumed by the slave nodes of the leafserver, so that the data stored by the slave nodes of the leafserver is consistent with the data stored by the master node. Specifically, after the master node writes the message to kafka, the slave node consumes the message in kafka and stores the message in the local storage of the slave node to achieve data synchronization between the master and slave. In this way, the slave node can carry the processing of read requests.

[0150] 6) Incremental index construction service: It is a processing job of Flink, mainly used to precipitate the messages in the kafka message queue into the hbase persistent storage.

[0151] 7) hbase: A distributed, column-oriented open-source database. Due to the existence of hbase, the above system can process the add, delete, modify, and query messages on the business side and persistently store the results in the hbase table.

[0152] 8) Full - volume index construction task: It is a batch processing task of Spark, which can be triggered regularly (usually daily) to construct an index for all the data in HBase, and then push it to the object storage (Cloud Object Storage, COS) for storage, so that the online leafserver can load and switch. Generally, index switching is only required when the leafserver node fails. Under normal circumstances, frequent switching is not necessary.

[0153] 9) Object storage (Cloud Object Storage, COS): It is a distributed storage service without a directory hierarchy structure, without data format restrictions, capable of accommodating a large amount of data, and supporting HTTP / HTTPS protocol access. Here, it can be used to store the index constructed by the full - volume index construction task.

[0154] Based on the above description of the overall architecture of the distributed computing and storage system provided in the embodiments of the present application, the method for expanding the master node provided in the embodiments of the present application will be described next, that is, how to achieve high - availability expansion of the master node without stopping the read - write service, so that the business and users are unaware. The expansion process involved in the method for expanding the master node provided in the embodiments of the present application is divided into 6 stages, and there are also 6 corresponding states corresponding to the state transitions of the system. They will be described separately below. Among them, o - master is the original master node, and n - master is the new master node after expansion.

[0155] See Figure 6 , Figure 6 It is a schematic diagram of state 1 in the master node expansion process provided in the embodiments of the present application. Here, state 1 is the preparation state before expanding the master node, and all slave nodes need to be expanded first. Specifically, considering ensuring the cleanliness of the system data, it is usually necessary to take the corresponding slave nodes offline to clear the previous data in the nodes. Therefore, when expanding the slave nodes, one row of slave nodes needs to be taken offline first, and then the slave nodes are expanded by a factor of two. For example, if the original master has only 2 nodes and is prepared to be expanded to 4 nodes, as Figure 6 shown, the slave nodes have been expanded to 4 nodes.

[0156] In this way, in state 1 of the master node expansion process, among them,

[0157] (1) There is only one row of master (i.e., 2 nodes) in the configserver cluster management information;

[0158] (2) The proxy server cluster routes write requests to the corresponding master. Since there is no forced one-to-one relationship between the proxy server and the leaf server, each proxy server will send write requests to any node of the master. Specifically, the write requests are sent to the corresponding master through the consistent hashing algorithm. At this time, the write requests are not speed-limited, and the sent messages do not carry any flags. This flag refers to the mark / identification of the message and can be represented by a field;

[0159] (3) According to the data volume of the business, the Kafka message queue needs to pre-allocate enough partitions to ensure that there will be no discard-type consumption after the subsequent expansion of the leaf server (that is, to ensure that the data under a certain partition can be pulled, and then the messages belonging to itself are selected for consumption). In the embodiment of the present application, the Kafka partition can be 128. Subsequently, it supports the leaf server to expand from 4 columns -> 8 columns -> 16 columns ->... -> 128 columns. If it exceeds 128 columns, it will result in discard-type consumption.

[0160] See Figure 7 , Figure 7 is a schematic diagram of state 2 in the master node expansion process provided by the embodiment of the present application. Here, in state 2 of the master node expansion process, the configserver can be notified to start the master expansion process through an external command. The configserver selects a row (i.e., the target number of master nodes after expansion) of slave nodes, and changes the state of the selected slave nodes to the master-try state (i.e., the above-mentioned trial state), and at the same time synchronizes the heartbeat to all proxy servers. At this time, there is a row of formal master nodes "o-master (including o-master0 and o-master1)" in the cluster, and a row of master nodes "n-master (including n-master0 to n-master3)" in the master-try state

[0161] See Figure 8 , Figure 8 is a schematic diagram of state 3 in the master node expansion process provided by the embodiment of the present application. Here, in state 3 of the master node expansion process, among them,

[0162] (1) After the state switch in the above state 2, the proxy server receives the heartbeats of two rows of masters, switches the message sending mode to the slow mode, that is, reduces the update speed of the read and write request messages on the business side, and sends messages with flag1 attached to the o-master;

[0163] (2) All write messages stored by n-master(try) before consuming the o-master node from the full index to implement message chasing, ensuring data synchronization from o-master to n-master. When the message chasing lag (i.e., the number of remaining unconsumed write messages) is less than a certain value (such as 10), n-master(try) feeds back to the configserver through the heartbeat, indicating that it has the ability to transition from the try state to the pre state (i.e., the above-mentioned preparatory state).

[0164] (3) The configserver synchronizes the information (i.e., status information) of the two masters (o-master and n-master-pre) to all proxyservers through the heartbeat.

[0165] See Figure 9 , Figure 9 which is a schematic diagram of state 4 in the master node expansion process provided by the embodiments of the present application. Here, in state 4 of the master node expansion process,

[0166] (1) After receiving the information of the two masters (o-master and n-master-pre) sent by the configserver, the proxyserver starts the process of double-writing the two masters (o-master and n-master-pre). Specifically, it can first write to n-master and then write to o-master, where

[0167] the o-master sends the received msg with flag2 to kafka and writes to local storage;

[0168] After receiving the message with flag2 (i.e., msg with flag2) sent by the proxyserver, n-master does not perform local updates but stores it in the set in memory. At the same time, it consumes the messages carrying flag1 and flag2, that is, continues to chase messages. When consuming the message carrying flag2, it needs to remove the corresponding item from the set (i.e., the sequence number of the message carrying flag2 in the set);

[0169] (2) When the number of remaining messages carrying flag2 in the set of n-master is less than a certain value (such as 10), n-master obtains the appointment heartbeat from the configserver to complete the appointment, that is, switches from the pre state to the online state. At this time, the o-master state continues to maintain the online state.

[0170] (3) Since the number of o-masters and n-masters is inconsistent, when an n-master is appointed at this time, the o-master cannot be taken offline directly because the n-master may only be able to carry part of the traffic of the o-master.

[0171] See Figure 10 , Figure 10 which is a schematic diagram of state 5 in the master node expansion process provided by the embodiments of the present application. Here, in state 5 of the master node expansion process,

[0172] (1) As the number of proxyservers with n-masters upgraded to the online state increases, the proxyserver will gradually send only the traffic (msg with no flag) on the o-master to the n-master. The n-master that receives the msg with no flag needs to first look up the messages in the local set to see if there is a message with the same mainkey. If there is, it returns a failure to the proxyserver to ensure message order. If not, it first writes the received message to the kafka message queue, then writes it to the local storage, and then returns a success to the proxyserver;

[0173] (2) When all n-masters are switched to the online state and passed to the proxyserver node through the configserver heartbeat, each proxyserver node can send write requests to the n-master. At this time, the o-master traffic is cleared. For read messages, those within the last 2 minutes are treated the same as write messages, and read requests more than 2 minutes old are routed to the remaining slave nodes;

[0174] (3) If the configserver self-checks and finds that all n-masters in the cluster are set to the online state and this state persists for 3 heartbeat cycles with the proxyserver, then all o-masters are set to offline.

[0175] (4) When it is obtained through calling the interface of the configserver that all o-masters in the entire cluster are in the offline state, all o-masters are taken offline. When machines need to be expanded, they can be brought back online as slaves.

[0176] See Figure 11 , Figure 11It is a schematic diagram of state 6 in the master node expansion process provided by an embodiment of the present application. Here, state 6 is the state after the master node expansion is completed. In state 6, n-master stops pulling kafka messages as the messages in its internal set are cleared. If a message other than flag2 is pulled, it is directly discarded. At this point, the new master cluster has been expanded. At this time, o-master can go offline and, when there is a need to expand the master node, it can become a new master node as a backup node based on the above expansion method. The single-write master of the proxyserver can then be restored to non-slow write mode.

[0177] In practice, the slow write policy can be omitted. Under normal circumstances, the write traffic supported by the cluster is far less than the system's maximum traffic rate. Therefore, the slow write policy is added to facilitate faster transitions between the n-master's try, pre, and online states, shortening the release switching process. Furthermore, the n-master's try state can be omitted, allowing for the pre, online, and two-state transitions.

[0178] The application of the above-mentioned embodiments of the present application can effectively solve the problem of transparency of external read and write services when the master node is expanded, with zero perception to the business party, and no data migration and read requests to the new and old masters. It greatly improves the availability of storage and computing services and reduces the operability and maintainability of expansion.

[0179] The following continues to describe the exemplary structure of the node expansion device 655 provided in the embodiment of the present application implemented as a software module. In some embodiments, such as Figure 2 As shown, the software modules stored in the node expansion device 655 of the memory 650 may include:

[0180] An acquisition module 6551 is configured to cause a standby node in a standby state to acquire a plurality of first messages written by the master node. The standby nodes in the standby state are obtained by switching the state of a target number of standby nodes in response to a node expansion instruction by the management node.

[0181] The cache module 6552 is configured to cache a second message corresponding to a message write request when receiving a message write request during the process of acquiring the plurality of first messages;

[0182] A sending module 6553, configured to release the second message when there is a first message that is the same as the second message among the multiple first messages, and send a first notification message to the management node when the remaining number of messages cached by the standby node is lower than a first quantity threshold, so that the management node, based on the first notification message, switches the standby node that sends the first notification message from the standby state to the online state to write messages as the primary node;

[0183] Wherein, the first notification message is used to indicate that the standby node in the standby state can be switched to the online state.

[0184] Applying the foregoing embodiments of the present application, when the management node receives a node expansion instruction, it controls a target number of standby nodes to be in the standby state, the primary node writes messages based on a message write request, and the standby nodes in the standby state obtain multiple first messages written by the primary node, and during the process of obtaining the multiple first messages, cache the second message corresponding to the received message write request; when the standby node determines that there is a first message that is the same as the second message among the obtained first messages, it releases the second message, and when the remaining number of messages cached by the standby node is lower than the first quantity threshold, it sends a first notification message to the management node; the management node, based on the received first notification message, switches the standby node that sends the first notification message from the standby state to the online state, so that the standby node writes messages as the primary node, thereby realizing the expansion of the primary node.

[0185] Here, during the entire primary node expansion process, the message write requests of the primary node are uninterrupted and normal writing is maintained, and the node expansion process is imperceptible to the service side; and the standby node obtains the first message after the primary node writes the first message corresponding to the message write request, realizes the message synchronization of the standby node relative to the primary node, and when there is a message that is the same as the second message cached for the message write request among the obtained first messages, it releases the second message, and determines whether the standby node can be used as the primary node based on whether the remaining quantity of the second message is lower than the quantity threshold (if it is lower, it means that the message synchronization of the standby node relative to the primary node is completed and it can be used as the primary node), avoiding the problem of expansion failure caused by data backup and migration failure, and improving the stability and availability of node expansion.

[0186] An embodiment of the present application further provides an electronic device, where the electronic device includes:

[0187] A memory, configured to store executable instructions;

[0188] A processor, configured to implement the node expansion method provided by the embodiments of the present application when executing the executable instructions stored in the memory.

[0189] The embodiments of the present application also provide a computer program product or a computer program. The computer program product or the computer program includes computer instructions, and the computer instructions are stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the node expansion method provided by the embodiments of the present application.

[0190] The embodiments of the present application also provide a computer-readable storage medium storing executable instructions, which, when executed by a processor, implement the node expansion method provided by the embodiments of the present application.

[0191] In some embodiments, the computer-readable storage medium may be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, flash memory, magnetic surface memory, optical disc, or CD-ROM; or may be various devices including one or any combination of the above memories.

[0192] In some embodiments, the executable instructions may be in the form of a program, software, software module, script, or code, and may be written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including being deployed as an independent program or being deployed as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0193] As an example, the executable instructions may or may not correspond to a file in the file system, and may be stored as a part of a file that stores other programs or data. For example, they may be stored in one or more scripts in a Hyper Text Markup Language (HTML) document, stored in a single file dedicated to the program being discussed, or stored in multiple cooperating files (such as files that store one or more modules, subroutines, or code portions).

[0194] As an example, the executable instructions may be deployed to execute on one computing device, or on multiple computing devices located at one location, or on multiple computing devices distributed at multiple locations and interconnected by a communication network.

[0195] As described above, the above are only embodiments of the present application and are not intended to limit the protection scope of the present application. Any modifications, equivalent replacements, and improvements made within the spirit and scope of the present application are all included in the protection scope of the present application.

Claims

1. A node expansion system, characterized in that, The system includes: a management node, a primary node, and at least one standby node, where the management node is configured to, in response to a node expansion instruction, switch the node states of a target number of standby nodes to a preparatory state; the primary node is configured to perform message writing when receiving a message writing request; the standby node is configured to, when in the preparatory state, obtain a plurality of first messages written by the primary node, and during the process of obtaining the plurality of first messages, when receiving a message writing request, cache a second message corresponding to the message writing request; when there is a first message that is the same as the second message among the plurality of first messages, release the second message, and when the remaining message quantity cached by the standby node is lower than a first quantity threshold, send a first notification message to the management node; the management node is further configured to, when receiving the first notification message, switch the standby node that sends the first notification message from the preparatory state to an online state to perform message writing as the primary node.

2. The system according to claim 1, wherein the management node is further configured to, in response to the node expansion instruction, switch the node states of a target number of standby nodes to a trial state; the standby node is further configured to, when in the trial state, obtain a plurality of first messages written by the primary node, and during the process of obtaining the plurality of first messages, when determining that the quantity of first messages not obtained is lower than a second quantity threshold, send a second notification message to the management node; the management node is further configured to receive the second notification message and switch the standby node that sends the second notification message from the trial state to the preparatory state.

3. The system according to claim 1, wherein The system further includes: a proxy node; the management node is further configured to, when the standby node undergoes a state switch, send state synchronization information to the proxy node; the proxy node is configured to receive the state synchronization information and update the node state of the corresponding standby node based on the state synchronization information.

4. The system according to claim 3, wherein the proxy node is further configured to, when the state synchronization information indicates that the standby node is in the preparatory state, reduce the sending rate of the message writing request.

5. The system according to claim 3, wherein the proxy node is further configured to, when receiving a message writing instruction and the state synchronization information indicates that the standby node is in the preparatory state, send a message writing request carrying a target identifier to the standby node; wherein the target identifier is used to indicate that the corresponding standby node caches a message corresponding to the message writing request carrying the target identifier; the standby node is further configured to, when receiving the message writing request carrying the target identifier, cache a second message corresponding to the message writing request.

6. The system according to claim 5, wherein the proxy node is further configured to, when receiving a message writing instruction and the number of standby nodes in the online state reaches a third quantity threshold, send a first message writing request not carrying the target identifier to the standby node; The standby node is further configured to determine whether there is a target message corresponding to the first message write request when receiving the first message write request; When it is determined that there is a target message corresponding to the first message write request, cancel the message write corresponding to the first message write request; When it is determined that there is no target message corresponding to the first message write request, perform the message write corresponding to the first message write request.

7. The system according to claim 1, wherein The standby node is further configured to stop obtaining a plurality of first messages written by the master node when the remaining number of cached messages reaches zero.

8. The system according to claim 1, wherein The target quantity is greater than the original quantity of the master node; The management node is further configured to switch the node state of the master node to the offline state when the target quantity of standby nodes are in the online state, so as to be used as a standby node to go online when receiving a standby node expansion instruction.

9. The system according to claim 1, wherein The number of standby nodes is greater than the target quantity. When the target quantity of standby nodes perform message writing as the master node, the other standby nodes except the target quantity of standby nodes are mirror nodes of the master node; The standby node is configured to, when acting as the master node, in response to a message write request, write a third message corresponding to the message write request into a message queue, and write the third message into local storage; The other standby nodes are configured to obtain the third message from the message queue and write the third message into local storage; And, when receiving a message read request, return the third message corresponding to the message read request.

10. A node expansion method, characterized in that, Applied to a node expansion system including a management node, a master node, and at least one standby node; the method includes: A standby node in a preparatory state obtains a plurality of first messages written by the master node. The standby node in the preparatory state is obtained by the management node performing a state switch on a target quantity of standby nodes in response to a node expansion instruction; During the process of obtaining the plurality of first messages, when receiving a message write request, cache a second message corresponding to the message write request; When there is a first message identical to the second message among the plurality of first messages, release the second message, and when the remaining number of cached messages in the standby node is lower than a first quantity threshold, send a first notification message to the management node, so that The management node, based on the first notification message, switches the standby node that sends the first notification message from the preparatory state to the online state to perform message writing as the master node.

Images