Distributed storage system, view management method and computing device cluster

Abstract

The invention provides a distributed storage system, a view management method and a computing device cluster, the distributed storage system comprises N storage nodes and computing nodes, N is a positive integer, and the N storage nodes comprise a first view generation node; the first view generation node is used for determining M view pushing nodes from the N storage nodes, and M is a positive integer smaller than or equal to N; the first view generation node is used for sending view information to the M view pushing nodes, and the view information is used for indicating a rule for distributing the data to the N storage nodes; the computing node is used for determining a first view pushing node from the M view pushing nodes, requesting view information from the first view pushing node, and performing data writing or data reading on the N storage nodes based on the requested view information; the view production node is used for view generation and view issuing, and the computing node requests the view from the view pushing node, so that the load of the view production node and the possibility of becoming a performance bottleneck are reduced.

Description

[0001] This application claims priority to Chinese Patent Application No. 202410725311.8, filed on June 5, 2024, entitled "A View Management Method and Distributed Storage Cluster", the entire contents of which are incorporated herein by reference. Technical Field

[0002] This invention relates to the field of storage technology, and in particular to a distributed storage system, a view management method, and a computing device cluster. Background Technology

[0003] In a distributed storage system, the system maintains a globally consistent view of data, recording the rules for distributing data across storage nodes. Storage and compute nodes in the distributed storage system query this view to determine where data is stored and retrieved. The distributed storage system performs view management: automatically updating the view information when a storage node joins or leaves, or when a storage node experiences a temporary failure, ensuring the accuracy and consistency of the view through passive querying or active updating. View management is mainly divided into centralized and decentralized management. Centralized management refers to a process with a dedicated management role that handles view updates and distribution. Decentralized management involves each storage node managing a portion of the view and synchronizing the view information on its node to other storage nodes through propagation protocols.

[0004] For centralized approaches, as the number of storage nodes in a distributed storage system increases, the load on the management nodes also increases, which may cause view management to become a performance bottleneck in the distributed storage system. Summary of the Invention

[0005] This invention provides a distributed storage system, a view management method, and a computing device cluster. By dividing storage nodes into view production nodes and view push nodes, the view production node determines the view push node and pushes the view of the distributed storage system to the view push node. When a computing node needs a view, it requests view information from any view push node. On the one hand, the view production node is used for view generation and distribution, and the computing node requests the view from the view push node, which reduces the load on the view production node in view management, making the storage nodes more scalable and allowing for flexible expansion of the storage node scale, reducing the possibility that view management will become a performance bottleneck. On the other hand, there is no need for view synchronization between multiple storage nodes, reducing the overhead of view synchronization. Furthermore, in scenarios where storage nodes in the distributed storage system change frequently, the view push path and time are predictable when the view production node pushes the view to the view push node, thereby improving the efficiency of view distribution.

[0006] In a first aspect, the present invention provides a distributed storage system, which includes: N storage nodes and computing nodes, where N is a positive integer, and the N storage nodes include a first view generation node;

[0007] The first view generation node is used to determine M view push nodes from N storage nodes, where M is a positive integer less than or equal to N;

[0008] The first view generation node is used to send view information to M view push nodes. The view information is used to indicate the rules for distributing data to N storage nodes.

[0009] The compute node is used to determine the first view push node from M view push nodes, request view information from the first view push node, and perform data writing or data reading on N storage nodes based on the requested view information.

[0010] In the solution provided by this invention, on the one hand, the view production node is used for view generation and view distribution, and the compute node requests the view from the view push node, which reduces the load on the view production node in view management, making the storage node more scalable and able to flexibly expand the scale of the storage node, reducing the possibility that view management will become a performance bottleneck; on the other hand, there is no need for view synchronization between multiple storage nodes, which reduces the overhead of view synchronization; furthermore, in the scenario where storage nodes in the distributed storage system change frequently, the view is pushed to the view push node by the view production node, and the view push path and time are predictable, thereby improving the efficiency of view distribution.

[0011] In some possible implementations, the first view generating node is used to determine the association between the second view pushing node and the H view consuming nodes. The H view consuming nodes are storage nodes other than the first view generating node and the M view pushing nodes among the N storage nodes, where H is a positive integer less than N; the second view pushing node is any one of the M view pushing nodes.

[0012] The first view generates nodes, which are used to send association relationships to nodes pushed to the second view.

[0013] The second view push node is used to determine the view consumer node associated with the second view push node based on the association relationship, and to send view information to the associated view consumer node.

[0014] In the solution provided by this invention, the view push node is associated with the view consumer node. The view push node needs to synchronize the view to the associated view consumer node, without the view producer needing to synchronize the view to the view consumer. This reduces the load on the view production node in view management and makes the storage node more scalable. The scale of the storage node can be flexibly expanded without affecting view management.

[0015] In some possible implementations, the distributed storage system also includes service nodes, which communicate with N storage nodes;

[0016] The first view generation node is used to determine the view push node information, and the view push node information is used to indicate M view push nodes.

[0017] The first view generation node is used to send view push node information to the service node;

[0018] Compute nodes are used to request view push node information from service nodes;

[0019] A compute node is used to determine the first view push node from M view push nodes based on view push node information.

[0020] In the solution provided by this invention, the computing node can select a view push node from a large number of view push nodes to request a view, without having to request it from the view generation node. This reduces the load on the view production node in view management, makes the storage node more scalable, and allows for flexible expansion of the storage node scale, reducing the possibility that view management will become a performance bottleneck.

[0021] In one possible example of this implementation, a service node is used to send the node status of N storage nodes to the first view generation node;

[0022] The first view generation node is used to determine view information based on the node status and data distribution rules of N storage nodes.

[0023] In the solution provided by this invention, the node status monitoring of storage nodes is undertaken by the service node, eliminating the need for the view generation node to monitor the storage node status. This reduces the load on the view production node in view management, giving the storage node strong scalability and allowing for flexible expansion of the storage node scale, thus reducing the possibility of view management becoming a performance bottleneck. Furthermore, in scenarios where storage nodes frequently change in a distributed storage system, the view production node can improve the speed of view updates by monitoring the node status through the service node.

[0024] In another possible example of this implementation, the service node is used to send the updated node status of the target storage node to the first view generating node after determining that the node status of the target storage node among N storage nodes has been updated.

[0025] In the solution provided by this invention, the service node monitors the node status of the storage node, and after the status is updated, sends the updated node status to the view production node, which then updates the view.

[0026] In another possible example of this implementation, a first view generating node is used to send view information to a service node; the service node is used to store the view information.

[0027] In the solution provided by this invention, the view production node sends the view information to the service node for persistence, thereby reducing the impact on view management in the event of a failure of any management node in the distributed storage system.

[0028] In some possible implementations, the first view generation node is used to determine that the third view push node among the M view push nodes is faulty. From the storage nodes other than the first view generation node and the M view push nodes among the N storage nodes, the fourth view push node is selected to replace the third view push node, and the updated view push node information is obtained.

[0029] The first view generation node is used to send updated view push node information to the service node;

[0030] The first view generates a node, which is used to send view information to the fourth view push node.

[0031] In the solution provided by this invention, the view production node replaces the faulty view push node after the view push node fails, thereby achieving fault handling.

[0032] In some possible implementations, the distributed storage system also includes a second view producer node;

[0033] In the event of a failure in the first view generation node, a second view production node is used to read view information and view push node information from the service node.

[0034] In the solution provided by this invention, after a view production node fails, other view production nodes take over view management, read view information and view push node information from the service node, and improve the efficiency of view production node failure handling.

[0035] In some possible implementations, a compute node is used to determine if requesting view information from the first view push node fails, and based on the view push node information, a fifth view push node is determined, and view information is requested from the fifth view push node.

[0036] In this implementation example, the compute node is used to determine if the request for view information has failed, request the latest view push node information from the service node, determine the sixth view push node based on the latest view push node information, and request the latest view information from the sixth view push node.

[0037] In the solution provided by this invention, when a computing node determines that it cannot request view information, it requests the latest view push node information from the service node and re-determines the view push node to request the view, thereby quickly updating the view when the view expires.

[0038] For example, a compute node is configured to determine that the request for view information has failed if the number of failed requests for view information is greater than or equal to a preset number, or if no view information is requested within a preset time period.

[0039] In some possible implementations, the first view generation node is the management node among N storage nodes.

[0040] In some possible implementations, the M view push nodes are storage nodes other than the management nodes among the N storage nodes.

[0041] Secondly, the present invention provides a view management method, which is implemented by a distributed storage system. The steps performed by the method and the beneficial effects achieved are described in the first aspect of the embodiments of the present invention, and will not be repeated here.

[0042] Specifically, the method is applied to a distributed storage system, which includes N storage nodes and computing nodes, where N is a positive integer, and the N storage nodes include a first view generation node; the method includes:

[0043] The first view generation node determines M view push nodes from N storage nodes, where M is a positive integer less than or equal to N;

[0044] The first view generation node sends view information to M view push nodes. The view information is used to indicate the rules for distributing data to N storage nodes.

[0045] The compute node determines the first view push node from M view push nodes, requests view information from the first view push node, and performs data writing or data reading on N storage nodes based on the requested view information.

[0046] Among some possible implementations, the method also includes:

[0047] The first view generation node determines the association between the second view push node and the H view consumption nodes. The H view consumption nodes are the storage nodes other than the first view generation node and the M view push nodes among the N storage nodes, where H is a positive integer less than N. The second view push node is any one of the M view push nodes.

[0048] The first view generation node sends the association relationship to the second view push node;

[0049] The second view push node determines the view consumer node associated with it based on the association relationship, and sends view information to the associated view consumer node.

[0050] In some possible implementations, the distributed storage system also includes service nodes, which communicate with N storage nodes;

[0051] The first view generation node determines the view push node information, which is used to indicate M view push nodes.

[0052] The first view generation node sends view push node information to the service node;

[0053] The compute node determines the first view push node from M view push nodes, including:

[0054] The compute node requests view push node information from the service node;

[0055] The compute node determines the first view push node from M view push nodes based on the view push node information.

[0056] Among some possible implementations, the method also includes:

[0057] After determining that the node status of the target storage node among N storage nodes has been updated, the service node sends the updated node status of the target storage node to the first view generation node.

[0058] Among some possible implementations, the method also includes:

[0059] The first view generation node sends view information to the service node;

[0060] Service nodes are used to store view information.

[0061] Among some possible implementations, the method also includes:

[0062] The first view generation node determines that the third view push node among the M view push nodes is faulty. From the N storage nodes other than the first view generation node and the M view push nodes, the fourth view push node is selected to replace the third view push node, and the updated view push node information is obtained.

[0063] The first view generation node sends the updated view push node information to the service node;

[0064] The first view generation node sends view information to the fourth view push node.

[0065] In some possible implementations, the distributed storage system also includes a second view producer node, and the method also includes:

[0066] In the event of a failure in the first view generation node, the second view production node reads view information and view push node information from the service node.

[0067] Among some possible implementations, the method also includes:

[0068] The compute node determines that it failed to request view information from the first view push node. Based on the view push node information, it determines the fifth view push node and requests view information from the fifth view push node.

[0069] Among some possible implementations, the method also includes:

[0070] If the compute node fails to determine the requested view information, it requests the latest view push node information from the service node. Based on the latest view push node information, it determines the sixth view push node and requests the latest view information from the sixth view push node.

[0071] Among some possible implementations, the method also includes:

[0072] If a compute node fails to request view information a number of times, or fails to request view information within a preset time period, it determines that the request for view information has failed.

[0073] In some possible implementations, the first view generation node is the management node among N storage nodes.

[0074] In some possible implementations, the M view push nodes are storage nodes other than the management nodes among the N storage nodes.

[0075] Thirdly, the present invention provides a computing device including a memory and a processor, the memory storing program instructions, and the processor executing the program instructions to perform the methods provided in the second aspect of the present invention and any possible implementation thereof.

[0076] Fourthly, the present invention provides a computing device cluster including multiple computing devices, each computing device including multiple processors and multiple memories, the multiple memories storing program instructions, and the multiple processors executing the program instructions, causing the computing device cluster to perform the methods provided in the second aspect of the present invention and any possible implementation thereof.

[0077] Fifthly, embodiments of the present invention provide a computer storage medium storing instructions that, when executed on a computer, cause the computer to perform the methods provided in the second aspect and any possible implementation thereof.

[0078] In a sixth aspect, embodiments of the present invention provide a computer program product comprising instructions that, when executed on a computer, cause the computer to perform the methods provided in the second aspect and any possible implementation thereof. Attached Figure Description

[0079] Figure 1a This is a schematic diagram of the architecture of the distributed storage system provided in an embodiment of the present invention;

[0080] Figure 1b This is a schematic diagram of the architecture of the distributed storage system provided in this embodiment of the invention;

[0081] Figure 2a This is a schematic diagram of the structure of the distributed storage system provided in an embodiment of the present invention;

[0082] Figure 2b This is a second schematic diagram of the structure of the distributed storage system provided in an embodiment of the present invention;

[0083] Figure 3 This is a schematic diagram of a view management scheme for a distributed storage system provided in an embodiment of the present invention;

[0084] Figure 4 This is a flowchart illustrating the view management method provided in an embodiment of the present invention;

[0085] Figure 5 This is a schematic diagram 2 of the view management method provided in an embodiment of the present invention;

[0086] Figure 6 This is a second schematic diagram of the view management scheme of the distributed storage system provided in this embodiment of the invention;

[0087] Figure 7 This is a schematic diagram of the view management method provided in an embodiment of the present invention. Figure 3 ;

[0088] Figure 8 This is a schematic diagram of the view management method provided in an embodiment of the present invention. Figure 4 ;

[0089] Figure 9 This is a schematic diagram of the view management method provided in an embodiment of the present invention. Figure 5 ;

[0090] Figure 10 This is a schematic diagram of the view management method provided in an embodiment of the present invention. Figure 6 ;

[0091] Figure 11 This is a schematic diagram of the view management method provided in an embodiment of the present invention. Figure 7 ;

[0092] Figure 12 This is a schematic diagram of the view management method provided in an embodiment of the present invention. Figure 8 ;

[0093] Figure 13 This is a schematic diagram of the view management method provided in an embodiment of the present invention. Figure 9 ;

[0094] Figure 14 This is a schematic diagram of a view management scenario provided in an embodiment of the present invention;

[0095] Figure 15 yes Figure 14 A schematic diagram of the view distribution scenario provided for view management;

[0096] Figure 16 yes Figure 14 A schematic diagram of the fault handling scenario provided by the view management;

[0097] Figure 17 This is a schematic diagram of the structure of the distributed storage system provided in an embodiment of the present invention;

[0098] Figure 18 This is a schematic diagram of the structure of a computing device provided in an embodiment of the present invention;

[0099] Figure 19 This is a schematic diagram of the structure of a computing device cluster provided in an embodiment of the present invention;

[0100] Figure 20 This is a schematic diagram illustrating the connection of computing devices in a computer cluster via a network, as provided in an embodiment of the present invention. Detailed Implementation

[0101] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0102] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of the invention. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0103] References to "one embodiment" or "some embodiments" as used in this specification mean that a particular feature, structure, or characteristic described in connection with that embodiment is included in one or more embodiments of the invention. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including, but not limited to," unless otherwise specifically emphasized.

[0104] Figure 1a A schematic diagram illustrating the architecture of a distributed storage system applied in an embodiment of the present invention is shown. Figure 1a As shown, the distributed storage system 100 includes a compute node cluster, a storage node cluster, and a service node 103. The compute node cluster includes one or more compute nodes 101, and these compute nodes 101 can communicate with each other. A compute node 101 can be a virtual machine or a computing device, such as a server, desktop computer, or storage array controller. Any compute node 101 can access any storage node 102 in the storage node cluster via a network. The storage node cluster includes multiple storage nodes 102. The service node 103 communicates with any storage node 102 in the storage node cluster to determine the node status of each storage node 102 in the storage node cluster. The node status can be normal operation or abnormal. It should be noted that this embodiment of the invention does not intend to limit the number of compute nodes 101, storage nodes 102, and service nodes 103; the specific number can be determined according to actual needs.

[0105] In this embodiment of the invention, storage node 102 stores actual data and provides data I / O services to the outside world. In practical applications, users access the distributed storage system through computing node 101. In some possible scenarios, users access computing node 101 through terminal devices; in other possible scenarios, the distributed storage system 100 is deployed in the cloud, and users access computing node 101 through a cloud management platform via terminal devices. The terminal devices can be, but are not limited to, various personal computers, laptops, smartphones, tablets, and portable wearable devices. Exemplary embodiments of the terminal devices involved in this embodiment include, but are not limited to, electronic devices running iOS, Android, Windows, Harmony OS, or other operating systems. This embodiment of the invention does not specifically limit the type of electronic device. The cloud management platform can be a separate electronic device or a device integrated into an electronic device; this embodiment of the invention does not limit the specific deployment method and location of the cloud management platform.

[0106] In a feasible scenario, compute node 101 and storage node 102 are deployed as a single unit. In this scenario, the distributed storage system 100 includes an application server cluster. For example... Figure 2a As shown, the application server cluster includes one or more application servers 110 ( Figure 2a The diagram illustrates two application servers 110 (but is not limited to two application servers 110), which can communicate with each other. An application server 110 is a device with both computing and storage capabilities, such as a server or desktop computer. On the software side, each application server 110 has an operating system. Virtual machines 115 can be created on application servers 110. The computing resources required by virtual machines 115 come from the local processor 111 and memory 112 of application servers 110, while the storage resources required by virtual machines 115 can come from the local hard disk 114 of application servers 110 or from hard disks 114 of other application servers 110. Furthermore, various applications can run in virtual machines 115, and users can trigger read / write data requests through these applications. In this embodiment of the invention, virtual machine 115 can serve as a computing node 101, and hard disk 103 can serve as a storage node 102.

[0107] In terms of hardware, such as Figure 2aAs shown, the application server 110 includes at least a processor 111, memory 112, network interface card (NIC) 113, and hard disk 114. The processor 111, memory 112, NIC 113, and hard disk 114 are connected via a bus. The processor 111 and memory 112 provide computing resources. The hard disk 114 provides storage resources, such as storing data. It can be a disk or other types of storage media, such as a solid-state drive (SSD) or a shingled magnetic recording hard disk. It should be noted that the hard disk 114 is merely an example of non-volatile memory and does not constitute a specific limitation. In practical applications, other non-volatile memories can also be used. This embodiment of the invention uses the hard disk 114 as an example for explanation. Specifically, the processor 111 is a central processing unit (CPU) used to process data access requests from outside the application server 110 or requests generated internally by the application server 110. For example, when the processor 111 receives a write data request from a user, it temporarily stores the data in the write data request in memory 112. When the total amount of data in memory 112 reaches a certain threshold, processor 111 sends the data stored in memory 112 to hard disk 114 for persistent storage. In addition, processor 111 is also used for data computation or processing, such as metadata management, deduplication, data compression, data verification, virtualization of storage space, and address translation. Figure 2a Only one processor 111 is shown in this embodiment. In practical applications, there are often multiple processors 111, and each processor 111 has one or more CPU cores. This embodiment does not limit the number of CPUs or the number of CPU cores.

[0108] Memory 112 refers to internal memory that directly exchanges data with the processor. It can read and write data at any time at high speed, serving as temporary data storage for the operating system or other running programs. Memory 112 includes at least two types of memory, such as random access memory (RAM) or read-only memory (ROM). For example, RAM can be Dynamic Random Access Memory (DRAM) or Storage Class Memory (SCM). DRAM is a semiconductor memory and, like most RAM, is a volatile memory device. SCM is a hybrid storage technology that combines the characteristics of traditional storage devices and RAM. Storage Class Memory offers faster read and write speeds than hard drives but is slower in operation than DRAM and is cheaper. However, DRAM and SCM are only illustrative examples in this embodiment; Memory 112 may also include other random access memories, such as Static Random Access Memory (SRAM). For read-only memory, examples include programmable read-only memory (PROM) and erasable programmable read-only memory (EPROM). Additionally, memory 113 can also be a dual in-line memory module (DIMM), i.e., a module composed of dynamic random access memory (DRAM), or a solid-state drive (SSD). In practical applications, multiple memory modules 112 of different types can be configured in computing node 101. This embodiment does not limit the number or type of memory 112. Furthermore, memory 112 can be configured to have a power-saving function. The power-saving function means that when the system experiences a power outage and then power is restored, the data stored in memory 112 will not be lost. Memory with a power-saving function is called non-volatile memory.

[0109] Network interface card 113 is used to communicate with other application servers 110.

[0110] In a feasible scenario, compute node 101 and storage node 102 are deployed separately. For example... Figure 2b As shown, the compute node cluster includes one or more compute nodes 101 ( Figure 2bThe diagram shows two compute nodes 101, but is not limited to two compute nodes 101. In terms of hardware, such as... Figure 2b As shown, computing node 101 includes at least a processor 121, memory 122, and a network interface card 123. Details of processor 121 are described above in the section on processor 111 and will not be repeated here. Details of memory 122 are described above in the section on memory 112 and will not be repeated here.

[0111] Network interface card 123 is used to communicate with storage node 102. For example, when the total amount of data in memory 122 reaches a certain threshold, compute node 101 can send a request to storage node 102 via network interface card 123 to persistently store the data. Additionally, compute node 101 may also include a bus for communication between components within compute node 101. Functionally, because... Figure 2b The primary function of compute node 101 is computing operations. When storing data, it can utilize remote storage for persistent storage, thus requiring less local storage than a conventional server, resulting in cost and space savings. However, this does not mean that compute node 101 cannot have local storage. In practice, compute node 101 can also have a small amount of internal hard disk or external hard disk.

[0112] Any compute node 101 can access any storage node 102 in the storage node cluster via the network. The storage node cluster includes multiple storage nodes 102 ( Figure 2b The diagram shows two storage nodes 102, but is not limited to two storage nodes 102. A storage node 102 includes one or more controllers 131 (on which the storage node 102 is deployed), a network interface card (NIC) 132, and multiple hard disks 133. The NIC 132 is used to communicate with the compute node 101. The hard disks 133 are used to store data. The controllers 131 are used to write data to, read from, or delete data from the hard disks 133 according to read / write data requests sent by the compute node 101. During the read / write process, the controllers 131 need to convert the address carried in the read / write data request into an address that the hard disk can recognize. Therefore, the controllers 131 also have some basic computing functions.

[0113] The network can be a wired network or a wireless network. For example, a wired network can be a cable network, a fiber optic network, a Digital Data Network (DDN), etc., while a wireless network can be a telecommunications network, an intranet, the Internet, a Local Area Network (LAN), a Wide Area Network (WAN), a Wireless Local Area Network (WLAN), a Metropolitan Area Network (MAN), a Public Service Telephone Network (PSTN), a ZigBee network, a Global System for Mobile Communications (GSM) network, etc., or any combination thereof. Understandably, a network can use any known network communication protocol to enable communication between different client layers and gateways. These network communication protocols can be various wired or wireless communication protocols, such as Ethernet, Universal Serial Bus (USB), FireWire, Global System for Mobile Communications (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Time Division Code Division Multiple Access (TD-SCDMA), Long Term Evolution (LTE), New Radio (NR), and other communication protocols.

[0114] In practical applications, the control unit 131 can take various forms. In one case, the control unit 131 includes a CPU and memory. The CPU is used to perform operations such as address translation and reading / writing data. The memory is used to temporarily store data to be written to the hard disk 133, or to read data from the hard disk 133 to be sent to the computing node 101. In another case, the control unit 131 is a programmable electronic component, such as a data processing unit (DPU). The DPU has the versatility and programmability of a CPU, but is more specialized, and can run efficiently on network packets, storage requests, or analysis requests. The DPU is distinguished from the CPU by its high degree of parallelism (the need to process a large number of requests). Optionally, the DPU here can also be replaced by a graphics processing unit (GPU), an embedded neural network processing unit (NPU), or other processing chips. Typically, the number of control units 131 can be one, two, or more. When storage node 102 contains at least two control units 131, there is an ownership relationship between hard disk 133 and control unit 131. Each controller can only access the hard disk belonging to it. Therefore, this often involves forwarding read / write data requests between control units 131, resulting in a longer data access path. In addition, if storage space is insufficient, adding a new hard disk 133 to storage node 102 requires rebinding the ownership relationship between hard disk 133 and controller 131, which is complex and results in poor scalability of storage space.

[0115] Therefore, in another implementation, the functions of the control unit 131 can be offloaded to the network interface card 132. In other words, the storage node 102 does not have a controller 131; instead, the network interface card 132 performs data reading and writing, address translation, and other computing functions. In this case, the network interface card 132 is a smart network interface card. It can contain a CPU and memory. The CPU is used to perform address translation and data reading and writing operations. The memory is used to temporarily store data to be written to the hard disk 133, or to read data from the hard disk 133 to be sent to the computing node 101. It can also be a programmable electronic component, such as a data processing unit (DPU). There is no ownership relationship between the network interface card 123 and the hard disk 133 in the storage node 102; the network interface card 132 can access any hard disk 133 in the storage node 102, making it easier to expand the hard disk when storage space is insufficient.

[0116] It should be understood that the distributed storage system to which the embodiments of the present invention are applicable can also be applied to Figure 1a Other distributed storage systems besides the one shown are not limited to in this embodiment of the invention.

[0117] In some possible scenarios, such as Figure 1b As shown, the storage node cluster may also include management nodes 104. The number of management nodes 104 can be determined based on actual needs, and this embodiment of the invention does not impose a specific limitation on this. In specific implementations, the nodes in the storage node cluster can be manually divided into management nodes 104 and storage nodes 102. In other possible implementations, the nodes in the storage node cluster negotiate to obtain management nodes 104 and storage nodes 102. It is worth noting that management node 104 is only one possible naming method, and different companies or different distributed storage systems may have different naming methods. For example, it can be a Master node.

[0118] It should be noted that there can be multiple management nodes 104. In one example, a primary-backup deployment can be used, that is, several clones of management node 104 are created. The information of the backup management node 104 must be kept consistent with that of the primary management node 104 at all times. If the primary management node 104 fails, the backup management node 104 immediately takes over. In another example, multiple management nodes 104 can use a consensus algorithm to elect a new leader. If the primary management node 104 fails, the other management nodes 104 use the consensus algorithm to re-elect a leader.

[0119] In related technologies, the distributed storage system 100 maintains a globally consistent view information. This view information describes the rules for distributing data across the storage node cluster. During operation, storage nodes 102 and compute nodes 101 query this view information to determine the physical location for data storage and retrieval. For example, the view information records the data storage status of each storage node 102 in the storage node cluster. This view information may include the data characteristics corresponding to each storage node 102 and the size of the storage space for those data characteristics on the storage node 102. Specifically, the view information may include: how many storage nodes 102 are in the storage node cluster, the remaining capacity of each storage node 102, its load, and which storage nodes 102 store which data, etc.

[0120] The sources of view information are mainly as follows:

[0121] 1. Configuration. This can be understood as the static information of the storage node cluster, such as the initial number of storage nodes 102, the physical topology of storage nodes 102, the capacity of each storage node 102, etc. Figure 1a In this architecture, storage node 102 can load this configuration information at startup. Figure 1bUnder this architecture, management node 104 can load this configuration information at startup.

[0122] 2. Report. In Figure 1b In this architecture, storage node 102 actively reports its status to management node 104 during operation, such as whether storage node 102 is alive, its load information, and what data it stores. During the operation of the distributed storage system 100, storage node 102 periodically reports to management node 104 via heartbeat and other methods.

[0123] With the view information, compute node 101 can have a grasp of the entire storage node cluster's status, and thus make a series of decisions, as follows:

[0124] 1. Scheduling. When a new write request arrives, which storage node (102) should it be assigned to? Usually, a node with a low load is chosen.

[0125] 2. Balance. As storage nodes 102 change and data is added or deleted, the data distribution across different machines may no longer be uniform. This can create read / write hotspots on some machines while causing resource waste on others, thus affecting overall system performance. Therefore, real-time monitoring and timely migration are necessary.

[0126] In related technologies, the distributed storage system 100 performs view management: when a storage node 102 joins or leaves, or when a storage node 102 temporarily fails, the view information is automatically updated. The accuracy and consistency of this view information in the distributed storage system 100 are ensured through passive querying or active updating.

[0127] View management is mainly divided into two methods: centralized management and decentralized management. Centralized management refers to the independent deployment of a single management role process to handle view updates and distribution. In this embodiment, this is called management node 104. Management node 104 can be deployed in a primary / backup configuration. When the primary management node 104 fails, the backup management node 104 can be promoted to primary to achieve failover. Decentralized management involves each storage node 102 managing a portion of the views and synchronizing the view information on its own storage node 102 to other storage nodes 102 through propagation protocols such as the Gossip protocol.

[0128] However, view management has the following problems:

[0129] 1. In a centralized management approach, management node 104 becomes a weak link in the system. If management node 104 fails, the view management function of the entire distributed storage system 100 may be affected. When deploying management node 104 in a primary-backup manner, if the primary management node 104 fails, the view updates of the distributed storage system 100 may be affected during the process of the backup management node 104 becoming the primary node.

[0130] Second, regarding the centralized management approach, as the scale of the distributed storage system 100 increases, the load on the management node 104 will increase rapidly, and view management will become a bottleneck for the distributed storage system 100.

[0131] Third, in decentralized management, when propagating view information of the distributed storage system 100 via a protocol such as Gossip, both time and spatial paths are uncertain. In an environment where storage nodes 102 change frequently, the view update speed may not keep up with the speed of change of storage nodes 102, causing the view information held by some storage nodes 102 to become outdated.

[0132] To address the aforementioned problems, this invention provides the following technical solution for view management.

[0133] Aspect 1: View hierarchical management. For example... Figure 3 As shown, this invention divides view management into three layers according to responsibility: view generator, view pusher, and view consumer. The view generator generates view information for the distributed storage system 100, sends the generated view information to the view pusher, and the view pusher sends the view information to the view consumer. Figure 1a Under the architecture shown, view generators can be pre-configured directly. Figure 1bIn the architecture shown, management node 104 acts as the view producer; the view pusher can be a pre-configured storage node 102, or it can be flexibly configured by the view producer according to the actual situation. Other storage nodes 102 and compute nodes 101 besides the view producer and view pusher act as view consumers. It should be noted that when the number of storage nodes 102 in the storage cluster is small, all storage nodes 102 except the view producer can be used as view pushers. In this case, compute nodes 101 act as view consumers. Compute node 101 selects any one of the view pushers to request view information from the distributed storage system 100. Based on the requested view information from the distributed storage system 100, it performs data I / O on storage node 102, where data I / O represents data reading and data writing. In some possible implementations, compute node 101 can obtain information from the view generator to indicate all view pushers (for ease of description and distinction, this can be called view pusher information); in other possible implementations, the view generator sends the view information of the distributed storage system 100 and the information to indicate all view pushers to the service node 103 for persistence, and the view consumer requests the view pusher information to indicate all view pushers from the service node 103.

[0134] Aspect 2: Fault Handling Mechanism. If any node in the storage node cluster fails, causing a view request to fail, view information can be retrieved from other storage nodes 102 or service nodes 103. In specific implementation, if the primary management node 104 fails, the backup management node 104 quickly becomes the primary. The new management node 104 can obtain the latest view information from service node 103, and the view pusher directly queries the view information from the new management node 104. If the view pusher fails, the view consumer can request view information from other view pushers.

[0135] The beneficial effects that can be achieved by the above technical solution are as follows:

[0136] 1. By having the view generator recommend views to the view pusher, the view synchronization between multiple storage nodes is eliminated, reducing the overhead of view synchronization.

[0137] 2. The view generator is used for view generation and view distribution. Compute nodes request views from the view pusher, which reduces the load on the view generator in view management, making the storage nodes more scalable and allowing for flexible expansion of the storage node scale. This reduces the possibility that view management will become a performance bottleneck in the distributed storage system.

[0138] 3. In scenarios where storage nodes in the distributed storage system 100 change frequently, the view is pushed from the view generator to the view pusher. The view push path and time are predictable, and there is no need for synchronization between storage nodes, thereby improving the efficiency of view distribution.

[0139] Next, in conjunction with the distributed storage system 100 provided above, a view management method provided by an embodiment of the present invention will be described in detail. The present invention will use the view generator as the management node 104, and the view generator, view pusher, and view consumer as the view production node, view pusher node, and view consumer node, respectively, to illustrate the solution of the present invention. It is assumed that the storage node cluster includes N storage nodes 102, and the N storage nodes 102 include a first view generation node. Figure 1a Under the architecture shown, the first view generation node can be pre-configured, or N storage nodes 102 can negotiate to determine the first view generation node; Figure 1b In the architecture shown, management node 104 acts as the first view generation node.

[0140] Figure 4 This is a flowchart illustrating the view management method provided in an embodiment of the present invention. This embodiment can be applied to a distributed storage system 100, such as... Figure 4 As shown, the view management method provided in this embodiment of the invention includes at least the following steps:

[0141] Step 401: The first view generation node determines M view push nodes from N storage nodes 102.

[0142] In some possible implementations, the first view generating node can use each storage node 102 other than the first view generating node as a view pushing node.

[0143] In some possible implementations, the first view generation node can use some of the M storage nodes 102 as view push nodes. The rules for selecting view push nodes are as follows:

[0144] 1. Random selection;

[0145] 2. Select according to proportion, for example, select a preset proportion of storage nodes 102 from M storage nodes 102 to act as view push nodes;

[0146] 3. Considering the hardware configuration of storage node 102, choose storage node 102 with the highest possible hardware configuration to serve as the view push node, so as to ensure that the view push node can perform tasks more efficiently.

[0147] 4. Considering reliability and stability, storage nodes 102 from different regions should be selected as much as possible to ensure the reliability and stability of the view push nodes. 5. Pre-configuration: Users pre-configure the view push node files; the first view generates a node parsing file to determine the M view push nodes.

[0148] Step 402: The first view generation node sends view information to the M view push nodes. The view information is used to indicate the rules for distributing data to N storage nodes.

[0149] The view information is used to describe the rules for distributing data to the storage node cluster. For example, the view information is used to record the data storage situation of each storage node 102 in the storage node cluster. For example, the view information includes the data characteristics corresponding to the storage node 102 and the size of the storage space of the data characteristics in the storage node 102.

[0150] In this embodiment of the invention, the data distribution can take into account factors such as data uniformity, data stability, node heterogeneity, and factors such as isolated fault domains and performance stability.

[0151] From the perspective of data uniformity, this mainly includes two aspects: First, the data stored across different storage nodes 102 should be as evenly distributed as possible to avoid overloading one or a few storage nodes 102 while leaving others with almost no data. For example, with 100GB of data across four identical storage nodes 102, it's desirable to distribute data as evenly as possible, such as storing 25GB of data on each node 102. Second, user access should also be evenly distributed to avoid situations where one or a few storage nodes 102 experience high access volumes while others remain untouched. For instance, with 1000 requests, the handling of user access requests across the four storage nodes 102 should be as even as possible, such as each node handling 250 requests. In practice, the difference between the requests handled by each node 102 should not be too significant.

[0152] From the perspective of data stability, when storage node 102 fails and needs to be removed or expanded, large-scale data migration should be avoided. For example, if there is 100GB of data and initially there are four identical storage nodes 102 (storage nodes 1-4), with each storage node 102 storing 25GB of data, and now storage node 2 fails, the data on node 2 should be migrated only to other nodes as much as possible, without migrating or storing a large amount of data, thus ensuring data stability.

[0153] From the perspective of node heterogeneity, the hardware configurations of different storage nodes 102 may vary greatly. For example, some storage nodes 102 may have high-end hardware configurations, capable of storing large amounts of data and handling more requests; however, some storage nodes 102 may have less advanced hardware configurations, limiting the amount of data they can store and the number of user accesses. For storage nodes 102 with such significant hardware differences, the amount of data allocated and the number of user accesses are roughly the same, which is essentially an imbalance.

[0154] From the perspective of isolating fault domains, this is to ensure data availability and reliability. For example, data reliability is usually achieved through backups. However, if each piece of data and its backup are distributed across the same hard drive or storage node 102, data availability and reliability cannot be guaranteed. Therefore, each piece of data should be mapped to a set of storage nodes 102, and these storage nodes 102 should be located in different fault domains as much as possible, such as different data centers or different racks, to ensure data availability and reliability.

[0155] From the perspective of performance stability, it is necessary to ensure the efficiency of data storage and retrieval, and the addition or removal of storage node 102 should not cause a serious decline in storage or access performance.

[0156] Step 403: Calculate node 101 to determine the first view push node from the M view push nodes.

[0157] In some possible implementations, compute node 101 may randomly select a view push node as the first view push node from the view push nodes indicated by the view push node information.

[0158] In some other possible implementations, computing node 101 can select the nearest view push node as the first view push node based on geographical affinity.

[0159] In some other possible implementations, compute node 101 may select the view push node with the lowest load as the first view push node.

[0160] Step 404: Compute node 101 pushes the node request for view information to the first view.

[0161] In a specific implementation, the compute node 101 sends a view information request to the first view push node, and the first view push node sends view information to the compute node 101. It should be noted that the view information sent by the first view push node to the compute node 101 is the latest view information, which is the last view information received by the first view push node from the first view generation node.

[0162] Step 405: Compute node 101 writes or reads data from N storage nodes 102 based on the requested view information.

[0163] In a specific implementation, compute node 101 generates data operation requests using view information and sends them to storage node 102. These data operation requests can be generated for processing data access requests from external sources or for processing requests generated internally within the distributed storage system 100. For example, processing may include metadata management, deduplication, data compression, data verification, virtualization of storage space, and address translation. Storage node 102 stores data, receives data operation requests from compute node 101, and performs data operations based on these requests. For instance, a data operation request may instruct the current data to be written to the target storage node 102, or to read data from the target storage node 102.

[0164] In one example, if data needs to be written, compute node 101 queries view information based on the current data to be written, determines the target storage node 102 to which the current data to be written needs to be written, and generates a data operation request based on the current data to be written and the target storage node 102.

[0165] In one example, if data needs to be read, the view information is queried based on the characteristics of the data to be read, the target storage node 102 where the data to be read is located is determined, and a data operation request is generated based on the characteristics of the data to be read and the target storage node 102.

[0166] In the solution provided by this invention, on the one hand, the view production node is used for view generation and view distribution, and the compute node requests the view from the view push node, which reduces the load on the view production node in view management, making the storage node more scalable and able to flexibly expand the scale of the storage node, reducing the possibility that view management will become a performance bottleneck; on the other hand, there is no need for view synchronization between multiple storage nodes, which reduces the overhead of view synchronization; furthermore, in the scenario where storage nodes in the distributed storage system change frequently, the view is pushed to the view push node by the view production node, and the view push path and time are predictable, thereby improving the efficiency of view distribution.

[0167] In some possible embodiments, such as Figure 3 and Figure 6 As shown, the view push node sends view information to the view consumer node.

[0168] Figure 5 A flowchart illustrating another view management method provided by an embodiment of the present invention is shown.

[0169] like Figure 5 As shown, in Figure 4 Based on steps 401 to 405 shown, this embodiment of the invention further includes at least the following steps:

[0170] Step 501: The first view generation node determines the association between the second view push node and the H view consumption nodes. The H view consumption nodes are the storage nodes 102 other than the first view generation node and the M view push nodes among the N storage nodes 102, where H is a positive integer less than N; the second view push node is any one of the M view push nodes.

[0171] In some possible implementations, the first view generating node uses a portion of the N storage nodes 102 as view pushing nodes and the remaining H unselected storage nodes 102 as view consuming nodes. In this embodiment of the invention, H is less than or equal to N - M - the number of view generating nodes.

[0172] In practical implementation, the first view generation node determines the association relationship between the second view push node (any one of the M view push nodes) and the H view consumer nodes for the second view push node. This association relationship describes the view consumer nodes associated with the second view push node, which are a subset of the H view consumer nodes. In one possible scenario, the first view generation node can consider regional affinity to determine the view consumer nodes associated with the second view push node. Specifically, for the second view push node, the second view push node and its associated view consumer nodes have a regional affinity relationship. For example, the view consumer node and the second view push node may be located in the same data center, or the view consumer node and the second view push node may be located in different data centers, with the distance between the different data centers not exceeding a preset threshold.

[0173] Step 502: The first view generation node sends the association relationship to the second view push node.

[0174] Step 503: Based on the association relationship, the second view push node determines the view consumer node associated with the second view push node and sends view information to the associated view consumer node.

[0175] In a specific implementation, the view information sent by the second view push node to the associated view consumer node is the latest view information. The latest view information is the view information that the second view push node last received from the first view generation node. For example, the second view push node can push the latest view information to the associated view consumer node, and the view consumer node will also pull the latest view information from the associated second view push node.

[0176] It is worth noting that in practical applications, the first view generation node needs to determine the association between each of the M view push nodes and the H view consumption nodes. For example, the H view consumption nodes can be evenly distributed among each view push node. Subsequently, each view push node is responsible for pushing the latest view information to the associated storage node 102.

[0177] In the solution provided by this invention, the view push node is associated with the view consumer node. The view push node needs to synchronize the view to the associated view consumer node, without the view generation node needing to synchronize the view to the view consumer node. This reduces the load on the view production node in view management and makes the storage node more scalable. The scale of the storage node can be flexibly expanded without affecting view management.

[0178] In some possible embodiments, such as Figure 6 As shown, the view generation node can send view push node information (used to indicate all view push nodes) to the service node 103 for storage. The view consumer node can request view push node information from the service node 103 and select the view push node to request view information based on the requested view push node information. It should be noted that the view consumer node obtaining view push node information from the service node 103 is merely an example and does not constitute a specific limitation. In some possible implementations, the view consumer node can also obtain view push node information from the view generation node.

[0179] Figure 7 A flowchart illustrating another view management method provided by an embodiment of the present invention is shown.

[0180] like Figure 7 As shown, in Figure 4 Based on steps 401 to 405 shown, this embodiment of the invention further includes at least the following steps:

[0181] Step 701: The first view generation node determines the view push node information, which is used to indicate M view push nodes.

[0182] The view push node information can be an identifier, such as a number, for each view push node. For example, the view push node information can be a list, where each item in the list is an identifier, such as a number, for the view push node.

[0183] Step 702: The first view generation node sends view push node information to the service node 103.

[0184] Step 703: Service node 103 stores view push node information.

[0185] In the actual implementation, service node 103 will persistently store the view push node information to ensure that the view push node information is not lost.

[0186] Then step 403 includes:

[0187] Step 4031: Compute node 101 requests view push node information from service node 103.

[0188] In a specific implementation, compute node 101 sends a view push node information request to service node 103, and service node 103 sends view push node information to compute node 101. It should be noted that the view push node information sent by service node 103 to compute node 101 is the latest. The latest view push node information is the view push node information sent by the first view generation node that service node 103 last received.

[0189] Step 4032: Compute node 101 determines the first view push node from M view push nodes based on the request to view push node information.

[0190] In the solution provided by this invention, the computing node can select a view push node from a large number of view push nodes to request a view, without having to request it from the view generation node. This reduces the load on the view production node in view management, makes the storage node more scalable, and allows for flexible expansion of the storage node scale, reducing the possibility that view management will become a performance bottleneck.

[0191] In some possible embodiments, such as Figure 6 As shown, the view generation node monitors the node status of each storage node 102 in the storage cluster through the service node 103, dynamically updates the view information based on the node status of each storage node 102 in the storage cluster, and sends it to the view push node. The view push node then sends it to the view consumer node, thereby sending it to all storage nodes 102.

[0192] Figure 8 A flowchart illustrating another view management method provided by an embodiment of the present invention is shown. Figure 7 As shown, in Figure 4 Based on steps 401 to 405 shown, this embodiment of the invention further includes at least the following steps:

[0193] Step 801: Service node 103 sends the node status of N storage nodes to the first view generation node.

[0194] For example, the node status can be normal operation or failure.

[0195] Step 802: The first view generation node determines the view information based on the node status and data distribution rules of N storage nodes.

[0196] In a practical implementation, the management node 104 can deploy an algorithm that describes the data distribution rules. The node status of each storage node 102 in the storage node cluster is input into the algorithm, and the algorithm outputs view information.

[0197] It should be noted that the view information is the latest view information, which is the view information last determined by the management node 104. Specifically, when the distributed storage system 100 initially runs, the first view information obtained is the latest view information. Subsequently, whenever the node state of any storage node 102 in the storage node cluster changes, the view information needs to be updated, and the updated view information becomes the latest view information. For details, please refer to the description of step 402 above, which will not be repeated here.

[0198] Step 803: Service node 103 determines that the node state of the target storage node 102 among the N storage nodes 102 has changed.

[0199] In some possible implementations, the change in node status can be due to the storage node 102 changing from normal to faulty, or from faulty to normal, or it can be due to being added. In the case of a newly added node, the target storage node 102 is the newly added storage node 102.

[0200] It should be noted that the target storage node 102 is a storage node 102 other than the view production node, and the number can be one or more.

[0201] Step 804: Service node 103 sends the updated node status of target storage node 102 to the first view generation node.

[0202] Step 805: The first view generation node updates the view information based on the updated node status and data distribution rules of the target storage node 102, and obtains the updated view information.

[0203] In practice, the management node 104 updates the view information based on the updated node status of the target storage node 102, the node status of other storage nodes 102, and the data distribution rules.

[0204] Step 806: The first view generation node sends the updated view information to the M view push nodes.

[0205] Subsequently, if compute node 101 sends a view information request to the view push node, the view push node will respond with the updated view information.

[0206] In the solution provided by this invention, the node status monitoring of storage nodes is undertaken by the service node, eliminating the need for the view generation node to monitor the storage node status. This reduces the load on the view production node in view management, giving the storage node strong scalability and allowing for flexible expansion of the storage node scale, thus reducing the possibility of view management becoming a performance bottleneck. Furthermore, in scenarios where storage nodes frequently change in a distributed storage system, the view production node can improve the speed of view updates by monitoring the node status through the service node.

[0207] In some possible embodiments, such as Figure 6 As shown, the view generation node sends the view information to the service node 103. The service node 103 is used to permanently store the view information, thereby reducing the impact on view management in the event of a failure of any management node in the distributed storage system.

[0208] Figure 9 A flowchart illustrating another view management method provided by an embodiment of the present invention is shown.

[0209] like Figure 9 As shown, in Figure 4 Based on steps 401 to 405 shown, this embodiment of the invention further includes at least the following steps:

[0210] Step 901: The first view generation node sends view information to the service node 103.

[0211] It should be noted that the view information is the latest view information, which is the view information last determined by the management node 104. Specifically, when the distributed storage system 100 initially runs, the first view information obtained is the latest view information. Subsequently, whenever the node state of any storage node 102 in the storage node cluster changes, the view information needs to be updated, and the updated view information becomes the latest view information. For details, please refer to the description of step 402 above, which will not be repeated here.

[0212] Step 902: Service node 103 stores view information.

[0213] In the specific implementation, the service node 103 will persistently store the view information to ensure that the view information is not lost, thereby reducing the impact on view management in the event of a failure of any management node in the distributed storage system 100.

[0214] In some possible embodiments, if the view push node fails, the view generation node can select storage node 102 to replace the failed view push node and update the view push node information.

[0215] Figure 10A flowchart illustrating another view management method provided by an embodiment of the present invention is shown.

[0216] like Figure 10 As shown, in Figure 4 Based on steps 401 to 405 shown, this embodiment of the invention further includes at least the following steps:

[0217] Step 1001: The first view generation node determines that the third view push node among the M view push nodes is faulty. From the N storage nodes 102 other than the first view generation node and the M view push nodes, the fourth view push node is selected to replace the third view push node, and the updated view push node information is obtained.

[0218] It should be noted that when the node status of the third view push node is faulty, service node 103 sends the node status of the third view push node to the first view generation node. The first view generation node determines the fault of the third view push node based on the node status of the third view push node sent by service node 103. It is important to note that the fault of the third view push node indicates a change in the node status of storage node 102 in the distributed storage system 100. In this case, the first view generation node needs to update the view information.

[0219] Step 1002: The first view generation node sends view information to the fourth view push node.

[0220] Step 1003: The first view generation node sends the updated view push node information to the service node 103.

[0221] Step 1004: Service node 103 stores the updated view push node information.

[0222] Subsequently, if compute node 101 sends a view push node information request to service node 103, service node 103 will respond with an updated push node information request.

[0223] It should be noted that, in the scenario where the target storage node 102 in step 804 is a third-view push node and the node status of the target storage node 102 is faulty, this embodiment of the invention is not intended to limit the scope of the invention. Figure 8 The execution order of steps 805 and 1001 can be specified, for example, steps 805 and 1001 can be executed in parallel.

[0224] In the solution provided by this invention, the view production node replaces the faulty view push node after the view push node fails, thereby achieving fault handling.

[0225] In some possible embodiments, the distributed storage system 100 also includes a second view generation node, which can be a backup node of the first view generation node. The first view generation node can synchronize the latest view information and view push node information to the second view generation node. After the first view generation node fails, the second view generation node takes over and replaces the failed first view generation node by obtaining the latest view information and view push node information from the service node 103 to realize view management. Figure 11 A flowchart illustrating another view management method provided by an embodiment of the present invention is shown.

[0226] like Figure 11 As shown, in Figure 4 Based on steps 401 to 405 shown, this embodiment of the invention further includes at least the following steps:

[0227] Step 1101: When the first view is generated, the second view is generated and a request for view-related information is sent to the service node 103.

[0228] It should be noted that the first view generation node sends heartbeat information to the second view generation node, and the second view generation node can determine whether the first view generation node is faulty based on the received heartbeat information.

[0229] In practical implementation, if there is only one second view generation node in the distributed storage system 100, and it acts as a slave node of the first view generation node, then the second view generation node will take over from the first view generation node. If there are multiple second view generation nodes in the distributed storage system 100, then an election can be held to select a second view generation node to take over.

[0230] Step 1102: Service node 103 sends the latest view information and the latest view push node information to the second view generation node.

[0231] It should be noted that service node 103 will send the view information and view push node information that it last received from the first view generation node as the latest view information and the latest view push node information to the second view generation node.

[0232] In the solution provided by this invention, after a view production node fails, other view production nodes take over view management, read view information and view push node information from the service node, and improve the efficiency of view production node failure handling.

[0233] In some possible implementations, if a view push node fails, a view consumer node can request view information from other view push nodes.

[0234] Figure 12 A flowchart illustrating another view management method provided by an embodiment of the present invention is shown.

[0235] like Figure 12 As shown, in Figure 4 Based on steps 401 to 405 shown, this embodiment of the invention further includes at least the following steps:

[0236] Step 1201: Calculation node 101 determines that the request for view information to the first view push node failed. Based on the requested view push node information, the fifth view push node is determined.

[0237] It should be noted that in the event of a failure of the first view push node, if compute node 101 requests view information from the first view push node, the first view push node will return an error code indicating that the request for view information failed. It is important to note that a failure of the first view push node signifies a change in the node state of storage node 102 in the distributed storage system 100. In this case, the first view generation node needs to update the view information, and also update the view push node information.

[0238] Step 1202: Compute node 101 sends a request for view information to the fifth view push node.

[0239] Step 1203: When the compute node 101 determines that the request for view information has failed, it requests the view push node information from the service node 103.

[0240] In this embodiment of the invention, if the number of times the computing node 101 requests view information reaches a preset threshold, or if it fails to request view information within a preset time period, it indicates that a large-scale failure has occurred in the view push node, and the request for view information has failed. If the request for view information fails, it indicates that the view push node information may have expired, and in this case, it is necessary to request the view push node information again.

[0241] In a specific implementation, compute node 101 sends a view push node information request to service node 103, and service node 103 sends view push node information to compute node 101. It should be noted that the view push node information sent by service node 103 to compute node 101 is the latest. The latest view push node information is the view push node information sent by the first view generation node that service node 103 last received.

[0242] Step 1204: Compute node 101 determines the sixth view push node based on the latest view push node information requested.

[0243] Step 1205: Compute node 101 pushes the node request for view information to the sixth view.

[0244] In the specific implementation, the computing node 101 sends a view information request to the sixth view push node, and the sixth view push node sends view information to the computing node 101. It should be noted that the view information sent by the sixth view push node to the computing node 101 is the latest, and the latest view information is the view information last determined by the first view generation node.

[0245] Step 1206: Based on the latest view information obtained from the request, compute node 101 performs data writing or data reading on N storage nodes 102.

[0246] For details, please refer to the description of step 405 above, which will not be repeated here.

[0247] In the solution provided by this invention, when a computing node determines that it cannot request view information, it requests the latest view push node information from the service node and re-determines the view push node to request the view, thereby quickly updating the view when the view expires.

[0248] Figures 4 to 12 The embodiments shown are merely basic examples of the method of the present invention. Other preferred embodiments of the method can be obtained by making certain optimizations and extensions based on them.

[0249] In one embodiment, this embodiment, based on the foregoing embodiments, provides a more detailed description and certain optimizations to the view management process. The method in this embodiment includes:

[0250] Figure 13 This is a flowchart illustrating the view management method provided in an embodiment of the present invention. This embodiment can be applied to a distributed storage system 100, where the management node 104 acts as the view generation node, such as... Figure 13 As shown, the view management method provided in this embodiment of the invention includes at least the following steps:

[0251] Step 1301: Each storage node 102 in the storage node cluster sends a heartbeat signal to the service node 103.

[0252] In this embodiment of the invention, the storage node 102 can send a heartbeat signal to the service node 103 according to a preset period.

[0253] Step 1302: Service node 103 determines the node status of each storage node 102 in the storage node cluster based on the received heartbeat signal.

[0254] It should be noted that the node status of storage node 102 is the latest node status.

[0255] Step 1303: Service node 103 sends the node status of each storage node 102 in the storage node cluster to management node 104.

[0256] It should be noted that when the distributed storage system 100 is first run, the service node 103 needs to send the initial node status of each storage node 102 in the storage node cluster to the management node 104. Subsequently, when the service node 103 discovers that the node status of the storage node 102 in the storage node cluster has been updated, it can send the updated node status to the management node 104.

[0257] Step 1304: Management node 104 determines view information based on the node status and data distribution rules of each storage node 102 in the storage node cluster.

[0258] It should be noted that the view information is the latest view information, which is the view information last determined by the management node 104. Specifically, when the distributed storage system 100 initially runs, the first view information obtained is the latest view information. Subsequently, whenever the node state of any storage node 102 in the storage node cluster changes, the view information needs to be updated, and the updated view information becomes the latest view information. For details, please refer to the description of step 402 above, which will not be repeated here.

[0259] Step 1305: Management node 104 sends view information to service node 103.

[0260] Step 1306: Service node 103 stores view information.

[0261] In the specific implementation, the service node 103 will persistently store the view information to ensure that the view information is not lost, thereby reducing the impact on view management in the event of a failure of any management node in the distributed storage system 100.

[0262] Step 1307: Management node 104 determines the view push node information and association information. The view push node information indicates the M view push nodes selected from the storage node cluster, and the association information indicates the information of the view consumer nodes associated with the M view push nodes respectively.

[0263] For details, please refer to the description of step 401, which will not be repeated here.

[0264] Step 1308: Management node 104 sends view push node information to service node 103.

[0265] Step 1309: Service node 103 stores view push node information.

[0266] In the actual implementation, service node 103 will persistently store the view push node information to ensure that the view push node information is not lost.

[0267] Step 1310: Management node 104 sends view information and information of associated view consumer nodes to M view push nodes respectively.

[0268] In a specific implementation, the view information sent by the management node 104 to the view push node is the latest view information, which is the view information last determined by the management node 104. For example, the management node 104 can push the latest view information to M view push nodes at the same time, and the view push nodes will also pull the latest view information from the management node 104.

[0269] Step 1311: The M view push nodes send view information to their associated view consumer nodes.

[0270] Step 1312: Compute node 101 sends a view push node information request to service node 103.

[0271] In some possible implementations, compute node 101 may start sending view push node information requests to service node 103 after startup, until it receives the view push node information sent by service node 103.

[0272] Step 1313: Service node 103 sends view push node information to compute node 101.

[0273] In the specific implementation, the view push node information sent by the service node 103 to the computing node 101 is the latest. The latest view push node is the view push node information last received by the service node 103 from the management node 104.

[0274] Step 1314: Compute node 101 determines the first view push node based on the requested view push node information.

[0275] For details, please refer to the description of step 403 above, which will not be repeated here.

[0276] Step 1315: Compute node 101 sends a view information request to the first view push node.

[0277] Step 1316: The first view push node sends view information to the computing node 101.

[0278] In the specific implementation, the view information sent by the first view push node to the computing node 101 is the latest view information, which is the last view information sent by the management node 104 that the first view push node received.

[0279] Step 1317: Compute node 101 writes or reads data from storage node 102 in the storage node cluster based on the requested view information.

[0280] In this solution, firstly, the view production node is used for view generation and distribution, while the service node handles the monitoring of the storage node's status. The compute node requests views from the view push node, reducing the load on the view production node in view management and enabling the storage node to have strong scalability, allowing for flexible expansion of the storage node scale and reducing the possibility of view management becoming a performance bottleneck. Secondly, there is no need for view synchronization between multiple storage nodes, reducing the overhead of view synchronization. Thirdly, in scenarios where storage nodes in the distributed storage system change frequently, the view production node updates the view by monitoring the node status through the service node, thereby improving the speed of view updates. In addition, the view production node pushes the view to the view push node, and the view push path and time are predictable, thus improving the efficiency of view distribution. Fourthly, the view generation node sends the view information to the service node for storage, which reduces the impact on view management in the event of a failure of any management node in the distributed storage system 100.

[0281] It should be noted that in practical applications, the distributed storage system 100 deploys various processes, such as a master management process, store worker processes, storage clients, and ZooKeeper. The master management process resides on management node 104, store worker processes reside on storage node 102, storage clients reside on compute node 101, and ZooKeeper resides on service node 103. In this embodiment, the master management process can act as a view producer node, each store worker process as a view push node, and the storage client as a view consumer node. Alternatively, the master management process can act as a view producer node, some store worker processes as view push nodes, and the storage client and another group of store worker processes as view consumer nodes.

[0282] In summary, combining Figures 3 to 13 In this embodiment of the invention, the service node 103 may have the following functions:

[0283] 1. Monitor the status of the distributed storage system 100, including the addition, removal, and other changes of nodes, and maintain the node information of the distributed storage system 100. Specifically, service node 103 can deploy a ZooKeeper service to monitor the node status of the distributed storage system 100.

[0284] 2. Persistent storage of view information and view push node information of distributed storage system 100.

[0285] The view generation node has the following functions:

[0286] 1. The view generation node monitors the node status of the entire distributed storage system 100 through the ZooKeeper service deployed on service node 103, including the addition, removal and other changes of nodes, and maintains the node information of the distributed storage system 100.

[0287] 2. The view generation node generates view information for the distributed storage system 100. Specifically, the view generation node generates view information for the distributed storage system 100 based on the node status of the current storage node 102 and the data distribution rules, and sends the view information to the service node 103 for persistent storage.

[0288] 3. The view generation node elects a batch of view push nodes from the storage nodes 102. For example, the view generation node selects a batch of nodes from all the storage nodes 102 to act as view push nodes, and associates the unselected storage nodes 102 with each view push node. Each view push node is responsible for synchronizing view information to its associated storage nodes 102. The rules for the view generation node to select view push nodes are: ① When the distributed storage system 100 is initialized, a batch of storage nodes 102 is randomly selected; ② When a view push node fails, the failed view push node is removed, and an unfailed storage node 102 is randomly selected from the remaining storage nodes 102 as a new view push node.

[0289] 4. The view generation node sends the information from the view push node to the service node 103 for persistent storage.

[0290] 5. For each view push node, the view generation node pushes the view information of the distributed storage system 100 and the relationship between the view push node and the view consumer node. The view push node also pulls the latest view information from the view generation node.

[0291] 6. The view generation node updates and pushes view information in real time. The view generation node monitors the node status of each storage node 102 in the distributed storage system 100 through ZooKeeper. When it detects that a storage node 102 has joined or left, it recalculates the view information and pushes it to the view push node.

[0292] 7. When a view generation node fails, other view generation nodes take over. The taking over view generation node pulls the latest view information and the latest view push node information from service node 103. The taking over view generation node pushes the latest view information to the view push node, and the view push node also pulls the latest view information from the taking over view generation node. For example, when the view generation node is deployed in a primary / backup management node 104 configuration, if the primary management node 104 fails, the backup management node 104 quickly becomes the primary, achieving failover, and the view push node directly pulls the latest view information from the backup management node 104.

[0293] View push nodes can have the following functions:

[0294] The system retrieves the latest view information from the view generation node; upon receiving a view information request from the compute node 101, it sends the latest view information to the compute node 101; when it needs to send the view information of the distributed storage system 100 to other storage nodes 102, it sends the view information to the associated view consumer node based on the association relationship sent by the management node 104 with the view consumer node, and the view consumer node also retrieves the latest view information from the view push node. After the view information is updated, it sends the updated view information to the associated view consumer node.

[0295] Storage node 102, which acts as a view consumer node, can have the following functions:

[0296] Retrieve the latest view information from the view push node, and receive the latest view information sent by the view push node.

[0297] Compute node 101 can have the following functions:

[0298] 1. After obtaining the view information, compute node 101 writes or reads data. Compute node 101 first obtains the view push node information from service node 103, randomly determines a view push node as the source node of the view, and actively pulls the view information from the source node.

[0299] 2. If compute node 101 fails to request view information from the view push node, compute node 101 requests view information from the remaining view push nodes. If the failure exceeds a certain number of times or exceeds a certain time, it obtains the latest view push node information from service node 103, determines a view push node as the source node of the view, and actively pulls view information from that source node.

[0300] Based on the view management methods provided above, the specific application of these methods will be explained. Figure 14 This is a flowchart illustrating a specific application of a view management method provided for the implementation of this invention. For example... Figure 14 As shown, the specific content includes:

[0301] like Figure 14 As shown, the distributed storage system 100 includes a master management node 1, slave management nodes 2, multiple storage nodes 102, multiple compute nodes 101, and a service node 103. The service node 103 is equipped with a cluster view persistence component (ZooKeeper). The multiple compute nodes 101 act as view consumer nodes, denoted as compute node C0, ..., compute node C p Multiple storage nodes 102 are denoted as storage node A0, ..., storage node A M Storage node B0, ..., Storage node B H .

[0302] exist Figure 14 On the basis of, such as Figure 15 As shown, the distributed storage system 100 uses the following process management system view.

[0303] (1) The master management node 1 generates view information for the distributed storage system 100 and elects a batch of view push nodes from storage nodes 102: storage node A0, ..., storage node A M And the remaining storage nodes B0, ..., storage nodes B H As a view consumer node, it persists the view information and view push node list of the distributed storage system 100 to the distributed coordination service.

[0304] (2) The master management node 1 synchronizes the view information of the distributed storage system 100 to the slave management node 2.

[0305] (3) The main management node 1 pushes nodes to the view: storage node A0, ..., storage node A M Push the latest view information; view push nodes: storage node A0, ..., storage node A M Pull the latest view information from the main management node 1.

[0306] (4) View push nodes: storage node A0, ..., storage node A M Consumer nodes for the view: storage node B0, ..., storage node B H Pushing the latest view information, view consumer nodes: storage node B0, ..., storage node B H Pushing nodes to the view: Storage node A0, ..., Storage node A M Retrieve the latest view information.

[0307] (5) Compute node C0 queries the distributed coordination service for the latest view push node list.

[0308] (6) Compute node C0 randomly selects a view push node based on the latest list of view push nodes, such as storage node A. M Send to storage node A M Request the latest view information.

[0309] exist Figure 15 On the basis of, such as Figure 16 As shown, when a node in the distributed storage system 100 fails, the following fault handling mechanism can be adopted.

[0310] (1) When the primary management node 1 fails, the backup management node 2 takes over and becomes the new cluster manager.

[0311] (2) Backup management node 2 reads the latest view information and view push node list from the distributed coordination service.

[0312] (3) Backup management node 2 pushes nodes to the view: storage node A0, ..., storage node A M Push the latest view information; view push nodes: storage node A0, ..., storage node A M Retrieve the latest view information from backup management node 2.

[0313] (4) If storage node A0 fails, compute node C0 will first push nodes to other views (e.g., storage node A). M Request the latest view information; if that still fails, continue trying other view push nodes.

[0314] (5) When a view request fails more than a certain number of times or after a certain period of time, the compute node C0 obtains the latest list of view push nodes from the distributed coordination service, and then continues to select view push nodes to obtain the latest view information based on the latest list of view push nodes.

[0315] This invention also provides a view management system, which can be found below. Figure 17 , Figure 17This is a schematic diagram of a view management system provided in an embodiment of the present invention. Details are as follows:

[0316] A view generation device is used to determine M view push nodes from N storage nodes, where M is a positive integer less than or equal to N;

[0317] A view generation device is used to send view information to M view push nodes. The view information is used to indicate the rules for distributing data to N storage nodes.

[0318] A computing device is configured to determine a first view push node from M view push nodes, request view information from the first view push node, and write or read data from N storage nodes based on the requested view information.

[0319] In some possible implementations, the view generation device is used to determine the association between the second view push node and H view consumer nodes, where the H view consumer nodes are storage nodes other than the first view generation node and the M view push nodes among the N storage nodes, and H is a positive integer less than N; the second view push node is any one of the M view push nodes.

[0320] A view generation device is used to send association relationships to a second view push node, so that the second view push node can determine the view consumer node associated with the second view push node based on the association relationships, and send view information to the associated view consumer node.

[0321] In some possible implementations, the distributed storage system 100 may also include a service unit.

[0322] In one embodiment, the service device is used to send the node status of N storage nodes to the view generation device;

[0323] A view generation device is used to determine view information based on the node status and data distribution rules of N storage nodes; and to send the view information to M view push nodes.

[0324] In one embodiment, the service device is configured to send the updated node status of the target storage node to the view generation device after determining that the node status of the target storage node among N storage nodes has been updated.

[0325] In one embodiment, the view generation device is used to determine view push node information, which is used to indicate M view push nodes; and to send the view push node information to the service device.

[0326] A computing device is used to request view push node information from a service device; and to determine a first view push node from M view push nodes based on the view push node information.

[0327] In one embodiment, the view generation device is used to send view information to the service device; the service device is used to store the view information.

[0328] In some possible implementations, the view generation device is used to determine that the third view push node among M view push nodes is faulty, and selects a fourth view push node from among the storage nodes other than the first view generation node and the M view push nodes among N storage nodes to replace the third view push node, thereby obtaining updated view push node information.

[0329] A view generation device is used to send updated view push node information to the service device;

[0330] A view generation device is used to send view information to the fourth view push node.

[0331] In some possible implementations, a computing device is used to determine that requesting view information to the first view push node has failed, and based on the view push node information, to determine the fifth view push node and request view information to the fifth view push node.

[0332] In this implementation example, the computing device determines that the request for view information has failed, requests the latest view push node information from the service device, determines the sixth view push node based on the latest view push node information, and requests the latest view information from the sixth view push node.

[0333] For example, the computing device is configured to determine that the request for view information has failed if the number of failed requests for view information is greater than or equal to a preset number, or if no view information is requested within a preset time period.

[0334] In this implementation example, the distributed storage system also includes a replacement device; when the view generation device fails, the replacement device reads view information and view push node information from the service device. It should be noted that "replacement device" and "view generation device" are merely possible naming conventions; in some possible scenarios, the view generation device may be referred to as the first view generation device, and the replacement device may be referred to as the second view generation device.

[0335] It should be noted that, in this embodiment of the invention, the service device is used to implement... Figures 3 to 13 The steps implemented by service node 103 in the middle, the view generation device is used to implement Figures 3 to 13 The steps implemented by the first view generation node or management node 104 in the computing device are as follows: Figures 3 to 13 The steps implemented by computing node 101 in the process are as follows. It is worth noting that the service device is not a necessary device.

[0336] The view generation device, service device, and computing device can all be implemented in software or hardware. For example, the implementation of the view generation device will be described below. Similarly, the implementation of the service device and computing device can refer to the implementation of the view generation device.

[0337] As an example of a software functional unit, a view generation apparatus may include code running on a compute instance. This compute instance may include at least one of a physical host (computing device), a virtual machine, or a container. Further, the compute instance may be one or more. For example, the view generation apparatus may include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the code may be distributed in the same region or in different regions. Further, the multiple hosts / virtual machines / containers used to run the code may be distributed in the same availability zone or in different availability zones, each availability zone including one data center or multiple geographically proximate data centers. Typically, a region may include multiple availability zones.

[0338] As an example of a hardware functional unit, a view generation device may include at least one computing device, such as a server. Alternatively, the view generation device may also be a device implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The aforementioned PLD may be implemented using a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), generic array logic (GAL), or any combination thereof.

[0339] It should be noted that, in other embodiments, the view generation device can be used to execute any step in the view management method, the service device can be used to execute any step in the view management method, the computing device can be used to execute any step in the view management method, and the steps implemented by the view generation device, the service device, and the computing device can be specified as needed. By implementing different steps in the view management method through the view generation device, the service device, and the computing device, all functions of the management platform can be realized.

[0340] The present invention also provides a computing device 1800. For example... Figure 18 As shown, the computing device 1800 includes a bus 1802, a processor 1804, a memory 1806, and a communication interface 1808. The processor 1804, the memory 1806, and the communication interface 1808 communicate with each other via the bus 1802. The computing device 1800 can be a server or a terminal device. It should be understood that the present invention does not limit the number of processors and memories in the computing device 1800.

[0341] The 1802 bus can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of representation, Figure 18 The bus is represented by only one line, but this does not mean that there is only one bus or one type of bus. Bus 1804 and bus 1802 may include a path for transmitting information between various components of computing device 1800 (e.g., memory 1806, processor 1804, communication interface 1808).

[0342] Processor 1804 may include any one or more processors such as a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0343] Memory 1806 may include volatile memory, such as random access memory (RAM). Memory 1806 may also include non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid state drive (SSD).

[0344] The memory 1806 stores executable code, which the processor 1804 executes to implement the functions of the aforementioned view generation device, service device, and computing device, thereby realizing the view management method. That is, the memory 1806 stores instructions for executing the view management method.

[0345] The communication interface 1808 uses transceiver modules, such as, but not limited to, network interface cards and transceivers, to enable communication between the computing device 1800 and other devices or communication networks.

[0346] This invention also provides a computing device cluster. The computing device cluster includes at least one computing device. The computing device can be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be a terminal device such as a desktop computer, a laptop computer, or a smartphone.

[0347] like Figure 19 As shown, the computing device cluster includes at least one computing device 1800. The memory 1806 of one or more computing devices 1800 in the computing device cluster may store the same instructions for executing view management methods.

[0348] In some possible implementations, the memory 1806 of one or more computing devices 1800 in the computing device cluster may also store a portion of the instructions for executing the view management method. In other words, a combination of one or more computing devices 1800 can jointly execute the instructions for executing the view management method.

[0349] It should be noted that the memory 1806 in different computing devices 1800 within the computing device cluster can store different instructions, each used to execute a portion of the functions of the view management system. That is, the instructions stored in the memory 1806 of different computing devices 1800 can implement the functions of one or more of the service device, view generation device, and computing device.

[0350] In some possible implementations, one or more computing devices in a computing device cluster can be connected via a network. This network can be a wide area network (WAN) or a local area network (LAN), etc.

[0351] Figure 20 One possible implementation is shown. For example... Figure 20 As shown, the two computing devices 1800A and 1800B are connected via a network. Specifically, they are connected to the network through communication interfaces in each computing device. In this possible implementation, the memory 1806 in computing device 1800A stores instructions for performing the functions of the service device. Simultaneously, the memory 1806 in computing device 1800B stores instructions for performing the functions of the view generation device and the computing device.

[0352] Figure 20The connection method between the computing device clusters shown can be based on the fact that the view management method provided by the present invention requires the service device to monitor the node status of storage node 102, so the service device is deployed separately on 1800A.

[0353] It should be understood that Figure 20 The functions of the computing device 1800A shown can also be performed by multiple computing devices 1800. Similarly, the functions of the computing device 1800B can also be performed by multiple computing devices 1800.

[0354] This invention also provides a computer program product containing instructions. The computer program product may be a software or program product containing instructions, capable of running on a computing device or stored on any usable medium. When the computer program product is run on at least one computing device, it causes the at least one computing device to execute a view management method.

[0355] This invention also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium that a computing device can store, or a data storage device such as a data center containing one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state drive). The computer-readable storage medium includes instructions that instruct the computing device to execute a view management method.

[0356] The above are merely specific embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

[0357] This invention also provides a computer program product containing instructions. The computer program product may be a software or program product containing instructions, capable of running on a computing device or stored on any available medium. When the computer program product is run on at least one computer device, it causes the at least one computer device to execute the view management method described above applied to a management platform.

[0358] This invention also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium that a computing device can store, or a data storage device such as a data center containing one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state drive). The computer-readable storage medium includes instructions that instruct the computing device to execute the view management method described above, applied to a management platform.

[0359] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the protection scope of the technical solutions of the embodiments of the present invention.

[0360] Those skilled in the art will clearly understand that the specific working process of the system, management platform or unit described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0361] The above are merely specific embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A distributed storage system, characterized in that, The distributed storage system includes: N storage nodes and computing nodes, where N is a positive integer, and the N storage nodes include a first view generation node; The first view generation node is used to determine M view push nodes from the N storage nodes, where M is a positive integer less than or equal to N; The first view generation node is used to send view information to the M view push nodes, and the view information is used to indicate the rules for distributing data to the N storage nodes; The computing node is used to determine a first view push node from the M view push nodes, request the view information from the first view push node, and write or read data from the N storage nodes based on the requested view information.

2. The distributed storage system according to claim 1, characterized in that, The first view generation node is used to determine the association between the second view push node and H view consumption nodes. The H view consumption nodes are storage nodes other than the first view generation node and the M view push nodes among the N storage nodes, and H is a positive integer less than N. The second view push node is any one of the M view push nodes; The first view generation node is used to send the association relationship to the second view push node; The second view push node is used to determine the view consumer node associated with the second view push node based on the association relationship, and send the view information to the associated view consumer node.

3. The distributed storage system according to claim 1 or 2, characterized in that, The distributed storage system also includes a service node, which communicates with the N storage nodes; The first view generation node is used to determine view push node information, and the view push node information is used to indicate the M view push nodes; The first view generation node is used to send the view push node information to the service node; The computing node is used to request the view push node information from the service node; The computing node is used to determine the first view push node from the M view push nodes based on the view push node information.

4. The distributed storage system according to claim 2, characterized in that, The service node is used to send the node status of the N storage nodes to the first view generation node; The first view generation node is used to determine view information based on the node status and data distribution rules of the N storage nodes.

5. The distributed storage system according to claim 3 or 4, characterized in that, The service node is configured to send the updated node status of the target storage node to the first view generation node after determining that the node status of the target storage node among the N storage nodes has been updated.

6. The distributed storage system according to any one of claims 3 to 5, characterized in that, The first view generation node is used to send the view information to the service node; The service node is used to store the view information.

7. The distributed storage system according to any one of claims 1 to 6, characterized in that, The first view generation node is used to determine that the third view push node among the M view push nodes is faulty, and select a fourth view push node from the storage nodes other than the first view generation node and the M view push nodes among the N storage nodes to replace the third view push node, so as to obtain the updated view push node information. The first view generation node is used to send the updated view push node information to the service node; The first view generation node is used to send the view information to the fourth view push node.

8. The distributed storage system according to any one of claims 1 to 7, characterized in that, The distributed storage system also includes a second view generation node; In the event of a failure of the first view generation node, the second view generation node is used to read the view information and the view push node information from the service node.

9. The distributed storage system according to any one of claims 1 to 8, characterized in that, The computing node is configured to determine that requesting the view information from the first view push node has failed, and based on the view push node information, determine a fifth view push node and request the view information from the fifth view push node.

10. The distributed storage system according to claim 9, characterized in that, The method further includes: The computing node is used to determine if the request for the view information fails, request the latest view push node information from the service node, determine the sixth view push node based on the latest view push node information, and request the latest view information from the sixth view push node.

11. A view management method, characterized in that, The method is applied to a distributed storage system, which includes N storage nodes and computing nodes, wherein the N storage nodes include a first view generation node; the method includes: The first view generation node determines M view push nodes from the N storage nodes, where M is a positive integer less than or equal to N; and sends view information to the M view push nodes, the view information being used to indicate rules for distributing data to the N storage nodes; The computing node determines a first view push node from the M view push nodes, requests the view information from the first view push node, and performs data writing or data reading on the N storage nodes based on the requested view information.

12. The method according to claim 11, characterized in that, The method further includes: The first view generation node determines the association between the second view push node and H view consumption nodes. The H view consumption nodes are storage nodes other than the first view generation node and the M view push nodes among the N storage nodes, where H is a positive integer less than N. The second view push node is any one of the M view push nodes. The first view generation node sends the association relationship to the second view push node; Based on the association, the second view push node determines the view consumer node associated with the second view push node and sends the view information to the associated view consumer node.

13. The method according to claim 11 or 12, characterized in that, The method further includes: The first view generation node determines the view push node information, and the view push node information is used to indicate the M view push nodes; The first view generation node sends the view push node information to the service node; The computing node determines the first view push node from the M view push nodes, including: The computing node requests the view push node information from the service node; The computing node determines the first view push node from the M view push nodes based on the view push node information.

14. The method according to claim 13, characterized in that, The method further includes: The service node sends the node status of the N storage nodes to the first view generation node; The first view generation node determines the view information based on the node status and data distribution rules of the N storage nodes.

15. The method according to claim 13 or 14, characterized in that, The method further includes: After determining that the node status of the target storage node among the N storage nodes has been updated, the service node sends the updated node status of the target storage node to the first view generation node.

16. The method according to any one of claims 13 to 15, characterized in that, The method further includes: The first view generation node sends the view information to the service node; The service node stores the view information.

17. The method according to any one of claims 11 to 16, characterized in that, The method further includes: The first view generation node determines that the third view push node among the M view push nodes is faulty, and selects a fourth view push node from the storage nodes other than the first view generation node and the M view push nodes among the N storage nodes to replace the third view push node, thereby obtaining updated view push node information. The first view generation node sends the updated view push node information to the service node; The first view generation node sends the view information to the fourth view push node.

18. The method according to any one of claims 11 to 17, characterized in that, The distributed storage system further includes a second view production node, and the method further includes: In the event of a failure of the first view generation node, the second view production node reads the view information and the view push node information from the service node.

19. The method according to any one of claims 11 to 18, characterized in that, The method further includes: The computing node determines that it has failed to request the view information from the first view push node, and based on the view push node information, determines the fifth view push node and requests the view information from the fifth view push node.

20. The method according to claim 19, characterized in that, The method further includes: The computing node determines that the request for the view information has failed, requests the latest view push node information from the service node, determines the sixth view push node based on the latest view push node information, and requests the latest view information from the sixth view push node.

21. A computing device cluster, characterized in that, The system includes multiple computing devices, each comprising multiple processors and multiple memories, the multiple memories storing program instructions, and the multiple processors executing the program instructions to cause the cluster of computing devices to perform the method of any one of claims 11 to 20.

22. A computer program product containing instructions, characterized in that, When the instructions are executed by a cluster of computer devices, the cluster of computer devices causes the cluster of computer devices to perform the method as described in any one of claims 11 to 20.

23. A computer-readable storage medium, characterized in that, Includes computer program instructions, which, when executed by a cluster of computing devices, perform the method as described in any one of claims 11 to 20.

Images