Distributed cluster-based memory storage synchronization method and device, terminal and medium

Abstract

The application discloses a memory storage synchronization method and device based on a distributed cluster, a terminal and a medium. The scheme provided by the application divides a plurality of nodes contained in the distributed cluster into master nodes and slave nodes, and then takes the master nodes as the center. The master nodes broadcast the data of the master nodes to each slave node in a broadcast mode, so that each node of the distributed cluster saves full data, each node in the cluster has the same user request processing capability, and the technical problem of low user request response efficiency of a traditional distributed cluster is solved.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a method, apparatus, terminal and medium for memory storage synchronization based on a distributed cluster. Background Technology

[0002] Distributed memory storage protocols are protocols that enable data synchronization and automatic renewal of frequently accessed data within the local memory of a distributed system. Application backends typically utilize local caching to improve query response speed, such as... Figure 1 As shown, a typical cluster caching method is local caching with single-node storage: typically, the local cache data in the backend application is stored within the current single node. When a user requests the corresponding service, the specific node that stores the relevant data must respond to the request. When the number of a certain type of request is too large, it can easily lead to an imbalance in node load, thereby affecting the user request response efficiency. Summary of the Invention

[0003] This application provides a memory storage synchronization method, apparatus, terminal, and medium based on a distributed cluster, which solves the technical problem of low user request response efficiency in existing local cache synchronization methods.

[0004] To address the aforementioned technical problems, the first aspect of this application provides a memory storage synchronization method based on a distributed cluster, wherein the distributed cluster includes a master node and slave nodes, and a subordinate relationship exists between the master node and the slave nodes. The method includes:

[0005] Receive the storage data reported by the slave node and preprocess the storage data;

[0006] The master node's storage data is synchronized to each slave node via broadcast, so that all nodes in the distributed cluster contain the same storage data.

[0007] Preferably, it further includes:

[0008] Receive the slave node hotspot data reported by the slave node, determine the global hotspot data that needs to be renewed according to the preset hotspot rules, and generate a renewal request based on the global hotspot data;

[0009] The renewal request is broadcast to each slave node.

[0010] Preferably, the step of receiving the slave node hotspot data reported by the slave node, determining global hotspot data according to preset hotspot rules, and generating a renewal request based on the global hotspot data specifically includes:

[0011] Receive hotspot data from slave nodes periodically reported by the slave nodes, determine global hotspot data that needs to be renewed according to preset hotspot rules, and generate a renewal request based on the global hotspot data.

[0012] Preferably, determining the global hotspot data requiring renewal based on preset hotspot rules specifically includes:

[0013] Based on the hot data of the slave nodes, the access frequency of each stored data is counted, and the global hot data is determined by combining the access frequency with a preset access frequency threshold.

[0014] Preferably, it further includes:

[0015] Iterate through the expiration dates of the stored data and clean up any stored data whose expiration dates have expired.

[0016] Preferably, it further includes:

[0017] When the distributed cluster receives a data update request, if the node receiving the data update request is the master node, the master node updates the data according to the data update request, and then synchronizes the updated data to each slave node through broadcast.

[0018] If the node receiving the data update request is a slave node, the data update request is forwarded to the master node, so that the master node updates the data according to the data update request, and then synchronizes the updated data to each slave node through broadcast.

[0019] Preferably, it further includes:

[0020] When the distributed cluster receives a non-data update request, the node that received the data update request will handle it directly.

[0021] Meanwhile, a second aspect of this application provides a memory storage synchronization device based on a distributed cluster, the distributed cluster including: a master node and slave nodes, wherein there is a subordinate relationship between the master node and the slave nodes, and the device includes:

[0022] The data aggregation unit is used to control the master node to receive the stored data reported by the slave node and to preprocess the stored data.

[0023] The data synchronization unit is used to control the master node to synchronize the local storage data to each of the slave nodes through broadcasting, so that all nodes of the distributed cluster contain the same storage data.

[0024] A third aspect of this application provides a memory storage synchronization terminal based on a distributed cluster, comprising: a memory and a processor;

[0025] The memory is used to store program code, which corresponds to the memory storage synchronization method based on a distributed cluster provided in the first aspect of this application;

[0026] The processor is used to execute the program code.

[0027] The fourth aspect of this application provides a computer-readable storage medium storing program code, which corresponds to the memory storage synchronization method based on a distributed cluster provided in the first aspect of this application.

[0028] As can be seen from the above technical solutions, the embodiments of this application have the following advantages:

[0029] The solution provided in this application divides the multiple nodes in the distributed cluster into master nodes and slave nodes. Then, with the master node as the center, the master node synchronizes the data of the master node to each of the slave nodes through broadcasting. This ensures that each node in the distributed cluster stores the full amount of data and that each node in the cluster has the same user request processing capability, thus solving the technical problem of low user request response efficiency in traditional distributed clusters. Attached Figure Description

[0030] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0031] Figure 1 A schematic diagram illustrating the system principle of in-memory storage for an existing distributed cluster.

[0032] Figure 2 This is a schematic diagram illustrating the system principle of a memory storage synchronization method based on a distributed cluster, as provided in this application.

[0033] Figure 3 This is a flowchart illustrating a memory storage synchronization method based on a distributed cluster, as provided in this application.

[0034] Figure 4 This is a flowchart illustrating the hot data reporting process in a distributed cluster-based memory storage synchronization method provided in this application.

[0035] Figure 5 This is a flowchart illustrating the hot data renewal process in a distributed cluster-based memory storage synchronization method provided in this application.

[0036] Figure 6 This is a flowchart illustrating a data update business process in a memory storage synchronization method based on a distributed cluster provided in this application.

[0037] Figure 7 This is a flowchart illustrating another data update business process in a distributed cluster-based memory storage synchronization method provided in this application.

[0038] Figure 8 This is a flowchart illustrating the routine business processing in a distributed cluster-based memory storage synchronization method provided in this application.

[0039] Figure 9 This application provides a schematic diagram of the structure of a memory storage synchronization device based on a distributed cluster. Detailed Implementation

[0040] This application provides a method, apparatus, terminal, and medium for memory storage synchronization based on a distributed cluster, which addresses the technical problem of low user request response efficiency in existing local cache synchronization methods.

[0041] To make the inventive objectives, features, and advantages of this application more apparent and understandable, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the embodiments described below are only some embodiments of this application, and not all embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0042] Please see Figure 2 and Figure 3 This application provides a memory storage synchronization method based on a distributed cluster. The distributed cluster includes a master node and slave nodes, wherein there is a subordinate relationship between the master node and the slave nodes. The method includes:

[0043] Step 101: Receive the storage data reported by the node and preprocess the storage data;

[0044] Step 102: Synchronize the local storage data to each slave node via broadcast, so that all nodes in the distributed cluster contain the same storage data.

[0045] The solution provided in this embodiment divides the multiple nodes in the distributed cluster into master nodes and slave nodes. Then, with the master node as the center, data from the slave nodes is aggregated to the master node, ensuring that the master node stores all the data of the entire distributed cluster. The master node then broadcasts its local data to all slave nodes, ensuring that each node in the distributed cluster maintains a complete copy of the data. This guarantees that all nodes in the cluster have the same user request processing capabilities, resolving the data inconsistency problem caused by single-node local caching. When faced with a large number of user requests, any node in the cluster can handle them, balancing the load across the cluster nodes and improving user request response efficiency.

[0046] The above content is a detailed description of a basic embodiment of a memory storage synchronization method based on a distributed cluster provided in this application. The following is a further description of an embodiment of a memory storage synchronization method based on a distributed cluster provided in this application.

[0047] Furthermore, it also includes:

[0048] Step 103: Receive the hotspot data reported by the slave nodes, determine the global hotspot data that needs to be renewed according to the preset hotspot rules, and generate a renewal request based on the global hotspot data.

[0049] Step 104: Broadcast the renewal request to each slave node.

[0050] It should be noted that in distributed memory systems, all data stored in memory should theoretically have a time limit (e.g., cached for 7 days, cached for 1 month, etc.). This is because memory is often precious and limited compared to persistent disk storage, and most data should not be stored permanently. However, some data is accessed infrequently, while other data is accessed frequently. Frequently accessed data is considered "hot data" and its expiration time should be extended to ensure it remains in memory for as long as possible. Therefore, in the solution of this embodiment, such as... Figure 4 and Figure 5 As shown, each slave node periodically reports its data access records to the master node. The master node's hotspot processor performs statistical analysis on all collected data access frequencies to determine global hotspot data. Based on the configuration rules and the data's lifespan, it calculates the global hotspot data that needs to be renewed and sends renewal requests to all nodes, thus completing the detection and automatic renewal of hotspot data.

[0051] Furthermore, step 103 specifically includes:

[0052] Receive hotspot data from slave nodes periodically reported by slave nodes, determine global hotspot data that needs to be renewed according to preset hotspot rules, and generate renewal requests based on global hotspot data.

[0053] Furthermore, the global hotspot data that needs to be renewed, determined according to the preset hotspot rules, specifically includes:

[0054] Based on the hot data of the slave nodes, the access frequency of each storage data is counted. Based on the access frequency and the preset access frequency threshold, the global hot data is determined.

[0055] Furthermore, it also includes:

[0056] Step 105: Traverse the expiration dates of the stored data and clean up the stored data whose expiration dates have expired.

[0057] It should be noted that when hot data in memory expires, a large number of front-end requests will usually be sent directly to the database, which can easily cause cache breakdown and put a lot of pressure on the database. Therefore, the automatic detection of hot data and automatic extension of the expiration time of hot data implemented in this embodiment can effectively avoid cache breakdown. At the same time, expired stored data, such as non-hot data that has not been renewed, can be cleaned up to release the memory it occupies.

[0058] Furthermore, it also includes:

[0059] When the distributed cluster receives a data update request, if the node receiving the data update request is the master node, the master node will update the data according to the data update request, and then synchronize the updated data to each slave node through broadcast.

[0060] If the node receiving the data update request is a slave node, it forwards the data update request to the master node so that the master node can update the data according to the data update request, and then synchronize the updated data to each slave node through broadcast.

[0061] When the distributed cluster receives a non-data update request, it is handled directly by the node that received the data update request.

[0062] It should be noted that, based on the distributed cluster memory storage synchronization method provided in this embodiment, the specific business processing example is as follows:

[0063] The first step is handling data update requests: such as... Figure 6 As shown, if the master node receives a data update request (such as a data modification request like adding, modifying, or deleting), the master node will complete the transaction processing in its local memory and then broadcast the transaction request to all slave nodes to complete data synchronization. Figure 7As shown, if a transaction is received by a slave node, the slave node will forward the request to the master node. The master node will complete the transaction processing in its local memory and broadcast the transaction request to all slave nodes to complete data synchronization.

[0064] Secondly, there's the handling of routine business: such as... Figure 8 As shown, when any node receives a non-transactional request (such as a query or other business that does not involve data modification), it will directly return the data to the requester (because each node stores the full amount of data).

[0065] Taking the signaling system for video network home security as an example, the backend service, by introducing this memory protocol, caches frequently used device configuration information locally, achieving distributed storage across multiple nodes. When a user watches a live stream, they request the signaling system backend. The signaling system first queries the local cache for configuration information; if the information exists locally, there's no need to query Redis, MySQL, or other databases. Because this memory protocol has hotspot detection and renewal functions, the configuration data of active devices can be detected and ensured to be continuously stored in the local cache of all backend service nodes. Compared to before the introduction of this protocol, which required querying Redis or MySQL every time, the speed of obtaining device data is greatly improved (from milliseconds to nanoseconds), reducing user lag and enhancing the live streaming viewing experience.

[0066] The solution provided in this application enables all nodes in the cluster to have the same user request processing capability, which can effectively solve the technical problem of low user request response efficiency in traditional distributed clusters.

[0067] The above content describes a memory storage synchronization method based on a distributed cluster provided in this application. The following is a detailed description of embodiments of a memory storage synchronization device, terminal, and computer-readable storage medium based on a distributed cluster provided in this application.

[0068] Please see Figure 9 The second embodiment of this application provides a memory storage synchronization device based on a distributed cluster. The distributed cluster includes a master node and slave nodes, wherein there is a subordinate relationship between the master node and the slave nodes. The device includes:

[0069] The data aggregation unit 201 is used to control the master node to receive the stored data reported by the slave nodes and to preprocess the stored data.

[0070] The data synchronization unit 202 is used to control the master node to synchronize local storage data to each slave node through broadcasting, so that all nodes in the distributed cluster contain the same storage data.

[0071] The third embodiment of this application provides a memory storage synchronization terminal based on a distributed cluster, including: a memory and a processor;

[0072] The memory is used to store program code, which corresponds to the memory storage synchronization method based on a distributed cluster provided in the first embodiment of this application;

[0073] The processor is used to execute program code.

[0074] The fourth embodiment of this application provides a computer-readable storage medium storing program code, which corresponds to the memory storage synchronization method based on a distributed cluster provided in the first embodiment of this application.

[0075] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the terminals, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0076] In the several embodiments provided in this application, it should be understood that the disclosed terminals, devices, and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection between devices or units through some interfaces, and may be electrical, mechanical, or other forms.

[0077] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the application described herein can be implemented, for example, in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0078] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0079] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0080] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0081] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A method for in-memory storage synchronization based on a distributed cluster, the distributed cluster comprising: A master node and a slave node, wherein a subordinate relationship exists between the master node and the slave node, characterized in that the method includes: Receive the storage data reported by the slave node and preprocess the storage data; The master node's storage data is synchronized to each slave node via broadcast, so that all nodes in the distributed cluster contain the same storage data. Receive the slave node hotspot data reported by the slave node, determine the global hotspot data that needs to be renewed according to the preset hotspot rules, and generate a renewal request based on the global hotspot data; Broadcast the renewal request to each slave node; Iterate through the expiration dates of the stored data and clean up any stored data whose expiration dates have expired.

2. The method of claim 1, wherein, The process of receiving the slave node hotspot data reported by the slave node, determining global hotspot data according to preset hotspot rules, and generating a renewal request based on the global hotspot data specifically includes: Receive hotspot data from slave nodes periodically reported by the slave nodes, determine global hotspot data that needs to be renewed according to preset hotspot rules, and generate a renewal request based on the global hotspot data.

3. A memory storage synchronization method based on a distributed cluster according to claim 1 or 2, characterized in that, The process of determining the global hotspot data that needs to be renewed according to preset hotspot rules specifically includes: Based on the hot data of the slave nodes, the access frequency of each stored data is counted, and the global hot data is determined by combining the access frequency with a preset access frequency threshold.

4. The memory storage synchronization method based on a distributed cluster according to claim 1, characterized in that, Also includes: When the distributed cluster receives a data update request, if the node receiving the data update request is the master node, the master node updates the data according to the data update request, and then synchronizes the updated data to each slave node through broadcast. If the node receiving the data update request is a slave node, the data update request is forwarded to the master node, so that the master node updates the data according to the data update request, and then synchronizes the updated data to each slave node through broadcast.

5. The memory storage synchronization method based on a distributed cluster according to claim 1, characterized in that, Also includes: When the distributed cluster receives a non-data update request, the node that received the data update request will handle it directly.

6. A memory storage synchronization device based on a distributed cluster, the distributed cluster comprising: A master node and a slave node, wherein a subordinate relationship exists between the master node and the slave node, characterized in that the device comprises: The data aggregation unit is used to receive the stored data reported by the slave node and to preprocess the stored data. The data synchronization unit is used to synchronize the storage data of the master node to each of the slave nodes by broadcasting, so that all nodes of the distributed cluster contain the same storage data. Receive the slave node hotspot data reported by the slave node, determine the global hotspot data that needs to be renewed according to the preset hotspot rules, and generate a renewal request based on the global hotspot data; Broadcast the renewal request to each slave node; Iterate through the expiration dates of the stored data and clean up any stored data whose expiration dates have expired.

7. A memory storage synchronization terminal based on a distributed cluster, characterized in that, include: Memory and processor; The memory is used to store program code, which corresponds to the memory storage synchronization method based on a distributed cluster as described in any one of claims 1 to 5; The processor is used to execute the program code.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium contains program code corresponding to the memory storage synchronization method based on a distributed cluster as described in any one of claims 1 to 5.

Images