Distributed storage server, distributed storage method and storage medium

Abstract

The present invention discloses a distributed storage server, a distributed storage method and a storage medium. The distributed storage server includes a data distribution processor, a main memory and at least one node memory, wherein the main memory and the node memory are respectively connected to the data distribution processor; the main memory is used to store resource data; the node memory is used to classify and store the resource data; the data distribution processor is used to distribute the resource data stored in the main memory to the target node memory according to the data category corresponding to the target node memory when the target node memory transmits a resource data application request in the node memory. The probability of resource data loss when the distributed storage server fails is reduced.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a distributed storage server, a distributed storage method and a storage medium. Background Art

[0002] Distributed storage is a data storage technology that achieves high reliability, scalability and high performance by storing data on multiple computers or servers. In a distributed storage system, in order to ensure high data reliability, data written into the distributed storage system is usually stored on multiple independent storage servers. These servers that store the same data are called server groups.

[0003] However, running a load balancing algorithm on an independent server is prone to single point failure problems. When the server running the load balancing algorithm fails, the data stored in the independent server is prone to loss. Summary of the invention

[0004] The present invention provides a distributed storage server, a distributed storage method and a storage medium to solve the problem of data loss caused by server failure.

[0005] According to one aspect of the present invention, there is provided a distributed storage server, comprising:

[0006] A data distribution processor, a main memory and at least one node memory, wherein the main memory and the node memory are respectively connected to the data distribution processor;

[0007] The main memory is used to store resource data;

[0008] The node storage is used to classify and store the resource data;

[0009] The data allocation processor is used for allocating the resource data stored in the main memory to the target node memory according to the data category corresponding to the target node memory when a target node memory in the node memory requests the target node memory to transmit resource data.

[0010] According to another aspect of the present invention, a distributed storage method is provided, characterized in that it is executed by a data distribution processor in the distributed storage server provided by the present invention, and the distributed storage method includes:

[0011] In the case of a target node memory transmission resource data application request in the node memory, the resource data stored in the main memory is allocated to the target node memory according to the data category corresponding to the target node memory.

[0012] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the distributed storage method described in any embodiment of the present invention when executed.

[0013] The technical solution of the embodiment of the present invention provides a distributed storage server, which includes a main memory and at least one node memory. The distributed storage server allocates resource data in the main memory to the node memory for storage through a data allocation processor, and at least one node memory stores resource data in the main memory according to data categories. Resource data in the distributed storage server is repeatedly stored in the main memory and the node memory, thereby reducing the probability of resource data loss when the distributed storage server fails.

[0014] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0016] Figure 1 is a structural diagram of a distributed storage server provided according to Embodiment 1 of the present invention;

[0017] Figure 2 This is a schematic diagram of a process of allocating resource data to a distributed storage server provided in accordance with Embodiment 2 of the present invention;

[0018] Figure 3 is a schematic diagram of a process of performing connection detection by a distributed storage server provided in Embodiment 3 of the present invention;

[0019] Figure 4 It is a schematic diagram of a process of recording allocation results of a distributed storage server provided according to Embodiment 4 of the present invention;

[0020] Figure 5 is a flow chart of a distributed storage method provided according to Embodiment 5 of the present invention;

[0021] Figure 6 It is a schematic diagram of the structure of a distributed storage device provided according to Embodiment 6 of the present invention. DETAILED DESCRIPTION

[0022] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0023] It should be noted that the terms "including" and "having" and any variations thereof in the present invention are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices. The components shown in the present invention, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required in the present invention.

[0024] With the rapid development of the Internet, a large number of needs involving massive data analysis, calculation, and statistics need to be solved by people, so the concepts of distributed computing and distributed storage were proposed.

[0025] Distributed analysis, calculation and statistics can save a lot of cost and computing time by using multiple relatively low-performance machines. The data required for distributed computing can be stored and retrieved from distributed storage. Distributed storage achieves high data reliability and scalability by distributing data on multiple servers. It provides a data foundation for distributed computing, ensuring that computing tasks can quickly and accurately obtain the required data for processing.

[0026] There are three indicators to measure the performance of storage systems: reliability, storage speed and availability. In order to ensure high data reliability, distributed storage systems usually store data written into the storage system on multiple independent storage servers. These servers that store the same data are called server groups.

[0027] In order to balance the workload of each server. In a distributed computing and distributed storage environment, when there are multiple servers involved, load balancing can ensure that computing tasks and data access requests can be reasonably distributed to different servers, avoiding the situation where some servers are overloaded while other servers are idle, thereby further optimizing system performance and improving resource utilization. It makes the operation of distributed computing and distributed storage more efficient, stable and reliable.

[0028] However, the existing load balancing algorithm running on an independent server is prone to single point failure problems. When the server running the load balancing algorithm fails, the function of the entire distributed database cluster will be affected and the load balancing function cannot be completed. In addition, the metadata required for load balancing is stored in the independent server, which is also prone to metadata loss problems.

[0029] Therefore, the present invention provides a new distributed storage server to solve the above technical problems.

[0030] Embodiment 1

[0031] Figure 1 1 is a schematic diagram of the structure of a distributed storage server provided according to Embodiment 1 of the present invention. This embodiment is suitable for reducing the probability of data loss in the distributed storage server. The distributed storage server is a server that performs distributed storage of stored resource data. Resource data may be data stored in a server that needs to be distributedly stored. Resource data may be data in a distributed storage server whose reliability requirement is greater than a set threshold. The reliability of resource data is improved by distributing the resource data in the distributed storage server.

[0032] Resource data can be data used for distributed computing, such as task-related parameters; it can also be business data, such as business-related data stored by the client to the server; it can also be data related to load balancing, such as metadata required for load balancing. Resource data can be data that supports distributed computing, distributed storage, and / or load balancing.

[0033] like Figure 1 As shown, the distributed storage server provided in this embodiment includes:

[0034] A data allocation processor 110, a main memory 120 and at least one node memory 130, wherein the main memory 120 and the node memory 130 are respectively connected to the data allocation processor 110; the main memory 120 is used to store resource data; the node memory 130 is used to classify and store the resource data; the data allocation processor 110 is used to allocate the resource data stored in the main memory 120 to the target node memory 130 according to the data category corresponding to the target node memory 130 when the target node memory 130 in the node memory 130 transmits a resource data application request.

[0035] The data allocation processor 110 can be considered as a processor for resource data allocation in the distributed storage server. In the present invention, the data allocation processor 110 is used to allocate resource data stored in the main memory 120 to the node memory 130 for storage. The data allocation processor 110 can also achieve load balancing, avoid overloading a certain node memory 130, and improve the performance of the distributed storage server.

[0036] The data distribution processor 110 can be various general and / or special processing components with processing and computing capabilities. Some examples of the data distribution processor 110 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc.

[0037] The main memory 120 may be a computer hardware storage unit for storing program instructions and data in a distributed storage server, such as a memory. The node memory 130 may be a node for auxiliary storage of resource data in a distributed storage server, and a memory for distributed storage of resource data in the main memory 120. In the present invention, the number of node memories 130 is at least one. Figure 1 Taking one node storage 130 as an example, the number is not limited here, and it can be two, three or more.

[0038] The main memory 120 and the distributed memory implement distributed storage of resource data. The main memory 120 can store all resource data stored in the distributed storage server, and the resource data can be shared data. Multiple node memories 130 can store resource data according to data categories, and one node memory 130 can correspond to resource data of at least one data category. The categories of resource data stored in each node memory 130 can be different.

[0039] The main memory 120 and the node memory 130 in the present invention can be a read-only memory (ROM) or a random access memory (RAM). The main memory 120 and / or the node memory 130 store a computer program that can be executed by the data distribution processor 110. The computer program is executed by the data distribution processor 110 so that the data distribution processor 110 can execute the method provided by the present invention, such as implementing distributed storage, performing I / O operations with the client, and detecting the connection status of the distributed storage server, and the connection status represents the connection status between the distributed server and the client. Various programs and data required for the operation of the distributed storage server can also be stored in the main memory 120 and the node memory 130.

[0040] The resource data is divided according to certain standards or attributes by the data allocation processor 110 of the distributed storage server, and then stored in the corresponding node storage 130, thereby realizing the storage of the resource data stored in the main memory 120 of the distributed storage server in the corresponding node storage 130 by category.

[0041] The classification method is not limited here. For example, data can be classified according to factors such as data type, usage frequency, and business importance. Classified storage of resource data not only improves the efficiency and response speed of data processing, but also helps reduce storage costs and improve the flexibility of data management.

[0042] In this embodiment, the node storage 130 may transmit a resource data application request to the data allocation processor 110. The resource data application request may be considered as a request for applying for resource data in the main storage 120. The target node storage 130 may be considered as the node storage 130 that sends the resource data application request. After the data allocation processor 110 obtains the resource data application request transmitted by the target node storage 130, the resource data may be allocated to the target node storage 130.

[0043] When allocating resource data to the target node memory 130 , the data allocation processor 110 may allocate the resource data stored in the main memory 120 to the target node memory 130 according to the data type.

[0044] Data categories can be classified into different categories or types according to certain standards or characteristics. The data categories corresponding to different node storages 130 can be determined by the data allocation processor 110; or when the node storage 130 requests resource data from the data allocation processor 110, it carries information indicating the corresponding data category, that is, it is determined by the characteristics of the node storage 130. For example, it is determined by the hardware characteristics of the node storage 130; or it can be the data categories that different node storages 130 can store, which are preset.

[0045] The data distribution processor 110, the main memory 120 and the node memory 130 are connected in any manner as long as they can be connected for communication.

[0046] In one embodiment, the data distribution processor 110, the main memory 120 and the node memory 130 are connected via a set communication protocol.

[0047] In this embodiment, the connection between the data distribution processor 110, the main memory 120 and the node memory 130 can be established according to the set communication protocol, and communication can be achieved. The communication protocol is a rule and language that specifies the format, sequence, control signal and other details of information exchange between the data distribution processor 110, the main memory 120 and the node memory 130. By following the communication protocol, the data distribution processor 110 can perform effective data transmission, instruction delivery and status interaction operations with the main memory 120 and the node memory 130.

[0048] The communication protocol can include TCP / IP protocol or a custom application layer protocol. The application layer protocol includes: header + length + address + reserved word + sequence number + command word + data, where the data is encoded in Base64.

[0049] The header is the beginning of a data packet, used to identify the beginning of a data packet. It usually contains special characters or byte sequences used to synchronize and identify data packets, such as information about the message type, version, or format. It allows the receiver to quickly identify and manage the type of data to be processed.

[0050] Length: Specifies the total length of the data packet. The receiver can use the length field to determine the integrity of the data and parse other fields in the data packet accordingly.

[0051] Address: Contains the address information of the destination, which is used to correctly route data packets to the destination in the network. The address can also include the source address, which is used to determine the sender of the data.

[0052] Reserved words: Reserved words are used for possible future protocol extensions or special purposes to increase the flexibility and extensibility of the protocol.

[0053] Sequence number: The sequence number is used to identify the uniqueness of the message and to achieve data sorting and reorganization. The receiver can restore the correct order of the data based on the sequence number, especially when the data packets may arrive out of order.

[0054] Command word: The command word indicates the function or operation type of the data packet, such as request, response, confirmation, etc., and is used to tell the receiver how to process the data packet.

[0055] Data: Data is the actual information content transmitted.

[0056] The distributed storage server provided in this embodiment includes a main memory and at least one node memory. The distributed storage server allocates resource data in the main memory to the node memory for storage through a data allocation processor, and at least one node memory stores resource data in the main memory according to data categories. Resource data in the distributed storage server is repeatedly stored in the main memory and the node memory, thereby reducing the probability of resource data loss when the distributed storage server fails.

[0057] Embodiment 2

[0058] Figure 2 This is a flow chart of resource data allocation by a distributed storage server provided according to the second embodiment of the present invention. This embodiment refines the operation of the data allocation processor allocating resource data stored in the main memory to the target node memory according to the data category corresponding to the target node memory when the target node memory transmits a resource data application request in the node memory. The node memory in the distributed storage server needs to apply for resource data to the data allocation processor, and the data allocation processor allocates resource data from the main memory to the node memory, forming a node on the storage resource.

[0059] See also Figure 2 , the data allocation processor allocates resource data including the following steps:

[0060] S210: When a target node memory in the node memory transmits a resource data application request, an estimated value of resource data allocation for each target node memory is determined based on the data capacity of the resource data in the main memory.

[0061] The data capacity of resource data can be considered as the size of the storage space occupied by resource data. Data capacity can represent the amount of data that a main memory can hold, including the total amount of various types of data such as text, images, audio, video, etc.

[0062] The resource data allocation estimate value can be considered as the estimated information about the resource data allocated to the target node storage, such as the size of the resource data, the category of the resource data, etc. The target node storage for determining the resource data allocation estimate value can be the node storage that has sent the resource data application request and has not yet been allocated resource data.

[0063] After the data allocation processor obtains the resource data application request transmitted by the target node memory, the data allocation processor needs to allocate resource data to the target node memory. When allocating resource data, the resource data allocation estimate value of the target node memory currently requesting resource data from the data allocation processor can be determined based on the data capacity of the resource data in the main memory.

[0064] The operation of determining the estimated value of resource data of each target node memory based on the data capacity of the resource data of the main memory can be implemented through a probability model. For example, when the node memory applies for resource data allocation to the data allocation module, the resource data of the node memory is set as a variable, the resource data of the main memory is set as a potential variable, and the unknown resource data is used as the allocation resource call to establish a probability model for resource data call.

[0065] The main memory receives resource data in real time. After the data allocation processor obtains the resource data application request of the target node memory, the content capacity parameter of the resource data is input into the probability model, and the amount of resource data that can be allocated to each target node memory is output, that is, the resource data allocation estimate. The weight value is sorted according to the amount of resource data that can be allocated. The target node memory in the first place is allocated resources first, and the allocation is re-sorted each time the resource data is received. In the process of resource data allocation, the data allocation processor sorts according to the weight value of each target node memory, and selects the appropriate target node memory for load balancing according to the current load status of the target node memory (such as the resource data allocation value), such as allocating resource data.

[0066] The probability model can combine the result of resource data classification in the process of resource data allocation estimation to determine the resource data allocation estimation value of the target node memory. The probability model can combine the classification result of resource data through classification parameters to realize the allocation of resource data to the target node memory, while ensuring that the node memory stores resource data in a classified manner.

[0067] The resource data stream in the main memory is centrally allocated to the node memory through the data allocation processor, and the resource data allocation estimate of the node memory is obtained through the classification result, so as to allocate the resource data based on the resource data allocation estimate. For example, if the allocation estimate meets the requested resource data allocation value, a classification call is performed. Among them, the resource data allocation value can represent the relevant information of the resource data requested by the corresponding target node memory to the data allocation processor. Such as the size of the resource data, the category of the resource data. When the estimated resource data estimate meets the requested resource data allocation value, the resource data can be allocated to the corresponding target node memory.

[0068] The probabilistic model takes into account the resource allocation between the main memory and the node memory, optimizes the data access and processing speed, and realizes the dynamic allocation of resource data between different nodes to meet the current data processing needs.

[0069] In the present invention, the node memory is used to apply for resource data allocation to the data allocation processor, and the estimated value of real-time resource data allocated to the node memory is combined with the resource data allocation value applied for according to the established resource data call probability model. If the resource data allocation estimated value meets the resource data allocation value applied for, a classified call is performed, which can realize the classification of unknown resource data, shorten the classification time of unknown resource data, and disperse forward according to the data type, effectively improving the dispersion degree of real-time data of the main memory. The present invention sets the resource data of the node memory as a variable, sets the resource data of the main memory as a potential variable, uses the unknown resource data as the allocation resource call, estimates the estimated value of real-time resource data allocated to the node memory and matches it, and further improves the dispersion degree of resource data.

[0070] S220: Determine the sorting result of each target node storage according to the corresponding resource data allocation estimation value.

[0071] After determining the resource data allocation estimation value of each target node memory, the target node memories may be sorted according to the size of the resource data allocation estimation value to obtain the target node memories sorted according to the resource data allocation estimation value.

[0072] S230. Allocate resource data of a corresponding data category to each target node storage according to the sorting result.

[0073] After obtaining the sorting result, the data allocation processor can allocate resource data of corresponding data categories to the target node memory in sequence according to the order of the target node memory in the sorting result, so as to realize the orderly allocation of target node memory resource data when there are multiple target node memories.

[0074] In this embodiment, the data allocation processor considers the resource data allocation estimation value of each target node memory in the process of allocating resource data to the target node memory, sorts the multiple target node memories based on the resource data allocation estimation value, and allocates resource data to the target node memories in an orderly manner, thereby realizing rapid allocation of resource data.

[0075] Based on the above embodiment, a variant embodiment of the above embodiment is proposed. It should be noted that in order to make the description concise, only the differences from the above embodiment are described in the variant embodiment.

[0076] In one embodiment, the data distribution processor is specifically used to:

[0077] For each target node memory, if the resource data allocation estimate value of the target node memory matches the resource data allocation value requested by the target node memory, the resource data of the data category corresponding to the target node memory in the main memory is called to the target node memory;

[0078] The resource data application request is a dynamic proxy action.

[0079] The resource data allocation estimate of the target node memory matches the resource data allocation value applied for by the target node memory, which can be considered as the resource data allocation estimate of the target node memory satisfying the resource data allocation value applied for by the target node memory. For example, if the content capacity of the resource data applied for by the target node memory is a, and the resource data allocation estimate estimated by the data allocation processor for the target node memory is b, and b is less than or equal to a, then the resource data allocation estimate of the target node memory can be considered as matching the resource data allocation value applied for by the target node memory.

[0080] The data allocation processor can call the resource data in the main memory, store the called resource data in the corresponding target node memory, and realize the transfer of the called resource data. The called resource data is not only stored in the main memory, but also in the node memory.

[0081] The present embodiment can utilize the Java Development Kit (Java Development Kit, JDK) reflection object mode to carry out dynamic proxy, utilize reentrant lock to lock the state when proxy action occurs, unlock after receiving the result feedback of calling main memory or node storage module data, utilize callback function to carry out calling process. Proxy action can be resource data application request. Callback function can be considered as a function that is passed to another function as a parameter, and is called and executed after specific event occurs or a certain operation is completed. Reentrant lock is a kind of synchronization mechanism, and it allows the thread that already has lock to obtain this lock again, and can not cause deadlock.

[0082] In one embodiment, the resource data allocation value of the application includes the content capacity and processing capability of the target node memory.

[0083] The content capacity of the target node memory can characterize the amount of data that the target node memory can store. Processing power can characterize the ability of the target node memory to process data. Such as the performance and efficiency of executing a certain processing task. Processing power can cover multiple aspects, such as computing power, that is, the speed and efficiency of mathematical operations, logical judgments, etc.; data processing capabilities, including the speed of reading, writing, analyzing, converting, and other operations on large amounts of data; concurrent processing capabilities, that is, the ability to process multiple tasks or requests at the same time; and professional processing capabilities for specific types of tasks (such as image processing, audio processing, etc.). Processing power is usually measured by some indicators, such as the number of transactions processed per second, data throughput, response time, etc. It reflects the amount of work that a system can complete in a unit of time and the complexity of processing.

[0084] Embodiment 3

[0085] Figure 3 1 is a flow chart of a distributed storage server performing connection detection according to Embodiment 3 of the present invention. In this embodiment, the data distribution processor in the above embodiment performs distributed storage server connection detection to prevent waste of distributed storage server resources. Figure 3 As shown, the data distribution processor performs the following steps:

[0086] S310. Store the connection information of the client connected to the distributed storage server in a security thread.

[0087] Connection information may refer to the relevant data used to describe the connection between the client and the distributed storage server. For example, the client ID (information that uniquely identifies the client), the network address (such as the client's IP address or domain name, etc., used to determine the client's location in the network), the connection status (which can indicate whether the current connection is connected, disconnected, or in other states), etc. The connection information can be used to effectively manage and monitor the connection between the client and the distributed storage server, ensuring the reliability and security of data transmission and accurate grasp of the connection status.

[0088] A safe thread can be considered as a thread that can ensure data security in a multi-threaded environment. This thread can ensure the integrity and consistency of data.

[0089] This operation can initiate a connection between the distributed storage server and the client through the Netty network architecture and save the successful connection in a secure thread.

[0090] The Netty startup process is as follows:

[0091] Step 1: Create an object instance. The underlying encapsulation uses a facade design to set the startup parameters.

[0092] Step 2: Set up two thread pools, bossGroup and workerGroup, where bossGroup is used to process connection events, i.e., events of connection between distributed storage server and client; workerGroup is used to process network I / O operations; when the client sends a connection request, the data distribution processor will select the network address of the appropriate node storage to the client, and the client will connect to the specific node storage node according to the selected network address, send a specific command request, and implement the I / O operation;

[0093] Step 3: Bind the listening port and start the server;

[0094] Step 4: Select a thread to monitor the server port and monitor the reading time. After reading, process it according to the registration method at startup;

[0095] Step 5: After the decoding is completed, the obtained object will be handed over to RpcHandler for processing. According to the request interface name, the corresponding instantiation object is found, and the final result is obtained by sending the call object and sent to the client. RpcHandler is a component used to process the request message received by the server in the remote procedure call (RPC).

[0096] S320. Check the connection corresponding to the connection information stored in the security thread through the timing thread, and close the connection that meets the closing condition.

[0097] A timed thread can be considered as a thread that executes tasks at a specific time interval or at a specific time point. This operation can be triggered by a timed thread to check the connection information stored in the security thread and the corresponding connection. The content of the check is not limited, such as checking whether the connection is a connection in a long-term restricted state. The definition of long-term is not limited and can be greater than the set duration. A connection in a long-term restricted state usually refers to a connection in a network communication where normal data transmission is restricted or blocked for various reasons, resulting in the connection being unable to be closed or restored to normal for a long time.

[0098] The closing condition may be a condition for closing the connection, such as a condition for closing the connection between the distributed storage server and the client. The content of the check corresponds to the closing condition. When the content of the check is whether the connection is in a long-term restricted state, the closing condition is that the connection is in a long-term restricted state.

[0099] This operation uses a timing thread to periodically close connections that have been in a restricted state for a long time to prevent long-term resource occupation.

[0100] Embodiment 4

[0101] Figure 4 is a flow chart of a distributed storage server recording allocation results according to Embodiment 4 of the present invention, see Figure 4 , the data distribution processor of the distributed storage server performs the following steps:

[0102] S410: Create a data table on the main memory.

[0103] A data table may be a table associated with a resource data call.

[0104] S420. After allocating the resource data stored in the main memory to the target node memory, record the allocation result in the data table, wherein the allocation result includes log information of the operation of the target node memory.

[0105] The allocation result may indicate the result of the resource data allocation. After the classification call is made, the allocation result is returned to the main memory, and before the allocation result is returned to the main memory, a data table corresponding to the called resource data is established on the main memory, and a log record is automatically generated and inserted into the data table after the classification call is completed. The log record is used to record the log information of the target node memory operation, and analyze the operation status according to the recorded log information.

[0106] This embodiment records the log information of resource data application implemented by the target node memory through the data table in the main memory, which is helpful for analyzing the operation status of the target node memory.

[0107] In one embodiment, the data distribution processor is further configured to:

[0108] In the case of failure in obtaining the allocation result, resource data is allocated to the target node memory according to the data address in the data log of the target node memory.

[0109] A data log is a document or file that records information related to data. A data address may be an address representing a storage location of resource data.

[0110] If the main memory cannot receive the log record of the target node memory, the resource data is written into the corresponding position in the node memory storage according to the data address in the data log of the target node memory.

[0111] By obtaining the records of data addresses in the data log, it is possible to identify the specific storage area where resource data should be placed, ensuring the accuracy and pertinence of resource allocation and avoiding confusion or incorrect allocation.

[0112] Embodiment 5

[0113] Figure 5 FIG. 5 is a flow chart of a distributed storage method provided according to Embodiment 5 of the present invention. Figure 5 As shown, the distributed storage method can be executed by a data distribution processor in a distributed storage server.

[0114] The data distribution processor executes various methods and processes described in the present invention, such as the distributed storage method.

[0115] In some embodiments, the distributed storage method can be implemented as a computer program, which is tangibly included in a computer-readable storage medium. In some embodiments, part or all of the computer program can be loaded and / or installed to a distributed storage server via a main memory and / or a node memory. When the computer program is loaded into the main memory and / or the node memory and executed by the data distribution processor, one or more steps of the distributed storage method can be executed. Alternatively, in other embodiments, the data distribution processor can be configured to perform the distributed storage method by any other appropriate means (e.g., by means of firmware).

[0116] The computer programs for implementing the method of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to the data distribution processor so that when the computer programs are executed by the data distribution processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer programs may be executed entirely on the distributed storage servers or partially on the distributed storage servers.

[0117] The distributed storage method can be applied to the case where resource data in a distributed storage server is distributed from a main memory to a node memory. The distributed storage method can be executed by a distributed storage device, which can be integrated in a data distribution processor of the distributed storage server.

[0118] The distributed storage method in this embodiment includes the following steps:

[0119] S510: Obtain a resource data application request transmitted by a target node storage.

[0120] S520. Allocate resource data stored in the main memory to the target node memory according to the data category corresponding to the target node memory.

[0121] In this embodiment, when a target node memory in the node memory transmits a resource data application request, resource data in the main memory is allocated to the target node memory, thereby realizing distributed storage of resource data in the main memory in the distributed storage server.

[0122] Based on the above embodiment, a variant embodiment of the above embodiment is proposed. It should be noted that in order to make the description concise, only the differences from the above embodiment are described in the variant embodiment.

[0123] In one embodiment, in the case of a target node memory in the node memory transmitting a resource data application request, allocating the resource data stored in the main memory to the target node memory according to the data category corresponding to the target node memory includes:

[0124] In case of a target node memory transmission resource data application request in the node memory, determining a resource data allocation estimate value of each target node memory based on the data capacity of the resource data of the main memory;

[0125] Determine the sorting result of each target node storage according to the corresponding resource data allocation estimation value;

[0126] According to the sorting result, resource data of the corresponding data category is allocated to each target node memory.

[0127] In one embodiment, allocating resource data of a corresponding data category to each target node memory includes:

[0128] For each target node memory, if the resource data allocation estimate value of the target node memory matches the resource data allocation value requested by the target node memory, the resource data of the data category corresponding to the target node memory in the main memory is called to the target node memory;

[0129] The resource data application request is a dynamic proxy action.

[0130] In one embodiment, the resource data allocation value of the application includes the content capacity and processing capability of the target node memory.

[0131] In one embodiment, the data distribution processor, the main memory and the node memory are connected via a set communication protocol.

[0132] In one embodiment, the distributed storage method further includes:

[0133] Storing connection information of a client connected to the distributed storage server in a security thread;

[0134] The connection corresponding to the connection information stored in the security thread is checked through the timing thread, and the connection that meets the closing condition is closed.

[0135] In one embodiment, the distributed storage method further includes:

[0136] Creating a data table on the main memory;

[0137] After allocating the resource data stored in the main memory to the target node memory, the allocation result is recorded in the data table, and the allocation result includes log information of the operation of the target node memory.

[0138] In one embodiment, the distributed storage method further includes:

[0139] In the case of failure in obtaining the allocation result, resource data is allocated to the target node memory according to the data address in the data log of the target node memory.

[0140] Embodiment 6

[0141] Figure 6 1 is a schematic diagram of the structure of a distributed storage device provided according to Embodiment 6 of the present invention. The distributed storage device can be integrated in a distributed storage server. The distributed storage device includes:

[0142] The acquisition module 610 is used to acquire a resource data application request transmitted by a target node memory.

[0143] The allocation module 620 is used to allocate the resource data stored in the main memory to the target node memory according to the data category corresponding to the target node memory.

[0144] In one embodiment, the allocation module 620 includes:

[0145] A determining unit, configured to determine an estimated value of resource data allocation of each target node memory based on the data capacity of the resource data of the main memory in the case of a target node memory transmission resource data application request in the node memory;

[0146] A sorting unit, used to determine the sorting result of each target node storage according to the corresponding resource data allocation estimation value;

[0147] An allocating unit is used to allocate resource data of a corresponding data category to each target node memory according to the sorting result.

[0148] In one embodiment, the allocation unit is specifically configured to:

[0149] For each target node memory, if the resource data allocation estimate value of the target node memory matches the resource data allocation value requested by the target node memory, the resource data of the data category corresponding to the target node memory in the main memory is called to the target node memory;

[0150] The resource data application request is a dynamic proxy action.

[0151] In one embodiment, the resource data allocation value of the application includes the content capacity and processing capability of the target node memory.

[0152] In one embodiment, the data distribution processor, the main memory and the node memory are connected via a set communication protocol.

[0153] In one embodiment, the distributed storage device further includes a storage module, which is used to:

[0154] Storing connection information of a client connected to the distributed storage server in a security thread;

[0155] The connection corresponding to the connection information stored in the security thread is checked through the timing thread, and the connection that meets the closing condition is closed.

[0156] In one embodiment, the distributed storage device further includes a recording module, which is used to:

[0157] Creating a data table on the main memory;

[0158] After allocating the resource data stored in the main memory to the target node memory, the allocation result is recorded in the data table, and the allocation result includes log information of the operation of the target node memory.

[0159] In one embodiment, the distributed storage device further includes a failure processing module, which is used to:

[0160] In the case of failure in obtaining the allocation result, resource data is allocated to the target node memory according to the data address in the data log of the target node memory.

[0161] The distributed storage device provided in the embodiment of the present invention can execute the distributed storage method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0162] In the context of the present invention, a computer-readable storage medium stores computer instructions, and the computer instructions are used to implement the distributed storage method provided by the present invention when the processor is executed. A computer-readable storage medium can be a tangible medium, which can contain or store a computer program for use by an instruction execution system, a device or an equipment or used in combination with an instruction execution system, a device or an equipment. A computer-readable storage medium can include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the above. Alternatively, a computer-readable storage medium can be a machine-readable signal medium. A more specific example of a machine-readable storage medium can include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0163] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a cathode ray tube (CRT) or a liquid crystal display (LCD monitor)) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or a trackball), through which the user may provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0164] The systems and techniques described herein can be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system can be interconnected by digital data communication (e.g., a communication network) in any form or medium. Examples of communication networks include: Local Area Network (LAN), Wide Area Network (WAN), blockchain network, and the Internet.

[0165] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship between the client and the server is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and virtual private servers (VPS) services.

[0166] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0167] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A distributed storage server, characterized in that: include: A data distribution processor, a main memory and at least one node memory, wherein the main memory and the node memory are respectively connected to the data distribution processor; The main memory is used to store resource data; The node storage is used to classify and store the resource data; The data allocation processor is used for allocating the resource data stored in the main memory to the target node memory according to the data category corresponding to the target node memory when a target node memory in the node memory requests the target node memory to transfer resource data; The data distribution processor is specifically used for: In case of a target node memory transmission resource data application request in the node memory, determining a resource data allocation estimate value of each target node memory based on the data capacity of the resource data of the main memory; Determine the sorting result of each target node storage according to the corresponding resource data allocation estimation value; According to the sorting result, resource data of the corresponding data category is allocated to each target node storage; The step of determining a resource allocation estimate value of each target node memory based on the data capacity of the resource data of the main memory comprises: The content capacity parameters of the resource data are input into the probability model, and the amount of resource data that can be allocated to each target node memory is output.

2. According to the distributed storage server of claim 1, the data distribution processor is specifically used for: For each target node memory, if the resource data allocation estimate value of the target node memory matches the resource data allocation value requested by the target node memory, the resource data of the data category corresponding to the target node memory in the main memory is called to the target node memory; in, The resource data application request is an action of a dynamic proxy.

3. According to the distributed storage server of claim 2, the resource data allocation value requested includes the content capacity and processing capability of the target node storage.

4. The distributed storage server according to claim 1, characterized in that: The data distribution processor, the main memory, and the node memory are connected via a set communication protocol.

5. The distributed storage server according to claim 1, characterized in that: The data distribution processor is also used to: Storing connection information of a client connected to the distributed storage server in a security thread; The connection corresponding to the connection information stored in the security thread is checked through the timing thread, and the connection that meets the closing condition is closed.

6. The distributed storage server according to claim 1, characterized in that: The data distribution processor is also used to: Creating a data table on the main memory; After allocating the resource data stored in the main memory to the target node memory, the allocation result is recorded in the data table, and the allocation result includes log information of the operation of the target node memory.

7. The distributed storage server according to claim 6, characterized in that: The data distribution processor is also used to: In the case of failure in obtaining the allocation result, resource data is allocated to the target node memory according to the data address in the data log of the target node memory.

8. A distributed storage method, characterized in that: The distributed storage method is performed by a data distribution processor in a distributed storage server according to any one of claims 1 to 7, wherein the distributed storage method comprises: In the case of a target node memory transmission resource data application request in the node memory, the resource data stored in the main memory is allocated to the target node memory according to the data category corresponding to the target node memory.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to implement the distributed storage method described in claim 8 when executed by a processor.

Images