Data storage method and device, and computer-readable storage medium

Abstract

The embodiment of the present invention discloses a data storage method, device, and storage medium, which receive a file upload request sent by a terminal; determine the storage location of the file using a custom method based on the file upload request and send it to the terminal; receive a file access request sent by the terminal; determine a target data center in a preset data center based on the file access request; the preset data center contains the storage location of the file; obtain the stored file based on the storage location through the target data center and send it to the terminal. In the above scheme, on the one hand, the storage location of the file is selected in the preset data center through a custom method; on the other hand, when reading the file, the target data center is determined in the preset data center containing multiple storage locations, and then the file is read from the target data center. The overall scheme saves resources, simplifies system complexity, and reduces the possibility of failure.

Description

Technical Field

[0001] The present invention relates to the field of big data technology, and in particular to a data storage method and device, and a computer-readable storage medium. Background Art

[0002] With the development of big data, various industries are becoming increasingly dependent on data. To ensure data accuracy and reliability, data disaster recovery is crucial. HBase disaster recovery generally refers to data center-level disaster recovery, typically using data center cluster server disaster recovery technology. Two HBase clusters are deployed in different data centers. Using specific technical means, data is written to both clusters simultaneously, or first written to a primary cluster and then replicated to a backup cluster, ensuring eventual consistency between the two clusters. This ensures that if a data center fails, data services can be switched to the backup cluster through failover and other technical means, ensuring data center-level disaster recovery for the HBase cluster.

[0003] However, in current data storage, only the primary cluster provides services. Under normal circumstances, the standby cluster only performs data synchronization and backup, but does not provide services. To ensure that the standby cluster can effectively synchronize data with the primary cluster and guarantee data reliability, the standby cluster's resource configuration must be consistent with that of the primary cluster. This often results in the standby cluster's computing resources being completely idle. Furthermore, in active-active disaster recovery solutions, the two clusters serve as both primary and standby, providing services simultaneously. This appears to fully utilize the computing resources of both clusters. However, to ensure consistent service delivery to all businesses, technical measures are required to ensure that only one cluster provides services at a time. Otherwise, duplicate computations occur across the two clusters, wasting computing resources. HDFS typically stores data in three different storage locations. This means that, in an active-active / active dual HBase cluster solution, a single piece of data is typically stored in six copies, significantly wasting storage resources. Furthermore, in an active-active / active dual cluster solution, only the primary cluster provides services for a particular business under normal circumstances. Only when the primary cluster fails will technical measures be used to switch services to the standby cluster. This inevitably requires the design of additional cluster health monitoring modules and failover modules, which increases system complexity, implementation difficulty, and the possibility of failure. Summary of the Invention

[0004] Embodiments of the present invention provide a data storage method and device, and a computer-readable storage medium, which can save resources, simplify system complexity, and reduce the possibility of failure.

[0005] The technical solution of the present invention is achieved as follows:

[0006] An embodiment of the present invention provides a data storage method, which is applied to a server. The method includes:

[0007] Receive the file upload request sent by the terminal;

[0008] According to the file upload request, using a custom method, determine the storage location of the file and send it to the terminal;

[0009] receiving a file access request sent by the terminal;

[0010] Determining a target data center in a preset data center according to the file access request; the preset data center includes a storage location of the file;

[0011] The target data center obtains the stored file according to the storage location and sends it to the terminal.

[0012] In the above solution, the preset data center includes a first data center and a second data center; the first data center and the second data center each include a plurality of different racks; the plurality of different racks respectively include a plurality of different data nodes; the distance value between the data nodes of different racks in the first data center is set to a first distance value; the distance value between the data nodes of different racks in the second data center is set to a second distance value;

[0013] The method of determining the storage location of the file and sending the file to the terminal using a custom method according to the file upload request includes:

[0014] Obtaining data nodes of the first data center and data nodes of the second data center;

[0015] The storage location of the file is determined based on the first distance value, the second distance value, and the data nodes of the first data center and the data nodes of the second data center, and the file is sent to the terminal.

[0016] In the above solution, determining the storage location of the file and sending it to the terminal based on the first distance value, the second distance value, and the nodes of the first data center and the second data center includes:

[0017] Get the first data node;

[0018] Determine, based on the first data node and in combination with the first distance value and the second distance value, a second data node, a third data node, and a fourth data node;

[0019] A storage location of a file is determined according to the first data node, the second data node, the third data node, and the fourth data node, and the storage location of the file is sent to the terminal.

[0020] In the above solution, obtaining the first data node includes:

[0021] Traversing multiple data nodes in a cluster, and if the terminal is not in the cluster, randomly obtaining the first data node from the multiple data nodes; the cluster includes the multiple data nodes;

[0022] If the terminal is in the cluster, the data node where the terminal is located is the first data node.

[0023] In the above solution, the sum of the distances between any two data nodes among the first data node, the second data node, the third data node and the fourth data node is a preset value;

[0024] The determining, based on the first data node and in combination with the first distance value and the second distance value, a second data node, a third data node, and a fourth data node, respectively, includes:

[0025] If the first data node is a node of the first data center, determining the second data node according to the first data node and the first distance value;

[0026] Determine the third data node according to the second data node and the second data center; the third data node is any node in the second data center;

[0027] determining a fourth data node according to the third data node and the second distance value;

[0028] In the above solution, the sum of the distances between any two data nodes among the first data node, the second data node, the third data node and the fourth data node is a preset value;

[0029] The determining, based on the first data node and in combination with the first distance value and the second distance value, a second data node, a third data node, and a fourth data node, respectively, includes:

[0030] If the first data node is a node of the second data center, determining the second data node according to the first data node and the second distance value;

[0031] Determine the third data node according to the second data node and the first data center; the third data node is any node in the first data center;

[0032] The fourth data node is determined according to the third data node and the first distance value.

[0033] In the above solution, determining a target data center from among preset data centers according to the file access request includes:

[0034] Comparing the distances between the first data center and the second data center and the terminal respectively; and determining a comparison result;

[0035] Determine the closest data center according to the comparison result, where the closest data center is the target data center;

[0036] In the above solution, obtaining the stored file according to the storage location through the target data center and sending it to the terminal includes:

[0037] If the first data center is the target data center and it is detected that the first data center has not failed, obtaining the file according to the storage location in the first data center and sending the file to the terminal;

[0038] If the first data center is the target data center and a failure is detected in the first data center, the file is obtained in the second data center according to the storage location and sent to the terminal.

[0039] In the above solution, if the first data center is the target data center and a failure of the first data center is detected, then obtaining the file according to the storage location in the second data center and sending the file to the terminal includes:

[0040] If a failure in the first data center is detected by the deployed monitoring and identification service layer, the relevant services are restarted in the second data center;

[0041] After the related service is started, the second data center obtains the file according to the storage location and sends the file to the terminal.

[0042] An embodiment of the present invention provides a data storage method, which is applied to a terminal. The method includes:

[0043] Send a file upload request to the server;

[0044] Receiving the file storage location sent by the server according to the file upload request;

[0045] According to the received storage location, the files are sent to the storage location in order for storage. In the above solution, after the files are sent to the storage location in order for storage according to the storage location, the method further includes:

[0046] If the file stored in the terminal is damaged, sending a file access request to the server;

[0047] Based on the file access request, the file sent by the server through the storage location is received for replacement.

[0048] An embodiment of the present invention provides a data storage device, which includes a first receiving unit, a first determining unit, and a first sending unit; wherein,

[0049] A first receiving unit, configured to receive a file upload request sent by a terminal;

[0050] A first sending unit is configured to determine the storage location of the file and send the file to the terminal using a custom method according to the file upload request;

[0051] The first receiving unit is further configured to receive a file access request sent by the terminal;

[0052] A first determining unit is configured to determine a target data center in a preset data center according to the file access request; the preset data center includes a storage location of the file;

[0053] The first sending unit is further configured to obtain the stored file according to the storage location through the target data center and send the file to the terminal.

[0054] An embodiment of the present invention provides a data storage device, which includes a second sending unit and a second receiving unit; wherein,

[0055] A second sending unit, configured to send a file upload request to the server;

[0056] a second receiving unit, configured to receive a file storage location sent by the server according to the file upload request;

[0057] The second sending unit is further used to send the files to the storage location in sequence for storage according to the received storage location.

[0058] An embodiment of the present invention provides a server, comprising:

[0059] A first memory, configured to store executable instructions;

[0060] The first processor is configured to execute the executable instructions stored in the first memory. When the executable instructions are executed, the first processor executes the data storage method applied to the server side.

[0061] An embodiment of the present invention provides a terminal, comprising:

[0062] a second memory for storing executable instructions;

[0063] The second processor is configured to execute the executable instructions stored in the second memory. When the executable instructions are executed, the second processor executes the data storage method applied to the terminal.

[0064] An embodiment of the present invention provides a storage medium storing executable instructions. When the executable instructions are executed, the storage medium causes a first processor to execute the data storage method according to the embodiment of the present invention.

[0065] An embodiment of the present invention provides a storage medium storing executable instructions. When the executable instructions are executed, the storage medium causes a second processor to execute the data storage method according to the embodiment of the present invention.

[0066] An embodiment of the present invention provides a data storage method and device, and a computer-readable storage medium, the method comprising: receiving a file upload request sent by a terminal; determining the storage location of the file using a custom method based on the file upload request and sending it to the terminal; receiving a file access request sent by the terminal; determining a target data center in a preset data center based on the file access request; the preset data center contains the storage location of the file; obtaining the stored file based on the storage location through the target data center and sending it to the terminal. Using the above scheme, on the one hand, a storage location of the file is selected in a preset data center using a custom method; on the other hand, when reading a file, a target data center is determined in a preset data center containing multiple storage locations, and then the file is read from the target data center. The overall scheme saves resources, simplifies system complexity, and reduces the possibility of failure. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] Figure 1 A schematic diagram of a data storage method for a server provided in an embodiment of the present invention Figure 1 ;

[0068] Figure 2 A schematic diagram of a data storage method for a server provided in an embodiment of the present invention Figure 2 ;

[0069] Figure 3 A schematic diagram of an adjacency matrix effect of a storage data center node provided by an embodiment of the present invention;

[0070] Figure 4 A schematic diagram of a data storage method for a server provided in an embodiment of the present invention Figure 3 ;

[0071] Figure 5A schematic diagram of a data storage method for a server provided in an embodiment of the present invention Figure 4 ;

[0072] Figure 6 A schematic diagram of a data storage method for a server provided in an embodiment of the present invention Figure 5 ;

[0073] Figure 7 A schematic diagram of a data storage method for a server provided in an embodiment of the present invention Figure 6 ;

[0074] Figure 8 A structural diagram of a metadata management module provided by an embodiment of the present invention;

[0075] Figure 9 A structural diagram of a cross-data center HBase disaster recovery system provided by an embodiment of the present invention;

[0076] Figure 10 A schematic flow chart of a method for storing data in a terminal according to an embodiment of the present invention;

[0077] Figure 11 A schematic diagram of a data storage method according to an embodiment of the present invention is provided;

[0078] Figure 12 A schematic structural diagram of a data storage device provided in an embodiment of the present invention;

[0079] Figure 13 A schematic structural diagram of another data storage device provided by an embodiment of the present invention;

[0080] Figure 14 A schematic diagram of the structure of a server provided in an embodiment of the present invention;

[0081] Figure 15 A schematic diagram of the structure of a terminal provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0082] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making any creative efforts shall fall within the scope of protection of the present invention.

[0083] In order to enable those skilled in the art to better understand the present invention, the present invention will be further described in detail below with reference to the accompanying drawings and specific implementation methods. Figure 1A schematic diagram of a data storage method for a server provided in an embodiment of the present invention Figure 1 , will combine Figure 1 The steps shown are explained.

[0084] S101: Receive a file upload request sent by a terminal.

[0085] In an embodiment of the present invention, a client wants to store a file and sends a file upload request to a server. After the server receives the file upload request sent by the terminal, it performs subsequent operations.

[0086] S102: According to the file upload request, a custom method is used to determine the storage location of the file and send it to the terminal.

[0087] In an embodiment of the present invention, after receiving the file upload request sent by the terminal, the server uses a custom method to determine the specific storage location of the file and sends the location to the client.

[0088] In an embodiment of the present invention, HBase data is stored on HDFS. To ensure data reliability, HDFS itself uses a multi-copy mechanism, typically three copies. That is, for one piece of data, HDFS typically records three copies in different storage locations. This means that for a dual HBase cluster in a master-slave / active-active solution, one piece of data is typically stored in six copies, which is a huge waste of storage resources. However, in the present invention, a customized method is used to deploy only one HBase cluster. By modifying the HDFS replica placement strategy, cross-data center disaster recovery is achieved at the storage layer. For one piece of data, only four copies need to be saved, and the four determined locations are ultimately sent to the customer.

[0089] S103: Receive a file access request sent by the terminal.

[0090] In an embodiment of the present invention, the server receives a file access request sent by the terminal.

[0091] In the embodiment of the present invention, after storing a file, the terminal wants to read the stored file again and sends a file access request to the server. The server receives the request and performs subsequent operations.

[0092] S104. Determine a target data center from among preset data centers according to the file access request; the preset data center includes a storage location of the file.

[0093] In an embodiment of the present invention, after the client stores a file according to the location sent by the server, if the client wants to retrieve the file, the client sends a file access request to the server. The server determines the target data center from multiple data centers containing the file storage location based on the file access request.

[0094] In the embodiment of the present invention, the preset data centers all include the storage location of the file, and then a target data center that meets the requirements is found in the preset data centers according to preset conditions.

[0095] S105. Obtain the stored file according to the storage location through the target data center and send it to the terminal.

[0096] In an embodiment of the present invention, after the target data center is found, the stored file is obtained according to the storage location and sent to the terminal.

[0097] As you can understand, in this embodiment of the present invention, a terminal sends a file upload request to the server, which then uses a custom method to select a file storage location. After storage is complete, the terminal sends a file access request. The server finds the target data center, retrieves the stored file based on the file storage location, and sends it to the terminal. This solution, by modifying HDFS's replica placement strategy and introducing a containerized approach to deploying the HBase cluster's HMaster and RegionServer, conserves resources, simplifies system complexity, and reduces the potential for failure.

[0098] In some embodiments of the present invention, see Figure 2 , Figure 2 This is a flow diagram of a method for storing data on a server provided by an embodiment of the present invention. Figure 2 , Figure 2 The S102 shown can be realized by S1021 to S1022, which will be combined Figure 2 The steps shown are explained.

[0099] S1021. Obtain data nodes of the first data center and data nodes of the second data center; multiple different racks respectively include multiple different data nodes; the distance value between the data nodes of different racks of the first data center is set to a first distance value; the distance value between the data nodes of different racks of the second data center is set to a second distance value.

[0100] In some embodiments of the present invention, multiple data nodes are divided to determine data nodes included in a first data center and data nodes included in a second data center.

[0101] In some embodiments of the present invention, the first data center and the second data center each include multiple racks, and the nodes included in the first data center and the second data center may also belong to different racks.

[0102] For example, suppose there are 12 data nodes, A, B, C, ..., L. AF belongs to the first data center, AC belongs to the same rack in the first data center, DF belongs to another rack in the first data center, GL belongs to the second data center, GI belongs to the same rack in the second data center, and IL belongs to another rack in the second data center.

[0103] S1022: Determine the storage location of the file based on the first distance value, the second distance value, and the data nodes of the first data center and the data nodes of the second data center, and send the file to the terminal.

[0104] In some embodiments of the present invention, the distance value between nodes that belong to the same first data center but are in different racks in the first data center is set as the first distance value.

[0105] For example, Figure 3 As shown, Figure 3 This diagram shows the effect of storing the adjacency matrix of data center nodes. The values ​​in this matrix represent the distances between nodes in the same data center and the same rack, the distances between nodes in the same data center but different racks, and the distances between nodes in different data centers. For example, if node A and nodes D, E, and F are in the first data center but different racks, the distances from node A to nodes D, E, and F are set to the first distance value, which can be 2. The same applies to other data nodes.

[0106] In some embodiments of the present invention, the distance between nodes that belong to the same second data center but are in different racks of the second center is set to a second distance value.

[0107] For example, Figure 3 As shown, node G and nodes J, K, and L are nodes in the second data center but in different racks, so the distances from node G to nodes J, K, and L are set to the second distance value, which may be 3. The same applies to other data nodes.

[0108] In some embodiments of the present invention, after obtaining the first distance value and the second distance value, the storage location of the file is determined by combining the nodes of the first data center and the nodes of the second data center, and the determined location is sent to the client.

[0109] It is understandable that in some embodiments of the present invention, the concept of data center is added to HDFS, so that when HDFS stores data, only a small number of copies are needed to ensure the consistency and reliability of data in two data centers.

[0110] In some embodiments of the present invention, see Figure 4 , Figure 4This is a flow diagram of a method for storing data on a server provided by an embodiment of the present invention. Figure 3 , Figure 4 The S1024 shown can be realized by S10241 to S10243, which will be combined Figure 4 The steps shown are explained.

[0111] S10241. Obtain a first data node.

[0112] In some embodiments of the present invention, a first data node is first obtained, and subsequent operations are performed based on the first data node in combination with the first distance value and the second distance value.

[0113] S10242. Determine, based on the first data node and in combination with the first distance value and the second distance value, a second data node, a third data node, and a fourth data node.

[0114] In some embodiments of the present invention, after the first data node is acquired, the first distance value and the second distance value are combined to determine the second data node, the third data node, and the fourth data node, respectively.

[0115] S10243. Determine a storage location of the file based on the first data node, the second data node, the third data node, and the fourth data node, and send the location to the terminal.

[0116] In some embodiments of the present invention, after the storage location of the file is obtained based on the four obtained data nodes, the location information is returned to the terminal.

[0117] Illustratively, if the first data node is located on a specific rack of the first data center, it indicates that the file is stored on a specific rack of the first data center. After the specific storage location of the file is obtained, the location is returned to the terminal.

[0118] It is understandable that in some embodiments of the present invention, the four selected data nodes are nodes in different data centers and different racks. Such a placement strategy ensures that different data centers have copies of a certain data block, and there is a certain fault tolerance mechanism in any data center.

[0119] In some embodiments of the present invention, S10241 can be implemented through S201 and S202, and a specific description is given in combination with the steps.

[0120] S201. Traverse multiple data nodes in a cluster. If the terminal is not in the cluster, randomly obtain a first data node from preset data nodes. The cluster includes multiple data nodes.

[0121] In some embodiments of the present invention, multiple data nodes in a cluster are traversed to determine whether the terminal is in the cluster. If the terminal is in the cluster, the data node where the terminal is located is directly selected as the first data node.

[0122] In some embodiments of the present invention, if D1 and D2 are two data centers, R1, R2, and R3 are three racks in D1, and R1 has three nodes, N1, N2, and N3, and N1, N2, and N3 are three nodes in a cluster, then if a terminal is on N1, it indicates that the terminal is in the cluster; if the terminal is on R1, it indicates that the terminal is not in the cluster. In the present invention, only one cluster is deployed, and when a terminal is not in the cluster, the first data node is randomly obtained from multiple data nodes.

[0123] S202: If the terminal is in the cluster, the data node where the terminal is located is the first data node.

[0124] In some embodiments of the present invention, when a terminal is in a cluster, the data node where the terminal is located is referred to as the first data node.

[0125] In some embodiments of the present invention, see Figure 5 , Figure 5 This is a flow diagram of a method for storing data on a server provided by an embodiment of the present invention. Figure 4 , Figure 5 The S10242 shown can be realized by S301 to S303, which will be combined Figure 5 The steps shown are explained.

[0126] S301: If the first data node is a node of a first data center, determine a second data node according to the first data node and a first distance value.

[0127] In some embodiments of the present invention, if the first data node is a node of a first data center, the second data node is determined based on the first data node and the first distance value.

[0128] In some embodiments of the present invention, if the first data node is a node in the first data center, a node that meets the requirements is determined in the first data center according to S201 and S202, and then a second data node is determined according to the first data node and the first distance value. Figure 3 As shown, if the first data node is node A in the first data center, because the first distance value is the distance value between data nodes in different racks in the first data center, the second node can be selected from nodes D, E, and F in the first data center.

[0129] S302. Determine the third data node according to the second data node and the second data center; the third node is any node in the second data center.

[0130] In some embodiments of the present invention, after the second data node is acquired, a third data node is determined based on the second data node and the second data center. The third data node is any node in the second data center.

[0131] For example, if the second data node is set to D according to S301, Figure 3 As shown, the distance from node D to any node in the second data center is 3, so any node is selected in the second data center as the third node.

[0132] S303: Determine a fourth data node according to the third data node and the second distance value.

[0133] In some embodiments of the present invention, after the third data node is acquired, the fourth data node is determined based on the third data node and the second distance value.

[0134] In some embodiments of the present invention, because the second distance value is the distance value between data nodes in different racks of the second data center, the fourth data node determined based on the third data node and the second distance value is a node in the same data center but a different rack.

[0135] In some embodiments of the present invention, the sum of the distances between any two data nodes among the finally selected first data node, second data node, third data node, and fourth data node is a preset value.

[0136] Exemplarily, the selected first data node is A, the second data node is D, the third data node is G, and the fourth data node is J. The preset value can be 16, then AD(2)+AG(3)+AJ(3)+DG(3)+DJ(3)+GJ(2)=16.

[0137] It can be understood that in some embodiments of the present invention, the four selected data nodes are nodes in different data centers and different racks. The storage location of the file is determined by this method, which increases the reliability and fault tolerance of the file storage.

[0138] In some embodiments of the present invention, see Figure 6 , Figure 6 This is a flow diagram of a method for storing data on a server provided by an embodiment of the present invention. Figure 5 , Figure 6 The S10242 shown can be realized by S401 to S403, which will be combined Figure 6 The steps shown are explained.

[0139] S401: If the first data node is a node of the second data center, determine the second data node according to the first data node and the second distance value.

[0140] In some embodiments of the present invention, if the first data node is a node of the second data center, the second data node is determined based on the first data node and the second distance value.

[0141] In some embodiments of the present invention, if the first data node is a node in the second data center, a node that meets the requirements is determined in the second data center according to S201 and S202, and then the second data node is determined according to the first data node and the second distance value. Figure 3 As shown, if the first data node is node G in the second data center, because the second distance value is the distance value between data nodes in different racks in the second data center, the second data node can be selected from nodes J, K, and L in the second data center.

[0142] S402: Determine a third data node based on the second data node and the first data center; the third data node is any node in the second data center.

[0143] In some embodiments of the present invention, after the second data node is acquired, a third data node is determined based on the second data node and the first data center, where the third data node is any node in the second data center.

[0144] For example, if the second data node is set to J according to S401, Figure 3 As shown, the distance between node J and any node in the first data center is 3, so any node is selected as the third node in the first data center.

[0145] S403: Determine a fourth data node according to the third data node and the first distance value.

[0146] In some embodiments of the present invention, after the third node is acquired, the fourth data node is determined according to the third node and the first distance value.

[0147] In some embodiments of the present invention, because the first distance value is the distance between data nodes in different racks of the second data center, the fourth data node determined based on the third data node and the first distance value is a node in the same data center but a different rack.

[0148] In some embodiments of the present invention, the sum of the distances between any two data nodes of the finally selected first data node, second data node, third data node, and fourth data node is a preset value.

[0149] Exemplarily, the selected first data node is G, the second data node is J, the third data node is D, the fourth data node is A, and the preset value can be 16, then GJ(2)+GD(3)+GA(3)+JD(3)+JA(3)+DA(2)=16.

[0150] It can be understood that in some embodiments of the present invention, the four selected data nodes are nodes in different data centers and different racks. The storage location of the file is determined by this method, which increases the reliability and fault tolerance of the file storage.

[0151] In some embodiments of the present invention, S103 can be implemented through S1031 to S1032, which will be described in detail in conjunction with the steps.

[0152] S1031. Compare the distances between the first data center and the second data center and the terminal respectively; and determine a comparison result.

[0153] In some embodiments of the present invention, the distance from the first data center to the terminal and the distance from the second data center to the terminal are respectively obtained, and the two distances are compared to determine a comparison result.

[0154] In some embodiments of the present invention, the storage location of a file is obtained and sent to a terminal. After the terminal obtains the location, the file is uploaded. After the file is uploaded, if the uploaded file is to be read, the first data center storing the file and the second data center are compared, a comparison result is determined, and the file is retrieved based on the comparison result.

[0155] S1032. Determine the nearest data center based on the comparison result; the nearest data center is the target data center.

[0156] In some embodiments of the present invention, after the comparison result is obtained, the closest data center is determined according to the comparison result.

[0157] In some embodiments of the present invention, after comparing the first data center and the second data center, the nearest data center is selected according to the proximity principle.

[0158] In some embodiments of the present invention, S1033 can be implemented through S10331 to S10332, which will be specifically described in conjunction with the steps.

[0159] S10331. If the first data center is the target data center and it is detected that no failure occurs in the first data center, obtain the file according to the storage location in the first data center and send it to the terminal.

[0160] In some embodiments of the present invention, if the first data center is the target data center, it is determined whether the data center has failed. If the data center has not failed, after finding the storage location of the file in the data center, the file is obtained and sent to the terminal.

[0161] S10332: If the first data center is the target data center and a failure is detected in the first data center, the file is obtained in the second data center according to the storage location and sent to the terminal.

[0162] In some embodiments of the present invention, if the first data center is the target data center and a failure occurs, the second data center retrieves the file based on the storage location of the file and sends it to the terminal.

[0163] It's understood that in some embodiments of the present invention, nodes within a multi-data center Kubernetes cluster are labeled with different availability zones. Based on node affinity, the HMaster / RegionServer is prioritized in the data center closest to the terminal. If the nearest data center fails, this restriction is ignored and the HMaster / RegionServer service is restarted in another data center. This saves computing resources.

[0164] In some embodiments of the present invention, if the second data center is the closest data center, the operation results are the same as S10331 and S10332.

[0165] In some embodiments of the present invention, see Figure 7 , Figure 7 This is a flow diagram of a method for storing data on a server provided by an embodiment of the present invention. Figure 6 , Figure 7 The S10332 shown can be realized by S501 to S502, which will be combined Figure 7 The steps shown are explained.

[0166] S501: If a failure occurs in the first data center through the deployed monitoring and identification service layer, related services are started in the second data center.

[0167] In some embodiments of the present invention, when a service layer identified by deployed monitoring detects a failure in the nearest data center, the relevant service is started in a second data center.

[0168] In some embodiments of the present invention, containerized deployment is implemented using a cross-data center Kubernetes cluster. To ensure that terminals are able to access HBase clusters in data centers that are close to each other in terms of network topology, nodes within the cross-data center Kubernetes cluster are labeled with different availability zones. Based on node affinity, the HMaster / RegionServer is prioritized in the first data center. If the first data center fails, this restriction is ignored and the HMaster / RegionServer service is restarted in the second data center.

[0169] In some embodiments of the present invention, in order to facilitate terminal access to the HBase cluster and prevent changes in the cluster connection information due to changes in the hmaster / regionserver containers, a unified service is used to expose the web UI of the containerized hmaster and regionserver. That is, a layer of monitoring and identification service is deployed above the hmaster and regionserver containers. This service can monitor changes in the hmaster / regionserver container IP in real time. Instead of directly exposing the container IP, the container IP is bound to a fixed service-exposed domain name, and the terminal only accesses it through the fixed domain name. When the service request reaches the monitoring service layer, it will be parsed and forwarded to the corresponding hmaster / regionserver. In this way, even if a failure occurs in the primary data center and the HBase cluster-related containers are restarted on the nodes of another data center, the connection information of the HBase cluster will not change, so that the terminal is unaware.

[0170] In some embodiments of the present invention, for HBase cluster, Zookeeper that stores metadata is also a crucial part, and disaster recovery and high availability also need to be considered. For Zookeeper, a cross-data center kubernetes cluster is also used for containerized deployment, and Zookeeper data is stored externally. In this way, even if the Zookeeper container is restarted in another data center, the consistency of Zookeeper data can be guaranteed. The structure diagram of the metadata management module Zookeeper is as follows: Figure 8As shown, ZooKeeper is primarily used to solve data management issues commonly encountered in distributed applications, such as unified naming services, state synchronization services, cluster management, and management of distributed application configuration items. Zone A and Zone B represent different data centers, and Node represents a data node. Ceph is a unified, distributed file system designed for excellent performance, reliability, and scalability, providing object storage, block device storage, and file system services. Kubernetes is an open-source container orchestration engine from Google that supports automated deployment, large-scale scalability, and containerized application management. When deploying an application in a production environment, multiple instances of the application are typically deployed to load balance application requests. In Kubernetes, multiple containers can be created, each running an application instance. Built-in load balancing policies enable management, discovery, and access to this group of application instances, eliminating the need for complex manual configuration and processing by operations personnel.

[0171] S502: After the relevant service is started, in the second data center, the file is obtained according to the storage location of the file and sent to the terminal.

[0172] In some embodiments of the present invention, after detecting that a failure occurs in the nearest data center, the relevant service is restarted on the node of the second data center, and then the storage location of the file is found in the second data center, and the file is obtained according to the location and sent to the terminal.

[0173] In some embodiments of the present invention, the related services refer to hmaster / regionserver services. The structure diagram of the entire solution is as follows: Figure 9 As shown, Figure 9This diagram illustrates the structure of a cross-data center HBase disaster recovery system, provided by an embodiment of the present invention. At the storage layer, a cross-data center HDFS cluster is employed, with a four-replica data placement strategy. For example, the red data block 1 is placed on different nodes in different zones and racks (e.g., red nodes 1, 2, 3, and 4 in the figure), achieving cross-data center fault tolerance. At the compute layer, under normal conditions, all deployed service containers are located in zone A. If zone A fails, the relevant container services are restarted in zone B, also achieving data center fault tolerance. Racks 1 and 2 represent racks, and the HMaster is the central node in the HBase master / slave cluster architecture. Typically, an HBase cluster has multiple HMaster nodes, one of which is the Active Master and the others are Backup Masters. HBase only has one HMaster server running at any given time. The HMaster assigns regions to region servers, coordinates the region server load, and maintains the cluster's status. The HMaster does not provide external data services; instead, the region servers handle read and write requests and operations for all regions. Since the hmaster only maintains the metadata of tables and regions and does not participate in the data input / output process, the failure of the hmaster will only cause all metadata to be unable to be modified, but the table data can still be read / written normally.

[0174] It is understandable that in some embodiments of the present invention, only one set of HBase computing resources is deployed through containerization. When a data center fails, the HBase computing resource container is restarted in another data center, saving computing resources while ensuring business continuity.

[0175] In the embodiments of the present invention, see Figure 10 , Figure 10 This is a flow chart of a method for storing data applied to a terminal provided by an embodiment of the present invention, which will be combined with Figure 10 The contents shown are described in detail.

[0176] S601: Send a file upload request to the server.

[0177] In the embodiment of the present invention, when the terminal wants to store a file, it sends a file upload request to the server.

[0178] S602: Receive the file storage location sent by the server according to the file upload request.

[0179] In an embodiment of the present invention, the server determines the storage location of the file according to the file upload request sent by the terminal, and sends the location to the terminal, and then the terminal receives the file storage location sent by the server.

[0180] S603: According to the received storage location, the files are sent to the storage location in order for storage.

[0181] In an embodiment of the present invention, the files are sent to the server in order according to the received file storage locations, so that the server stores the files.

[0182] In the embodiment of the present invention, a total of 4 copies are stored. After copy 1 is uploaded to the corresponding location, copy 2 is uploaded.

[0183] In some embodiments of the present invention, S603 further includes S701 , and S701 can be implemented through S7011 to S7012 , which will be specifically described in conjunction with the steps.

[0184] S7011. If the file stored in the terminal is damaged, a file access request is sent to the server.

[0185] In an embodiment of the present invention, after the file is stored, if the file stored in the terminal is damaged, the terminal sends a file access request to the server.

[0186] S7012. Based on the file access request, the receiving server replaces the file sent by the storage location.

[0187] In an embodiment of the present invention, after a file access request is sent to a server, the server obtains the stored file and sends it to the terminal, and the terminal receives the file sent by the server.

[0188] In an embodiment of the present invention, the terminal selects a server (based on the principle of proximity, then randomly) and requests to read data.

[0189] It can be understood that in an embodiment of the present invention, the terminal sends a file upload request to the server, stores files according to the number of stored files sent by the server that is different from the existing number, and utilizes the sent file access request, which not only improves the fault tolerance of file storage, but also saves storage and resources.

[0190] The embodiment of the present invention provides a method for data storage, the optional flow chart is as follows Figure 11 shown.

[0191] S1. The terminal sends a file upload request to the server.

[0192] S2. The server receives the file upload request sent by the terminal.

[0193] S3. The server determines the storage location of the file based on the received file upload request using a custom method and sends the file to the terminal.

[0194] S4. After sending the file upload request, the terminal receives the file storage location sent by the server.

[0195] S5. The terminal sends the files to the server in order according to the file storage location, so that the server stores the files.

[0196] S6. The terminal sends a file access request.

[0197] S7. The server receives the file access request sent by the terminal, and determines a target data center from the preset data centers according to the file access request.

[0198] S7. The server obtains the stored file according to the determined target data center and sends it to the terminal.

[0199] S8. After sending the file access request, the terminal receives the stored file sent by the server.

[0200] Understandably, in current file storage, six copies of a single piece of data are typically stored in a dual HBase cluster with a master-slave / active-active solution. This represents a significant waste of storage resources. This invention, by deploying an HBase cluster, modifying HDFS's replica placement strategy, and introducing a containerized approach to deploying the HBase cluster's HMaster and RegionServer, conserves resources, simplifies system complexity, and reduces the likelihood of failure.

[0201] An embodiment of the present invention provides a data storage device, such as Figure 12 As stated, Figure 12 A structural diagram of a data storage device provided in an embodiment of the present invention is applied to a server. The data storage device includes: a first receiving unit 1201, a first determining unit 1202, and a first sending unit 1203; wherein,

[0202] The first receiving unit 1201 is configured to receive a file upload request sent by a terminal.

[0203] The first sending unit 1202 is configured to determine the storage location of the file and send the file to the terminal using a custom method according to the file upload request.

[0204] The first receiving unit 1201 is further configured to receive a file access request sent by the terminal.

[0205] The first determining unit 1202 is configured to determine a target data center in a preset data center according to the file access request; the preset data center includes a storage location of the file.

[0206] The first sending unit 1203 is further configured to obtain the stored file according to the storage location through the target data center and send the file to the terminal.

[0207] In some embodiments of the present invention, the preset data center includes a first data center and a second data center; the first data center and the second data center each include multiple different racks; the multiple different racks respectively include multiple different data nodes; the distance value between the data nodes of different racks of the first data center is set to a first distance value; the distance value between the data nodes of different racks of the second data center is set to a second distance value; the first determination unit 1202 is also used to obtain the data nodes of the first data center and the data nodes of the second data center; and based on the first distance value, the second distance value, combined with the data nodes of the first data center and the data nodes of the second data center, determine the storage location of the file and send it to the terminal.

[0208] In some embodiments of the present invention, the first determination unit 1202 is further used to obtain a first data node; and determine a second data node, a third data node, and a fourth data node respectively based on the first data node in combination with the first distance value and the second distance value; and determine the storage location of the file based on the first data node, the second data node, the third data node, and the fourth data node and send it to the terminal.

[0209] In some embodiments of the present invention, the first determination unit 1202 is further used to randomly obtain the first data node from the multiple data nodes if the terminal is not in the cluster; the cluster includes the multiple data nodes; and if the terminal is in the cluster, the data node where the terminal is located is the first data node.

[0210] In some embodiments of the present invention, the sum of the distances between any two data nodes among the first data node, the second data node, the third data node, and the fourth data node is a preset value, and the first determining unit 1202 is further configured to, if the first data node is a node of the first data center, determine the second data node based on the first data node and the first distance value; and determine the third data node based on the second data node and the second data center; the third data node is any node of the second data center; and determine the fourth data node based on the third data node and the second distance value;

[0211] In some embodiments of the present invention, the sum of the distances between any two data nodes among the first data node, the second data node, the third data node and the fourth data node is a preset value, and the first determination unit 1202 is further used to determine the second data node based on the first data node and the second distance value if the first data node is a node of the second data center; and determine the third data node based on the second data node and the first data center; the third data node is any node of the first data center; and determine the fourth data node based on the third data node and the first distance value.

[0212] In some embodiments of the present invention, the first determination unit 1202 is further used to compare the distances between the first data center and the second data center and the terminal respectively; determine the comparison result; and determine the nearest data center based on the comparison result; and if it is detected that the nearest data center has not failed, obtain the file in the nearest data center based on the storage location of the file and send it to the terminal; and if it is detected that the nearest data center has failed, obtain the file in another data center based on the storage location of the file and send it to the terminal.

[0213] In some embodiments of the present invention, the first sending unit 1203 is further configured to compare the distances between the first data center and the second data center and the terminal, respectively; determine a comparison result; and determine the closest data center based on the comparison result, the closest data center being the target data center;

[0214] In some embodiments of the present invention, the first determination unit 1202 is further used to, if the first data center is the target data center and it is detected that the first data center has not failed, obtain the file according to the storage location in the first data center and send it to the terminal; and if the first data center is the target data center and it is detected that the first data center has failed, obtain the file according to the storage location in the second data center and send it to the terminal.

[0215] In some embodiments of the present invention, the first determination unit 1202 is further used to detect, through the deployed monitoring and identification service layer, that a failure has occurred in the first data center, and then restart the relevant services in the second data center; and in the second data center after the relevant services are started, obtain the file according to the storage location and send it to the terminal.

[0216] It can be understood that in the above device implementation, the server receives a file upload request from a terminal, uses a custom method to determine the file's storage location, and sends it to the terminal. Furthermore, after receiving a file access request from the terminal, the server determines a target data center within the pre-set data centers based on the file access request. Finally, the server retrieves the stored file from the target data center and sends it to the terminal. By customizing the HDFS replica placement strategy and introducing a containerized approach to deploy cluster-related services, resources are saved, system complexity is simplified, and the likelihood of failure is reduced.

[0217] The embodiment of the present invention provides a data storage device, such as Figure 13 As stated, Figure 13 A structural diagram of another data storage device provided in an embodiment of the present invention, which is applied to a terminal, includes: a second sending unit 1301 and a second receiving unit 1302; wherein,

[0218] The second sending unit 1301 is configured to send a file upload request to the server.

[0219] The second receiving unit 1302 is configured to receive the storage location of the file sent by the server according to the file upload request.

[0220] The second sending unit 1301 is further configured to send the files to the storage location in sequence for storage according to the received storage location.

[0221] The second sending unit 1301 is further configured to send a file access request to the server if the file stored in the terminal is damaged.

[0222] The second receiving unit 1302 is further configured to receive, based on the file access request, the file sent by the server through the storage location for replacement.

[0223] It can be understood that in the above-mentioned device implementation scheme, the terminal sends a file upload request to the server, stores files according to the number of stored files sent by the server that is different from the existing number, and utilizes the sent file access request, which not only improves the fault tolerance of file storage, but also saves storage and resources.

[0224] Based on the method of the above embodiment, the embodiment of the present invention provides a structural diagram of a server, such as Figure 14 As shown, Figure 14A structural diagram of a data storage device provided in an embodiment of the present application, wherein the server 14 includes: a first processor 1401 and a first memory 1402; the first memory 1402 stores one or more programs executable by the first processor 1401, and when one or more programs are executed, a data storage method corresponding to the server in the embodiment described above is executed by the first processor 1401.

[0225] Based on the method of the above embodiment, the embodiment of the present invention provides a structural diagram of a terminal, such as Figure 15 As shown, Figure 15 A structural diagram of a data storage device provided in an embodiment of the present application, wherein the terminal 15 includes: a second processor 1501 and a second memory 1502; the second memory 1502 stores one or more programs executable by the second processor 1501, and when one or more programs are executed, a data storage method corresponding to the terminal in the embodiment described above is executed by the second processor 1501.

[0226] An embodiment of the present invention provides a computer-readable storage medium, which is applied to a server and stores executable instructions for causing a first processor to execute the instructions to implement the method described in the server.

[0227] An embodiment of the present invention provides a computer-readable storage medium, which is applied to a terminal and stores executable instructions for causing a second processor to execute the instructions to implement the method described in the terminal.

[0228] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Thus, the present invention may take the form of hardware embodiments, software embodiments, or embodiments combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage and optical storage, etc.) containing computer-usable program code.

[0229] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0230] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0231] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0232] The above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention.

Claims

1. A data storage method, characterized in that: Applicable to servers, including: Receive the file upload request sent by the terminal; According to the file upload request, using a custom method, determining the storage location of the file and sending it to the terminal; receiving a file access request sent by the terminal; Determining a target data center in a preset data center according to the file access request; the preset data center includes a storage location of the file; Obtaining the stored file according to the storage location through the target data center and sending it to the terminal; The preset data center includes a first data center and a second data center; the first data center and the second data center each include a plurality of different racks; the plurality of different racks respectively include a plurality of different data nodes; the distance value between the data nodes of different racks in the first data center is set to a first distance value; the distance value between the data nodes of different racks in the second data center is set to a second distance value; The method of determining the storage location of the file and sending the file to the terminal using a custom method according to the file upload request includes: Obtaining data nodes of the first data center and data nodes of the second data center; Determine the storage location of the file based on the first distance value, the second distance value, and the data nodes of the first data center and the data nodes of the second data center, and send the file to the terminal; Determining a target data center from among preset data centers according to the file access request includes: Comparing the distances between the first data center and the second data center and the terminal respectively; and determining a comparison result; Determining the nearest data center based on the comparison result; The data center closest to the target data center is used as the target data center.

2. The method according to claim 1, characterized in that The determining, based on the first distance value and the second distance value, a storage location of the file in combination with a data node of the first data center and a data node of the second data center, and sending the file to the terminal includes: Get the first data node; Determine, based on the first data node and in combination with the first distance value and the second distance value, a second data node, a third data node, and a fourth data node; A storage location of a file is determined according to the first data node, the second data node, the third data node, and the fourth data node, and the storage location of the file is sent to the terminal.

3. The method according to claim 2, characterized in that The obtaining of the first data node includes: Traversing multiple data nodes in the cluster, and if the terminal is not in the cluster, randomly obtaining the first data node from the multiple data nodes; If the terminal is in the cluster, the data node where the terminal is located is the first data node.

4. The method according to claim 2, characterized in that: The sum of the distances between any two data nodes among the first data node, the second data node, the third data node and the fourth data node is a first distance value; Determining a second data node, a third data node, and a fourth data node, respectively, based on the first data node and in combination with the first distance value, includes: If the first data node is a node of the first data center, determining the second data node according to the first data node and the first distance value; Determine the third data node according to the second data node and the second data center; the third data node is any node in the second data center; The fourth data node is determined according to the third data node and the second distance value.

5. The method according to claim 2, characterized in that: The determining, based on the first data node and in combination with the first distance value and the second distance value, a second data node, a third data node, and a fourth data node, respectively, includes: If the first data node is a node of the second data center, determining the second data node according to the first data node and the second distance value; Determine the third data node according to the second data node and the first data center; the third data node is any node in the first data center; The fourth data node is determined according to the third data node and the first distance value.

6. The method according to claim 1, characterized in that The acquiring, through the target data center, the stored file according to the storage location and sending the file to the terminal includes: If the first data center is the target data center and it is detected that the first data center has not failed, obtaining the file according to the storage location in the first data center and sending the file to the terminal; If the first data center is the target data center and a failure is detected in the first data center, the file is obtained in the second data center according to the storage location and sent to the terminal.

7. The method according to claim 6, characterized in that If the first data center is the target data center and a failure of the first data center is detected, obtaining the file according to the storage location in the second data center and sending the file to the terminal includes: If a failure in the first data center is detected by the deployed monitoring and identification service layer, the relevant services are restarted in the second data center; After the related service is started, the second data center obtains the file according to the storage location and sends the file to the terminal.

8. A data storage device, characterized in that include: A first receiving unit, configured to receive a file upload request sent by a terminal; A first sending unit, configured to determine the storage location of the file and send the file to the terminal using a custom method according to the file upload request; The first receiving unit is further configured to receive a file access request sent by the terminal; A first determining unit is configured to determine a target data center in a preset data center according to the file access request; the preset data center includes a storage location of the file; The first sending unit is further configured to obtain the stored file according to the storage location through the target data center and send the file to the terminal; The preset data center includes a first data center and a second data center; the first data center and the second data center each include a plurality of different racks; the plurality of different racks respectively include a plurality of different data nodes; the distance value between the data nodes of different racks in the first data center is set to a first distance value; the distance value between the data nodes of different racks in the second data center is set to a second distance value; The first determining unit is further configured to obtain a data node of the first data center and a data node of the second data center; determine a storage location of the file based on the first distance value, the second distance value, and the data node of the first data center and the data node of the second data center, and send the location to the terminal; The first determination unit is further configured to compare the distances between the first data center and the second data center and the terminal respectively; determine a comparison result; determine the nearest data center based on the comparison result; and use the nearest data center as the target data center.

9. A server, characterized in that: The server includes: A first memory, configured to store executable instructions; The first processor is configured to implement the method according to any one of claims 1 to 7 when executing the executable instructions stored in the first memory.

10. A computer-readable storage medium, characterized in that The storage medium stores executable instructions, which, when executed, cause the first processor to execute the data storage method according to any one of claims 1 to 7.

Images