Solutions to insufficient disk space in object storage

Abstract

This invention provides a solution to disk space shortage based on object storage, comprising the following steps: dynamic disk expansion by adding data nodes until an odd number of data nodes are reached; building an NFS system using physical machines; interconnection between HBase and NFS; and data synchronization between the two NFS servers. The advantages of this invention are: cost savings, avoiding expensive physical machines; the ability to use older or inexpensive physical machines to run the NFS system; high data security, as this method does not compromise data security; dynamic expansion; and simple cold backup, as long as the data directory remains consistent, data services can be provided externally.

Description

Technical Field

[0001] This invention belongs to the field of storage technology, and in particular relates to a solution for insufficient disk space based on object storage. Background Technology

[0002] The distributed file system (Ceph) utilizes public open-source technologies to implement data storage and retrieval functions. For example, in the invention patent application number 2018107335697, a data protocol module interface manages the objects to be stored, enabling the use of the Hadoop file system. This achieves support for Hadoop and object storage services on CephFS, allowing users to access data stored in the object storage service through the data protocol module interface. Its object storage primarily focuses on the implementation between CephFS and Hadoop. While Hadoop clusters can guarantee data security, they inevitably suffer from data redundancy, thus requiring optimization to address disk space shortages. Summary of the Invention

[0003] In view of this, the present invention aims to overcome the shortcomings of the above-mentioned problems in the prior art and propose a solution to the problem of insufficient disk space based on object storage.

[0004] To achieve the above objectives, the technical solution of the present invention is implemented as follows:

[0005] The solution to insufficient disk space based on object storage includes the following steps:

[0006] S1. Dynamic disk expansion adds data nodes and eventually reaches an odd number of data nodes;

[0007] S2. Build an NFS system using physical machines;

[0008] Interoperability between S3.HBase and NFS;

[0009] S4. The two NFS servers synchronize data.

[0010] Furthermore, in step S1, the dynamic disk expansion process is as follows:

[0011] S11. Add a new machine environment configuration, and configure it as follows:

[0012] S111. The operating system uses Cloudera Manager;

[0013] S112. Use RAID as the namenode's storage to protect metadata;

[0014] S113. Mount the disk to a different directory;

[0015] S114. Deploy the nodes in the same network segment;

[0016] S115. Modify hostname and mapping;

[0017] S116. Set the start level to 3;

[0018] S117. Start the firewall and SELinux;

[0019] S118. Configure the number of system files that can be opened and the maximum number of user processes;

[0020] S119. Configure NTP service to synchronize the clock of the newly added node with the clock of the cluster host;

[0021] S1110. Configure SSH to enable passwordless SSH login with the host;

[0022] S1111. Install JDK and configure environment variables;

[0023] Install dependencies using S1112.yum;

[0024] S1113. Add a new node to replicate the MySQL driver package;

[0025] S1114. Create a cloudera-scm user on the new node;

[0026] S1115. Create a Parcel package distribution directory on the newly added node;

[0027] S1116. Disable the large-area compression command and add it to the / etc / rc.local script;

[0028] Set the S1117.swappiness parameter to 0;

[0029] S1118. Pay attention to Python script issues to avoid exceptions during Parcel package distribution;

[0030] S12. Copy the cloudera-manager directory of the node to the new node;

[0031] S13. Configure and start the Cloudera Manager Agent, and the process is as follows:

[0032] S131. Create the agent's running directory on all nodes;

[0033] S132. Copy the Cloudera Manager Agent startup script to the / etc / init.d / directory;

[0034] S133. Configure and start the Cloudera Manager Agent;

[0035] S14. Add new machines and services on the Cloudera Manager web management page of the host node;

[0036] S15. After the new cluster stabilizes, perform a load balancing process.

[0037] Furthermore, in step S3, the interconnection process between HBase and NFS is as follows: First, NFS needs to be mounted on the data nodes deployed in HBase. Second, the directory address that has been migrated to NFS is updated in the HBase table. Finally, the data is viewed and verified. Ultimately, the client obtains the NFS path through the HBase API and retrieves the actual data based on the obtained path.

[0038] Furthermore, in step S4, the two NFS servers serve as cold backups for each other and perform data synchronization.

[0039] Compared with the prior art, the present invention has the following advantages:

[0040] The disk space shortage solution based on object storage described in this invention saves costs and avoids expensive physical machines. It can use older or inexpensive physical machines equipped with an NFS system; data security is high, and this method does not compromise data security; it allows for dynamic expansion; and data cold backup is simple—as long as the data directory remains consistent, data services can be provided externally. Attached Figure Description

[0041] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an undue limitation of the invention. In the drawings:

[0042] Figure 1 This is a service architecture diagram of object storage as described in an embodiment of the present invention;

[0043] Figure 2 This is a logic diagram of the NFS system described in an embodiment of the present invention. Detailed Implementation

[0044] It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other.

[0045] The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0046] As shown in the figure, the solution to insufficient disk space based on object storage includes the following steps:

[0047] S1. Dynamic disk expansion adds data nodes, eventually reaching an odd number of data nodes. For example, if the cluster has five data nodes, then the number of new data nodes will be 2N (2, 4, etc.).

[0048] S2. Build an NFS system using physical machines;

[0049] Interoperability between S3.HBase and NFS;

[0050] S4. Data is synchronized between the two NFS servers to ensure data security.

[0051] In step S1, the dynamic disk expansion process is as follows:

[0052] S11. Add a new machine environment configuration, and configure it as follows:

[0053] S111. The operating system uses Cloudera Manager;

[0054] S112. RAID (Redundant Array of Independent Disks) is used as the namenode's storage to protect metadata. However, using RAID as the datanode's storage device does not benefit HDFS; the inter-node data replication technology provided by HDFS already meets data backup needs, eliminating the need for RAID redundancy. Furthermore, although RAID striping (RAID 0) is widely used to improve performance, its speed is still slower than the JBOD (Just a Bunch of Disks) configuration used in HDFS.

[0055] JBOD cycles through all disks to schedule HDFS blocks; RAID 0 read and write operations are limited by the speed of the slowest disk in the array, while JBOD disk operations are independent, resulting in an average read and write speed higher than the slowest disk; it is important to emphasize that the performance of each disk can vary considerably in actual use, even for disks of the same model; finally, if a disk in a JBOD configuration fails, HDFS can ignore that disk and continue operating; however, the failure of a disk in a RAID array will render the entire array unusable, thus causing the corresponding node to fail.

[0056] S113. Mount the disk to a different directory;

[0057] S114. Deploy the nodes in the same network segment;

[0058] S115. Modify hostname and mapping;

[0059] S116. Set the start level to 3;

[0060] S117. Start the firewall and SELinux;

[0061] S118. Configure the number of system files that can be opened and the maximum number of user processes;

[0062] Edit the file ` / etc / security / limits.conf`.

[0063] * soft nofile 65536

[0064] * hard nofile 65536

[0065] * soft nproc 16384

[0066] * hard nproc 16384

[0067] S119. Configure NTP service to synchronize the clock of the newly added node with the clock of the cluster host;

[0068] S1110. Configure SSH to enable passwordless SSH login with the host;

[0069] S1111. Install JDK and configure environment variables;

[0070] Install dependencies using S1112.yum;

[0071] S1113. Add a new node to replicate the MySQL driver package;

[0072] S1114. Create a cloudera-scm user on the new node;

[0073] S1115. Create a Parcel package distribution directory on the newly added node;

[0074] S1116. Disable the large-area compression command and add it to the / etc / rc.local script;

[0075] Set the S1117.swappiness parameter to 0;

[0076] Modify the value of swappiness. See the following example for guidance:

[0077] 1) Temporary settings (will be lost after restart)

[0078] Query command #: sysctl -a | grep vm.swappiness

[0079] The result shows: vm.swappiness = 30

[0080] Temporary settings # :echo 10 > / proc / sys / vm / swappiness

[0081] The query command is: # sysctl -a | grep vm.swappiness

[0082] The result shows # vm.swappiness = 10

[0083] Note: You must log in as the root user;

[0084] The following methods are available:

[0085] #: sysctl -w vm.swappiness=10

[0086] vm.swappiness = 10

[0087] #: cat / proc / sys / vm / swappiness

[0088] Result: 10

[0089] 2) Permanent settings

[0090] Edit ` / etc / sysctl.conf` and add the following parameter (if it exists).

[0091] vm.swappiness = 10;

[0092] Then restart the machine or execute the command `#:sysctl -p` to make the changes take effect. In the actual production environment, `vm.swappiness = 0` is set.

[0093] S1118. Pay attention to Python script issues to avoid exceptions during Parcel distribution; the best solution is to directly replace the script on the new machine with the already processed script using scp, so that Parcel distribution exceptions will not occur during subsequent installations.

[0094] S12. Copy the cloudera-manager directory from the node to the new node; the method is to copy the cloudera-manager file on the slave machine to the / opt / directory on the new machine using scp;

[0095] S13. Configure and start the Cloudera Manager Agent, and the process is as follows:

[0096] S131. Create the agent's running directory on all nodes;

[0097] S132. Copy the Cloudera Manager Agent startup script to the / etc / init.d / directory;

[0098] S133. Configure and start the Cloudera Manager Agent; Here, starting the Agent may fail for various reasons, so it is necessary to check the log file to locate the error information and fix it;

[0099] S14. Add new machines and services on the Cloudera Manager web management page of the host node;

[0100] S15. After the new cluster stabilizes, perform load balancing. After successful machine expansion and deployment of component services, observe the cluster. If the cluster is stable after a period of observation, perform cluster load balancing.

[0101] In step S3, the interconnection process between HBase and NFS is as follows: First, NFS needs to be mounted on the data nodes deployed in HBase. Second, the directory address that has been migrated to NFS is updated in the HBase table. Finally, the data is viewed and verified. Ultimately, the client obtains the NFS path through the HBase API and retrieves the actual data based on the obtained path.

[0102] In step S4, the two NFS servers act as cold backups for each other and synchronize data to ensure data security. In this embodiment, if one machine fails, the other can take over immediately without data loss; the two NFS servers act as cold backups for each other. In addition, there are storage rule restrictions. The data in NFS is simply a snappy compressed file stream. To obtain the actual data, rules based on business requirements, time, rowkey, file type, etc., are required.

[0103] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A solution to disk space shortage based on object storage, characterized in that, Includes the following steps: S1. Dynamic disk expansion adds data nodes and eventually reaches an odd number of data nodes; S2. Build an NFS system using physical machines; The interconnection between S3.HBase and NFS is achieved through the following process: First, NFS needs to be mounted on the data nodes deployed in HBase. Second, the directory address that has been migrated to NFS is updated in the HBase table. Finally, the data is viewed and verified. Ultimately, the client obtains the NFS path through the HBase API and retrieves the actual data based on the obtained path. S4. The two NFS servers synchronize data.

2. The solution to insufficient disk space based on object storage according to claim 1, characterized in that, In step S1, the dynamic disk expansion process is as follows: S11. Add a new machine environment configuration, and configure it as follows: S111. The operating system uses Cloudera Manager; S112. Use RAID as the namenode's storage to protect metadata; S113. Mount the disk to a different directory; S114. Deploy the nodes in the same network segment; S115. Modify hostname and mapping; S116. Set the start level to 3; S117. Start the firewall and SELinux; S118. Configure the number of system files that can be opened and the maximum number of user processes; S119. Configure NTP service to synchronize the clock of the newly added node with the clock of the cluster host; S1110. Configure SSH to enable passwordless SSH login with the host; S1111. Install JDK and configure environment variables; Install dependencies using S1112.yum; S1113. Add a new node to replicate the MySQL driver package; S1114. Create a cloudera-scm user on the new node; S1115. Create a Parcel package distribution directory on the newly added node; S1116. Disable the large-area compression command and add it to the / etc / rc.local script; Set the S1117.swappiness parameter to 0; S1118. Pay attention to Python script issues to avoid exceptions during Parcel package distribution; S12. Copy the cloudera-manager directory of the node to the new node; S13. Configure and start the Cloudera Manager Agent, and the process is as follows: S131. Create the agent's running directory on all nodes; S132. Copy the Cloudera Manager Agent startup script to the / etc / init.d / directory; S133. Configure and start Cloudera Manager Agent; S14. Add new machines and services on the Cloudera Manager web management page of the host node; S15. After the new cluster stabilizes, perform a load balancing process.

3. The solution to insufficient disk space based on object storage according to claim 1, characterized in that: In step S4, the two NFS servers serve as cold backups for each other and synchronize data.

Images