A cloud storage outbound traffic control method and system based on FUSE technology

Abstract

The present invention provides a cloud storage outbound traffic control method and system based on FUSE technology. Cloud storage files are directly mapped to the local disk of a file server through FUSE technology. When users access the file server, there is no need to read or write to the local disk of the file server, and the actual storage of the file server is not occupied. When the outbound traffic of the file server reaches a threshold, it can quickly switch to a new file server.

Description

Technical Field

[0001] The present invention belongs to the technical field of cloud computing and relates to a cloud storage outbound traffic control method and system based on FUSE technology. Background Art

[0002] In IT systems, content that needs to be stored, such as images, music, and videos, is typically stored directly in cloud storage. A typical example is AWS's Simple Storage Service, commonly referred to as S3 in the industry. When users need to view images, listen to music, or watch videos, they must download the corresponding files from cloud storage. File downloads generate outbound network traffic, and cloud vendors typically bill based on this outbound traffic.

[0003] Currently, outbound traffic is mainly controlled through the following methods, thereby reducing the costs generated by some outbound traffic.

[0004] 1. Use the CDN provided by the cloud vendor

[0005] For example, AWS provides a CDN called CloudFront. While the cost of data flowing from S3 to CloudFront is zero, the cost of data flowing out of CloudFront does not significantly decrease. The primary function of a CDN is to increase user access speeds.

[0006] 2. Use a third-party CDN

[0007] CloudFlare, the most widely used CDN overseas, offers 2TB of free outbound traffic per day. Within this free limit, users ingesting content from CloudFlare is free, but outbound traffic from AWS S3 to CloudFlare is still billed. This means that data flowing from cloud storage to CloudFlare is still billed based on traffic volume, and CloudFlare only provides 2TB of free traffic per day when exporting data to users; any traffic exceeding this limit is still billed.

[0008] 3. Use CDN + Lightsail

[0009] Lightsail is a virtual machine service that provides AWS with very cheap outbound traffic. Its outbound traffic charges are much lower than those of cloud storage.

[0010] Currently, there are practices for storing files directly on servers. However, Lightsail disks cannot be dynamically scaled, requiring servers with large disk capacities. However, these high-capacity servers are expensive, and their maximum capacity is only 640GB, which is insufficient for most scenarios. Furthermore, when traffic capacity needs to be expanded, storage costs will increase exponentially.

[0011] 4. Self-encode, dynamically request files from S3 and output them to the requesting end

[0012] S3 has a comprehensive API that supports custom coding for file forwarding, including file list display, paging, and query functionality, as well as the ability to read files from local disk and output them to the requesting client. Meanwhile, Lightsail shares CPU and disk, resulting in relatively low disk performance. However, custom coding is complex and requires significant development effort. Files must be read, written, and forwarded on the same machine, but the performance of Lightsail servers is relatively weak, reducing file retrieval performance. This significantly increases system complexity to maintain stable program operation. Summary of the Invention

[0013] The present invention provides a cloud storage outbound traffic control method based on FUSE technology. When the outbound traffic of a file server reaches a threshold, a new file server can be quickly switched to.

[0014] The present invention provides a method for controlling cloud storage outbound traffic based on FUSE technology, comprising the following steps:

[0015] (1) Install a Docker container on the file server and pull the cloud storage image. That is, mount the directory of the cloud storage server to the internal directory of the Docker container. The Docker container runs a file system based on FUSE technology.

[0016] (2) Create a mount point directory on the file server;

[0017] (3) Run the Docker image and mount the mount point directory to the internal directory of the Docker container;

[0018] (4) Install Nginx on the file server. The server with Nginx installed is the file server;

[0019] (5) Monitor the outgoing traffic of the file server. If the outgoing traffic reaches a set threshold, create a new file server according to steps (1) to (4).

[0020] This invention uses FUSE technology to directly map cloud storage files to the local disk of the file server. This eliminates the need for users to read or write to the local disk of the file server when accessing the file, and does not occupy the actual storage of the file server. When the outbound network traffic of the file server reaches a threshold, the file server can be quickly switched to a new file server. Since files do not need to be stored on the file server, dynamic scaling of the file server's capacity is very convenient.

[0021] Preferably, the file server created according to steps (1) to (4) is made into a system image. When the outgoing network traffic reaches a set threshold, a new file server is created by reusing the system image, which greatly facilitates the creation of a new file server.

[0022] Preferably, after the new file server is used normally, the old file server is deleted.

[0023] Preferably, after the file server is created, the DNS settings of the server accessing the file server are adjusted, and the IP address of the file server is added to the server accessing the file server.

[0024] Preferably, the cloud storage is AWS S3, the file system running on the Docker container is S3FS, and the file server is a Lightsail server. Currently, the outbound fee for Lightsail is 2.5 USD / TB, while the direct outbound fee for AWS S3 is 90 USD / TB, a price reduction of approximately 97%, which can significantly reduce traffic costs.

[0025] The present invention also provides a cloud storage outbound traffic control system based on FUSE technology, which is used to connect to a cloud storage server, including a file server and a monitoring and scheduling system; the file server is created in the following manner: a Docker container is installed on the file server, and a cloud storage image is pulled, that is, a directory of the cloud storage server is mounted to an internal directory of the Docker container, a file system based on FUSE technology runs on the Docker container, a mount point directory is created on the file server, the Docker image is run, the mount point directory is mounted to the internal directory of the Docker container, and Nginx is installed on the file server; the monitoring and scheduling system is used to monitor the outbound traffic of the file server, and when the outbound traffic reaches a set threshold, a new file server is created.

[0026] Preferably, the constructed file server is made into a system image, and when the outbound network traffic reaches a set threshold, the monitoring and scheduling system creates a new file server by reusing the system image.

[0027] Preferably, the monitoring and scheduling system is also used to access the DNS settings of the server of the file server, and add the IP address of the file server to the server of the file server.

[0028] Preferably, when the new file server is used normally, the monitoring and scheduling system deletes the old file server.

[0029] Preferably, the cloud storage is AWS S3, the file system running on the Docker container is S3FS, and the file server is a Lightsail server.

[0030] The present invention directly maps cloud storage files to the local disk of the file server through FUSE technology. When users access the file server, there is no need to read and write the local disk of the file server, and the actual storage of the file server is not occupied. When the outbound traffic of the file server reaches the threshold, it can quickly switch to a new file server. Since the files do not need to be actually stored on the file server, it is very convenient to dynamically scale the capacity of the file server. Using CloudFlare for file forwarding service, the files of the cloud storage service are forwarded to the user. There is no need to develop additional file forwarding services, and the impact on performance is small. Using existing mature technologies, including FUSE, Docker, Nginx, etc., the system has high stability. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 This is a diagram of the control system architecture of the present invention;

[0032] Figure 2 This is a diagram of the file server architecture of the present invention;

[0033] Figure 3 This is a flow chart of the control method of the present invention. DETAILED DESCRIPTION

[0034] Cloud storage is a model of online storage that stores data on multiple virtual servers, typically hosted by a third party, rather than dedicated servers. Hosting companies operate large data centers, and those who need data storage can meet their data storage needs by purchasing or leasing storage space from them. Data center operators prepare virtualized storage resources on the back end based on customer needs and provide them as storage pools. Customers can then use these storage pools to store their files or objects. In reality, these resources may be distributed across numerous server hosts.

[0035] FUSE: Filesystem in Userspace, is a module in Linux used to mount certain network spaces, such as SSH, to the local file system.

[0036] S3FS: It is a FUSE-based file system that allows Linux and Mac OS X to mount S3 buckets on the local file system. S3FS can maintain the original format of the object. S3FS is a large subset of POSIX, including reading / writing files, directories, symbolic links, modes, uid / gid and extended attributes. It is compatible with Amazon S3, Google Cloud Storage and other S3-based object storage.

[0037] Docker is an open-source application container engine that allows developers to package their applications and dependencies into a portable image and then publish it to any machine running a popular Linux or Windows operating system, also enabling virtualization. Containers are completely sandboxed, with no interfaces between them.

[0038] Nginx is an HTTP and reverse proxy web server.

[0039] Cloudflare's primary business is providing customers with website security management, performance optimization, and related technical support. Through technologies such as its reverse proxy-based Content Delivery Network (CDN), Anycast technology, its Nginx+Lua-based Web Application Firewall (WAF), and its Distributed Domain Name Server (DDNS), Cloudflare can help protected sites defend against most network attacks, including Distributed Denial of Service (DDoS) attacks, ensuring the site's long-term online presence while also improving site performance and access speed for a better visitor experience.

[0040] The present invention sets up a file server and CloudFlare between the cloud storage server and the user. The file server is installed with a Docker container and Nginx to map the files on the cloud storage server to the file server. Since the file server is built on the cloud server of the cloud storage service provider, there is no outbound traffic when the files on the cloud storage server are mapped to the file server, and the user only needs to pay the fee for renting the server space. As for the outbound traffic from the file server to CloudFlare, the cloud storage service provider provides free or preferential quotas. As long as the outbound traffic from the file server to CloudFlare is well controlled, the traffic fee can be controlled. By providing a service of delivering file server data to users through CloudFlare, the outbound traffic fee charged by CloudFlare is relatively low. When the outbound traffic from the file server to CloudFlare is about to exceed the free quota or preferential quota, a new file server is built and used, and a new free quota or preferential quota can be obtained, thereby greatly saving costs.

[0041] The control method of the present invention comprises the following steps:

[0042] (1) Install the Docker container on the server, pull the cloud storage image, and mount the cloud storage server directory to the internal directory of the Docker container. The Docker container runs a file system based on FUSE technology.

[0043] (2) Create a mountpoint directory on the server.

[0044] (3) Run the Docker image and match the mount point directory created on the server with the file directory on the corresponding cloud storage, that is, mount the mount point directory to the internal directory of the Docker container.

[0045] (4) Install and configure Nginx on the server, and use the server with Nginx installed as a file server. The host directory (i.e., the mount point directory) is just an ordinary local disk for Nginx. Nginx can directly read files and output them to CloudFlare via the HTTP protocol.

[0046] (5) The file server built in steps 1-4 is made into a system image so that when the free outbound traffic or preferential traffic is about to be exceeded, the system image can be reused to create a new file server.

[0047] (6) Adjusting CloudFlare's DNS settings so that CloudFlare can access the file server. The present invention has a monitoring and scheduling system that monitors the outbound traffic of the file server in real time. When the outbound traffic is about to exceed the free quota or preferential quota provided by the cloud storage service provider, a new file server is created, CloudFlare's DNS settings are adjusted, the IP address of the new file server is added to CloudFlare, and after confirming that the new file server is working properly, the old file server is deleted.

[0048] Through FUSE technology, cloud storage files can be directly mapped to local disks without the need for direct disk reading and writing. CloudFlare actually accesses files directly from cloud storage without occupying the actual storage of the file server.

[0049] The following uses AWS S3 as an example to explain in detail the cloud storage outbound traffic control method and control system based on FUSE technology provided by the present invention. Data is stored on the cloud storage AWS S3, which uses the S3FS file system based on FUSE technology.

[0050] like Figure 1 、 2 As shown, the control system includes a Lightsail server as a file server. The Lightsail server is provided by AWS. A Docker container is installed on the Lightsail server, which runs the S3FS file system. The Lightsail server also has Nginx installed. The Lightsail server's related file directories are ordinary local disks for Nginx. Nginx can directly read files and output them to CloudFlare through the HTTP protocol, and then CloudFlare outputs them to users. The control system also includes a monitoring and scheduling system, which monitors the outbound traffic of the file server in real time. When the outbound traffic reaches a set threshold (generally set to almost reach the free outbound traffic quota provided by AWS), the monitoring and scheduling system will create a new file server and modify CloudFlare's DNS settings so that CloudFlare can access the new file server. After the new Lightsail server is running normally, the old Lightsail server will be deleted.

[0051] like Figure 3 As shown, the control method of the present invention is as follows:

[0052] 1. Install a Docker container on the Lightsail server and pull the S3FS image. Then, run the S3FS file system on the Docker container. Use the S3FS file system to mount the AWS S3 bucket to the / opt / s3fs / bucket directory within the Docker container.

[0053] The sample code is as follows:

[0054] Bash

[0055] #Get the installation script

[0056] curl-fsSL https: / / get.docker.com-o get-docker.sh

[0057] #Execute the installation script

[0058] sudo sh get-docker.sh

[0059] #Pull the corresponding image

[0060] docker pull efrecon / s3fs:1.91

[0061] 2. Create a mountpoint directory on the Lightsail server.

[0062] The sample code is as follows:

[0063] Bash

[0064] mkdir -p / home / ubuntu / s3dir

[0065] 3. Run the Docker image on the Lightsail server and map the local file directory " / home / ubuntu / s3dir" to the corresponding AWS S3 Bucket. That is, through the Docker container, mount the host directory (i.e., the mount point directory) " / home / ubuntu / s3dir" to the internal directory " / opt / s3fs / bucket" of the Docker container.

[0066] Assuming the AWS S3 Bucket is named xxxx, the sample code is as follows:

[0067]

[0068] For cloud storage from other cloud vendors, similar FUSE technology is also supported to map cloud storage to local directories.

[0069] 4. Install and configure Nginx on the Lightsail server, using it as a file server. The host directory (i.e., the mount point directory) appears to Nginx as a regular local disk, allowing it to directly read files and export them to CloudFlare via HTTP.

[0070] The installation command is as follows:

[0071] Bash

[0072] sudo apt install -y nginx

[0073] The key configurations are as follows:

[0074]

[0075]

[0076] Assume that the domain name is files.test.com, and the directory of the file to be accessed in the bucket is bucket_name / dir1 / dir2 / filename.jpg. The actual access path is: files.test.com / dir1 / dir2 / filename.jpg.

[0077] 5. After completing steps 1-4, create a system image of the Lightsail server you've built. This allows you to reuse the system image to create a new Lightsail server when your free or discounted outbound traffic is about to be exceeded. This step can use the system service of your cloud storage service provider and does not require additional development.

[0078] 6. The monitoring and scheduling system adjusts CloudFlare's DNS settings and adds the Lightsail server's IP address to CloudFlare, allowing CloudFlare to access the Lightsail server. The monitoring and scheduling system monitors the Lightsail server's outbound traffic in real time. If the set threshold is reached, a new Lightsail server is created and its IP address is added to CloudFlare. After confirming that the new Lightsail server is functioning properly, the old Lightsail server is deleted. CloudFlare is a commonly used file forwarding server platform, and forwarding can also be performed from other existing servers.

[0079] The above is an explanation using AWS as an example. The solution provided by the present invention can also be applied to cloud storage services such as Alibaba Cloud and Tencent Cloud. Alibaba Cloud and Tencent Cloud provide "lightweight servers" for use as file servers. The Docker container, Nginx, etc. of the present invention can be deployed on lightweight servers.

Claims

1. A cloud storage outbound traffic control method based on FUSE technology, characterized in that: The following steps are involved: (1) Install a Docker container on the file server and pull the cloud storage image. That is, mount the directory of the cloud storage server to the internal directory of the Docker container. The Docker container runs a file system based on FUSE technology. (2) Create a mount point directory on the file server; (3) Run the Docker image and mount the mount point directory to the internal directory of the Docker container; (4) Install Nginx on the file server; (5) Monitor the outgoing traffic of the file server. If the outgoing traffic reaches a set threshold, create a new file server according to steps (1) to (4).

2. The cloud storage outbound traffic control method based on FUSE technology according to claim 1, characterized in that: The file server created according to steps (1) to (4) is made into a system image. When the outgoing network traffic reaches the set threshold, a new file server is created by reusing the system image.

3. The cloud storage outbound traffic control method based on FUSE technology according to claim 1 or 2, characterized in that: After the new file server is in normal use, delete the old file server.

4. The cloud storage outbound traffic control method based on FUSE technology according to claim 1 or 2, characterized in that: After the file server is created, adjust the DNS settings of the server that accesses the file server and add the file server's IP address to the server that accesses the file server.

5. The cloud storage outbound traffic control method based on FUSE technology according to claim 1 or 2, characterized in that: The cloud storage is AWS S3, the file system running on the Docker container is S3FS, and the file server is a Lightsail server.

6. A cloud storage outbound traffic control system based on FUSE technology, used to connect to a cloud storage server, characterized in that: The system includes a file server and a monitoring and scheduling system; the file server is created by installing a Docker container on the file server and pulling a cloud storage image, that is, mounting a directory of the cloud storage server to an internal directory of the Docker container, running a file system based on FUSE technology on the Docker container, creating a mount point directory on the file server, running the Docker image, mounting the mount point directory to an internal directory of the Docker container, and installing Nginx on the file server; the monitoring and scheduling system is used to monitor the outgoing network traffic of the file server, and when the outgoing network traffic reaches a set threshold, a new file server is created.

7. The cloud storage outbound traffic control system based on FUSE technology according to claim 6, characterized in that: The constructed file server is made into a system image. When the outbound network traffic reaches a set threshold, the monitoring and scheduling system creates a new file server by reusing the system image.

8. The cloud storage outbound traffic control system based on FUSE technology according to claim 6 or 7, characterized in that: The monitoring and scheduling system is also used to access the DNS settings of the server of the file server, and add the IP address of the file server to the server of the file server.

9. The cloud storage outbound traffic control system based on FUSE technology according to claim 6 or 7, characterized in that: When the new file server is in normal use, the monitoring and scheduling system deletes the old file server.

10. The cloud storage outbound traffic control system based on FUSE technology according to claim 6 or 7, characterized in that: The cloud storage is AWS S3, the file system running on the Docker container is S3FS, and the file server is a Lightsail server.

Images