Cloud rendering platform storage implementation method based on S3 storage protocol

Abstract

The invention discloses a cloud rendering platform storage implementation method based on an S3 storage protocol. The method comprises the following steps that S1, a user completes registration on a cloud rendering platform; s2, using an administrator account to create an AK and SC key pair stored in S3 for a new user; s3, the cloud rendering platform database stores the user key pair and creates a unique S3 storage bucket for the user; s4, setting a file life cycle for a file corresponding to the storage bucket in the S3 according to the user storage cleaning configuration; s5, the user assets upload or render the output file; s6, uploading to an S3 storage bucket of a corresponding user through an S3 storage protocol API; s7, the user files are backed up on different storage nodes; s8, the user performs file interaction operation through cloud rendering asset management; and S9, when the user file expires, triggering life cycle management and automatic cleaning. According to the invention, the storage efficiency is improved, the cost is reduced, and the data security is protected.

Description

Technical Field

[0001] The present invention relates to the field of storage technology, and in particular to a cloud rendering platform storage implementation method based on the S3 storage protocol. Background Art

[0002] With the rapid development of cloud computing and big data, cloud rendering platforms, as a new type of rendering solution, are gradually replacing traditional local rendering. Cloud rendering platforms use the powerful computing power and large-scale data storage capabilities of cloud computing to achieve high-quality rendering results and fast rendering speeds. In cloud rendering platforms, the storage part plays an important role and affects the efficiency of data reading and processing. Therefore, the selection and implementation of storage technology are crucial to the overall performance of cloud rendering platforms.

[0003] Currently, the commonly used cloud rendering platform storage implementation methods include local storage and third-party storage services. The local storage method stores data on the physical server where the cloud rendering platform is located. This method is not only costly, but also has limited storage capacity and cannot meet the storage needs of large-scale data. Another way is to use third-party storage services, such as Amazon S3 (Simple Storage Service). Amazon S3 is a highly scalable, secure and reliable cloud object storage service that provides a simple and cost-effective data storage solution.

[0004] However, there are some problems with the existing cloud rendering platform storage implementation. First, the local storage method cannot meet the needs of large-scale data processing due to its limited storage capacity, resulting in low storage efficiency. Second, third-party storage services usually require payment, which is costly and not tightly integrated with the cloud rendering platform, resulting in slow data reading and writing speeds. In addition, data security is also a concern because the stored data often contains important information and business secrets of users.

[0005] Therefore, the prior art has defects and needs to be improved. Summary of the invention

[0006] The technical problem to be solved by the present invention is to provide a cloud rendering platform storage implementation method based on the S3 storage protocol to improve storage efficiency, reduce costs and protect data security.

[0007] The technical solution of the present invention is as follows: a cloud rendering platform storage implementation method based on the S3 storage protocol is provided, comprising the following steps.

[0008] S1: The user completes registration on the cloud rendering platform.

[0009] S2: Use the administrator account to create an AK and SC key pair for the new user in S3 storage.

[0010] S3: The Cloud Rendering Platform database saves the user key pair and creates a unique S3 bucket for the user.

[0011] S4: Set the file lifecycle for the corresponding S3 bucket file based on the user storage cleanup configuration.

[0012] S5: User asset upload or rendering output file.

[0013] S6: Upload to the S3 storage bucket of the corresponding user through the S3 storage protocol API.

[0014] S7: Back up user files on different storage nodes.

[0015] S8: Users perform file interaction operations through cloud rendering asset management.

[0016] S9: User files expire, triggering automatic cleanup of lifecycle management.

[0017] The file interaction operations include: adding, deleting, modifying, querying, and downloading.

[0018] Use the S3 storage protocol to implement basic interactive functions; specifically include the following aspects.

[0019] Use the API provided by the S3 storage protocol to manage and operate stored data, such as creating users, creating buckets, uploading files, downloading files, deleting files, modifying files, retrieving files, etc.

[0020] Bucket management: Implement operations such as bucket creation, deletion, and query; users can use the standard interface and methods of the S3 storage protocol to create buckets by specifying bucket names and corresponding permission configurations. At the same time, users can also query bucket metadata and properties, and delete buckets that are no longer needed as needed.

[0021] Object operations: read, write, and delete objects in a bucket. Users can use the API provided by the S3 storage protocol to upload objects to a bucket by specifying the object name and bucket name. Users can also download objects from a bucket and delete objects that are no longer needed.

[0022] Access control: The access control mechanism provided by the S3 storage protocol is used to control access to buckets and objects. Users can set permission levels for buckets and objects, including public read and write, private access, etc., to ensure data security and privacy protection.

[0023] Metadata management: Implement metadata management for buckets and objects; users can set and update metadata for buckets and objects through methods provided by the S3 storage protocol, including file size, creation time, last modification time, etc. The management of these metadata helps to better organize and manage stored data.

[0024] Error handling and logging: When implementing basic interactive functions, consider error handling and logging. For errors and exceptions that occur, corresponding error prompts and error codes should be provided to facilitate users to troubleshoot and locate problems. At the same time, use the logging system to record logs of bucket and object operations to better track and audit data access and operations.

[0025] In step S7, the backup specifically includes: regularly performing multi-node backup processing on the user's asset files on the storage to avoid asset losses of some users caused by an abnormality of a certain storage node, thereby improving storage reliability and disaster tolerance.

[0026] Regular backup: Set up regular backup strategies based on demand and data importance. Through regular backup, data in the storage system is copied to other storage nodes or storage areas to ensure data security and recoverability.

[0027] Data redundancy: Data is stored redundantly, that is, multiple copies of data are stored on different storage nodes or storage devices. When a storage node or device fails, the data can be restored from other copies to ensure data availability and durability.

[0028] Backup strategy management: To facilitate the management and control of backup strategies, a backup strategy management system can be set up, and backup plan scheduling and monitoring functions can be provided. Administrators can adjust backup plans as needed and monitor backup status and progress.

[0029] The cloud rendering platform applies for a unique account and storage bucket for each user assigned to S3 storage, obtains the user's unique AK and SC key pair (access_key and secret_key), saves it to the cloud rendering platform database and associates it with the corresponding user, and isolates the user's asset files through a unique storage bucket, configures a specific permission range, and only allows access to the specified user storage bucket through the corresponding user's key pair, ensuring that assets between users cannot be accessed and called by each other, ensuring the security and privacy of S3 storage.

[0030] Optimizing S3 storage transmission efficiency includes at least one of the following methods.

[0031] Multi-threaded transmission: Transmitting data in parallel through multiple threads can improve transmission efficiency. Slicing large files or data into multiple small blocks and using multiple threads to transmit these small blocks simultaneously can effectively reduce transmission time and improve transmission efficiency.

[0032] Intelligent storage node selection: Intelligently select the best storage node for data transmission based on network conditions and node load conditions; by measuring the delay and response time between storage nodes, the nearest node with the shortest response time can be dynamically selected for transmission, thereby reducing network congestion and transmission delays and improving data transmission efficiency.

[0033] Efficient compression and acceleration algorithms: Using efficient compression algorithms (compression algorithms based on multi-layer perceptrons, compression algorithms based on layered neural networks, compression algorithms based on generative adversarial networks (GANs), etc.) to compress data into a smaller size can reduce the bandwidth occupied by data transmission and improve transmission efficiency. In addition, acceleration algorithms, such as parallel algorithms or accelerators, can also be applied to speed up the data transmission process and improve the efficiency of the entire transmission process.

[0034] Cache and pre-fetch technology: By setting up cache on local storage devices, data reading efficiency is improved; cache can save frequently used data locally in advance, reduce the number of visits to remote storage nodes, and speed up reading. At the same time, through pre-fetch technology, the expected data is acquired in advance to the local cache to avoid waiting time and improve data accessibility and transmission efficiency.

[0035] Data sharding and distributed transmission: Slice large files or data into multiple small blocks, and store these small blocks in a distributed manner on multiple storage nodes to achieve simultaneous data transmission and reading operations, reduce transmission time and improve transmission efficiency.

[0036] The S3 storage protocol cleanup implementation method is: configure the bucket cleanup policy by configuring the natively supported rgw lifecycle function. The cleanup policy automatically deletes expired or no longer needed objects according to the set rules and time periods. For example, cleanup operations can be performed based on the creation time, last access time, or storage time of the object; through the rgw lifecycle function, one or more rules can be set for each bucket. The rules can define the life cycle of objects in the bucket according to needs. For example, a rule can be set to automatically delete objects in the bucket 30 days after creation; in the cloud rendering platform, bucket cleanup policies can be easily configured and managed through the management interface or API. Administrators can set different rules and schedules for different buckets according to different needs; according to the rule settings, the storage system will automatically perform cleanup operations. These operations can include deleting expired objects, archiving or migrating data, etc. The cleanup operation can be automatically triggered according to the set schedule, or it can be triggered manually; based on the above storage cleanup mechanism, the user's storage cleanup policy can be personalized and personalized cleanup can be performed for the user's expired asset files.

[0037] The backend service framework of the cloud rendering platform includes: SpringCloud microservices, dubbo service dispatch center, MySQL database, Mongodb database, and Redis cache.

[0038] Use the official S3 storage API interface to implement the interaction between the cloud rendering platform and the S3 storage protocol, and realize basic storage functions such as asset management, user management, declaration cycle management, storage statistics, etc. in the cloud rendering platform.

[0039] The present invention adopts the following technical solutions: 1. Establishing a cloud rendering platform storage system based on the S3 storage protocol, including an S3 storage server, a network and a cloud rendering computing device; 2. Using the S3 storage protocol for large-scale data storage, and performing data reading, writing, updating and deletion operations through the cloud rendering computing device; 3. On the basis of the S3 storage protocol, realizing data redundancy, replication, sharding and fault switching, and improving data availability and durability; 4. Using the access control and encryption mechanism of the S3 storage protocol to enhance data protection and prevent illegal access; 5. Realizing high-speed data transfer and distribution through the network of the cloud rendering platform, reducing network latency, and improving the operating efficiency of the entire cloud rendering platform; 6. On the basis of the S3 storage protocol, realizing statistical management and timed cleaning of expired user data, and reducing the manual operation and maintenance cost of the cloud rendering platform; achieving improved storage efficiency, reduced costs, and data security protection.

[0040] The method of the present invention combines the cloud rendering platform and the S3 storage protocol, making full use of the advantages of the S3 storage protocol, improving storage efficiency, reducing costs, and protecting data security. By adopting the S3 storage protocol to perform data reading, writing, updating, and deleting operations, large-scale data storage can be achieved, and features such as data redundancy, replication, sharding, and fault switching are provided to improve data availability and durability. At the same time, the access control and encryption mechanisms of the S3 storage protocol can be used to protect data security and prevent illegal access and data leakage. Through the network of the cloud rendering platform, high-speed data transfer and distribution can be achieved, network latency can be reduced, and the operating efficiency of the entire cloud rendering platform can be improved. Therefore, this cloud rendering platform storage implementation method based on the S3 storage protocol has important application value and broad market prospects.

[0041] By adopting the above scheme, the present invention provides a cloud rendering platform storage implementation method based on the S3 storage protocol, which has the following technical effects:

[0042] (1) High reliability and availability: By using the S3 storage protocol and data backup strategy, redundant storage and regular backup of data are ensured. Even if a storage node fails or other unexpected situations occur, data can still be restored from other nodes or backups, ensuring data reliability and availability.

[0043] (2) Scalability and elasticity: The distributed architecture based on the S3 storage protocol can easily expand storage capacity and computing resources. The cloud rendering platform storage system can be expanded horizontally and vertically according to demand to adapt to the growing data storage and processing needs. At the same time, the storage system also has the advantages of elasticity and automation. When demand changes, it can automatically adjust resource allocation and network configuration to ensure efficient operation of the system.

[0044] (3) Efficient data storage and transmission: Improve data reading and writing efficiency by optimizing data transmission and storage strategies. Technical measures such as multi-threaded transmission, intelligent node selection, compression and acceleration algorithms can effectively reduce transmission time, improve storage efficiency, and speed up data reading through caching and pre-fetching technologies.

[0045] (4) Data security and privacy protection: The access control mechanism and encryption technology provided by the S3 storage protocol ensure data security and privacy protection. Administrators can flexibly set access permissions for buckets and objects and use encryption algorithms to encrypt data to prevent unauthorized access and data leakage.

[0046] (5) Simplified management and operation: Through standard interfaces and methods based on the S3 storage protocol, the management and operation of buckets and objects are simplified. Users can use unified interfaces and tools to store and process data conveniently and quickly, improving work efficiency and operational convenience. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 The figure is a flow chart of the method of the present invention. DETAILED DESCRIPTION

[0048] The present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments.

[0049] See also Figure 1 The present invention provides a cloud rendering platform storage implementation method based on the S3 storage protocol, comprising the following steps.

[0050] S1: The user completes registration on the cloud rendering platform.

[0051] S2: Use the administrator account to create an AK and SC key pair for the new user in S3 storage.

[0052] S3: The Cloud Rendering Platform database saves the user key pair and creates a unique S3 bucket for the user.

[0053] S4: Set the file lifecycle for the corresponding S3 bucket file based on the user storage cleanup configuration.

[0054] S5: User asset upload or rendering output file.

[0055] S6: Upload to the S3 storage bucket of the corresponding user through the S3 storage protocol API.

[0056] S7: Back up user files on different storage nodes.

[0057] S8: Users perform file interaction operations through cloud rendering asset management.

[0058] S9: User files expire, triggering automatic cleanup of lifecycle management.

[0059] The file interaction operations include: adding, deleting, modifying, querying, and downloading.

[0060] Use the S3 storage protocol to implement basic interactive functions; specifically include the following aspects.

[0061] Use the API provided by the S3 storage protocol to manage and operate stored data, such as creating users, creating buckets, uploading files, downloading files, deleting files, modifying files, retrieving files, etc.

[0062] Bucket management: Implement operations such as bucket creation, deletion, and query; users can use the standard interface and methods of the S3 storage protocol to create buckets by specifying bucket names and corresponding permission configurations. At the same time, users can also query bucket metadata and properties, and delete buckets that are no longer needed as needed.

[0063] Object operations: read, write, and delete objects in a bucket. Users can use the API provided by the S3 storage protocol to upload objects to a bucket by specifying the object name and bucket name. Users can also download objects from a bucket and delete objects that are no longer needed.

[0064] Access control: The access control mechanism provided by the S3 storage protocol is used to control access to buckets and objects. Users can set permission levels for buckets and objects, including public read and write, private access, etc., to ensure data security and privacy protection.

[0065] Metadata management: Implement metadata management for buckets and objects; users can set and update metadata for buckets and objects through methods provided by the S3 storage protocol, including file size, creation time, last modification time, etc. The management of these metadata helps to better organize and manage stored data.

[0066] Error handling and logging: When implementing basic interactive functions, consider error handling and logging. For errors and exceptions that occur, corresponding error prompts and error codes should be provided to facilitate users to troubleshoot and locate problems. At the same time, use the logging system to record logs of bucket and object operations to better track and audit data access and operations.

[0067] In step S7, the backup specifically includes: regularly performing multi-node backup processing on the user's asset files on the storage to avoid asset losses of some users caused by an abnormality of a certain storage node, thereby improving storage reliability and disaster tolerance.

[0068] Regular backup: Set up regular backup strategies based on demand and data importance. Through regular backup, data in the storage system is copied to other storage nodes or storage areas to ensure data security and recoverability.

[0069] Data redundancy: Data is stored redundantly, that is, multiple copies of data are stored on different storage nodes or storage devices. When a storage node or device fails, the data can be restored from other copies to ensure data availability and durability.

[0070] Backup strategy management: To facilitate the management and control of backup strategies, a backup strategy management system can be set up, and backup plan scheduling and monitoring functions can be provided. Administrators can adjust backup plans as needed and monitor backup status and progress.

[0071] The cloud rendering platform applies for a unique account and storage bucket for each user assigned to S3 storage, obtains the user's unique AK and SC key pair (access_key and secret_key), saves it to the cloud rendering platform database and associates it with the corresponding user, and isolates the user's asset files through a unique storage bucket, configures a specific permission range, and only allows access to the specified user storage bucket through the corresponding user's key pair, ensuring that assets between users cannot be accessed and called by each other, ensuring the security and privacy of S3 storage.

[0072] Optimizing S3 storage transmission efficiency includes at least one of the following methods.

[0073] Multi-threaded transmission: Transmitting data in parallel through multiple threads can improve transmission efficiency. Slicing large files or data into multiple small blocks and using multiple threads to transmit these small blocks simultaneously can effectively reduce transmission time and improve transmission efficiency.

[0074] Intelligent storage node selection: Intelligently select the best storage node for data transmission based on network conditions and node load conditions; by measuring the delay and response time between storage nodes, the nearest node with the shortest response time can be dynamically selected for transmission, thereby reducing network congestion and transmission delays and improving data transmission efficiency.

[0075] Efficient compression and acceleration algorithms: Using efficient compression algorithms (compression algorithms based on multi-layer perceptrons, compression algorithms based on layered neural networks, compression algorithms based on generative adversarial networks (GANs), etc.) to compress data into a smaller size can reduce the bandwidth occupied by data transmission and improve transmission efficiency. In addition, acceleration algorithms, such as parallel algorithms or accelerators, can also be applied to speed up the data transmission process and improve the efficiency of the entire transmission process.

[0076] Cache and pre-fetch technology: By setting up cache on local storage devices, data reading efficiency is improved; cache can save frequently used data locally in advance, reduce the number of visits to remote storage nodes, and speed up reading. At the same time, through pre-fetch technology, the expected data is acquired in advance to the local cache to avoid waiting time and improve data accessibility and transmission efficiency.

[0077] Data sharding and distributed transmission: Slice large files or data into multiple small blocks, and store these small blocks in a distributed manner on multiple storage nodes to achieve simultaneous data transmission and reading operations, reduce transmission time and improve transmission efficiency.

[0078] The S3 storage protocol cleanup implementation method is: configure the bucket cleanup policy by configuring the natively supported rgw lifecycle function. The cleanup policy automatically deletes expired or no longer needed objects according to the set rules and time periods. For example, cleanup operations can be performed based on the creation time, last access time, or storage time of the object; through the rgw lifecycle function, one or more rules can be set for each bucket. The rules can define the life cycle of objects in the bucket according to needs. For example, a rule can be set to automatically delete objects in the bucket 30 days after creation; in the cloud rendering platform, bucket cleanup policies can be easily configured and managed through the management interface or API. Administrators can set different rules and schedules for different buckets according to different needs; according to the rule settings, the storage system will automatically perform cleanup operations. These operations can include deleting expired objects, archiving or migrating data, etc. The cleanup operation can be automatically triggered according to the set schedule, or it can be triggered manually; based on the above storage cleanup mechanism, the user's storage cleanup policy can be personalized and personalized cleanup can be performed for the user's expired asset files.

[0079] The backend service framework of the cloud rendering platform includes: SpringCloud microservices, dubbo service dispatch center, MySQL database, Mongodb database, and Redis cache.

[0080] Use the official S3 storage API interface to implement the interaction between the cloud rendering platform and the S3 storage protocol, and realize basic storage functions such as asset management, user management, declaration cycle management, storage statistics, etc. in the cloud rendering platform.

[0081] In summary, the present invention provides a cloud rendering platform storage implementation method based on the S3 storage protocol, which has the following technical effects:

[0082] (1) High reliability and availability: By using the S3 storage protocol and data backup strategy, redundant storage and regular backup of data are ensured. Even if a storage node fails or other unexpected situations occur, data can still be restored from other nodes or backups, ensuring data reliability and availability.

[0083] (2) Scalability and elasticity: The distributed architecture based on the S3 storage protocol can easily expand storage capacity and computing resources. The cloud rendering platform storage system can be expanded horizontally and vertically according to demand to adapt to the growing data storage and processing needs. At the same time, the storage system also has the advantages of elasticity and automation. When demand changes, it can automatically adjust resource allocation and network configuration to ensure efficient operation of the system.

[0084] (3) Efficient data storage and transmission: Improve data reading and writing efficiency by optimizing data transmission and storage strategies. Technical measures such as multi-threaded transmission, intelligent node selection, compression and acceleration algorithms can effectively reduce transmission time, improve storage efficiency, and speed up data reading through caching and pre-fetching technologies.

[0085] (4) Data security and privacy protection: The access control mechanism and encryption technology provided by the S3 storage protocol ensure data security and privacy protection. Administrators can flexibly set access permissions for buckets and objects and use encryption algorithms to encrypt data to prevent unauthorized access and data leakage.

[0086] (5) Simplified management and operation: Through standard interfaces and methods based on the S3 storage protocol, the management and operation of buckets and objects are simplified. Users can use unified interfaces and tools to store and process data conveniently and quickly, improving work efficiency and operational convenience.

[0087] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A cloud rendering platform storage implementation method based on the S3 storage protocol, characterized in that: The following steps are involved: S1: The user completes registration on the cloud rendering platform; S2: Use the administrator account to create an AK and SC key pair for the new user in S3 storage; S3: The cloud rendering platform database saves the user's key pair and creates a unique S3 bucket for the user; S4: Set the file lifecycle for the corresponding S3 bucket file according to the user storage cleanup configuration; S5: User asset upload or rendering output file; S6: Upload to the corresponding user's S3 bucket through the S3 storage protocol API; S7: back up user files on different storage nodes; S8: Users perform file interaction operations through cloud rendering asset management; S9: User files expire, triggering automatic cleanup of lifecycle management.

2. According to a cloud rendering platform storage implementation method based on the S3 storage protocol according to claim 1, it is characterized in that: The file interaction operations include: adding, deleting, modifying, querying, and downloading.

3. According to a cloud rendering platform storage implementation method based on the S3 storage protocol according to claim 1, it is characterized in that: Use the S3 storage protocol to implement basic interactive functions, including: Use the API provided by the S3 storage protocol to manage and operate stored data; Bucket management: implement bucket creation, deletion, and query operations; Object operations: implement read, write, and delete operations on objects in a bucket; Access control: The access control mechanism provided by the S3 storage protocol is used to control access rights to buckets and objects. Metadata management: implement metadata management of buckets and objects; Error handling and logging: Consider error handling and logging when implementing basic interactive functionality.

4. According to a method for implementing storage of a cloud rendering platform based on the S3 storage protocol according to claim 1, it is characterized in that: In step S7, the backup specifically includes: regularly performing multi-node backup processing on the user's asset files on the storage to avoid asset losses of some users caused by an abnormality of a certain storage node, thereby improving storage reliability and disaster tolerance.

5. According to a method for implementing storage of a cloud rendering platform based on the S3 storage protocol according to claim 1, it is characterized in that: The cloud rendering platform applies for a unique account and storage bucket for each user assigned to S3 storage, obtains the user's unique AK and SC key pair, saves them to the cloud rendering platform database and associates them with the corresponding user, and isolates the user's asset files through a unique storage bucket, configures a specific permission range, and only allows access to the specified user storage bucket through the corresponding user's key pair, ensuring that assets between users cannot be accessed and called by each other, ensuring the security and privacy of S3 storage.

6. The method for implementing storage of a cloud rendering platform based on the S3 storage protocol according to claim 1, characterized in that: Optimizing S3 storage transmission efficiency includes at least one of the following methods: Multi-threaded transmission: Data can be transmitted in parallel through multiple threads to improve transmission efficiency. Large files or data can be sliced ​​into multiple small blocks, and these small blocks can be transmitted simultaneously using multiple threads, which can effectively reduce transmission time and improve transmission efficiency. Intelligent storage node selection: intelligently select the best storage node for data transmission based on network conditions and node load conditions; Efficient compression and acceleration algorithms; Cache and pre-fetch technology: Improve data reading efficiency by setting cache on local storage devices; Data sharding and distributed transmission: Slice large files or data into multiple small blocks, and store these small blocks in a distributed manner on multiple storage nodes to achieve simultaneous data transmission and reading operations, reduce transmission time and improve transmission efficiency.

7. The method for implementing cloud rendering platform storage based on the S3 storage protocol according to claim 1, characterized in that: The S3 storage protocol cleanup implementation method is: configure the bucket cleanup policy by configuring the natively supported rgw lifecycle function. The cleanup policy automatically deletes expired or no longer needed objects according to the set rules and time period.

8. The method for implementing cloud rendering platform storage based on the S3 storage protocol according to claim 1, characterized in that: The backend service framework of the cloud rendering platform includes: SpringCloud microservices, dubbo service dispatch center, MySQL database, Mongodb database, and Redis cache.

Images