Cloud distributed storage management and acceleration system

Abstract

The invention provides a cloud distributed storage management and acceleration system, which relates to the field of cloud distributed storage, and comprises a two-stage storage mechanism and an intelligent data storage mechanism: the two-stage storage mechanism comprises near-end storage and far-end storage; the plurality of near-end memories form a single near-end storage node, and the near-end memories are high-performance storage media mounted on the service nodes; the remote storage is a file which is not accessed by a user for a long time and is low in importance; the intelligent data storage mechanism comprises file scheduling and file storage. The cloud distributed storage management level acceleration scheme shows remarkable technical advantages and application effects. A two-stage storage mechanism can effectively solve the performance difference when files are accessed in different regions and different networks, ambiguity generated by cooperative communication is reduced, and the user experience is effectively improved; and through data hierarchical management design, the storage management requirements of files in different periods are effectively met.

Description

Technical Field

[0001] The present invention relates to the field of cloud distributed storage, and in particular to a cloud distributed storage management and acceleration system. Background Art

[0002] "Giant Mojing" is a cloud-based AI painting platform for the pan-entertainment industry, focusing on providing efficient and convenient creation tools and resources for art creators. The platform integrates an advanced model reasoning framework to support a variety of complex AI functions, such as style transfer, image generation, and intelligent completion. The cloud-based distributed storage management and acceleration solution design optimizes cloud-based drawing, especially high-resolution large file reading and management, improves file reading efficiency, and brings users a smooth creation experience.

[0003] At present, various intelligent painting platforms continue to emerge, but they cannot solve the problem of team collaborative creation in different office scenarios. The main manifestations are:

[0004] 1. Classify the content created in different periods according to the access frequency:

[0005] 2. When cross-regional teams collaborate and communicate, differences in file access efficiency occur due to geographical and network influences. Summary of the invention

[0006] In order to make up for the above shortcomings, the present invention provides a cloud-based distributed storage management and acceleration system, which aims to improve the various intelligent painting platforms in the prior art that cannot well solve the problem of team collaborative creation in different office scenarios.

[0007] The present invention is achieved in that:

[0008] The present invention provides a cloud distributed storage management and acceleration system, including a two-level storage mechanism and an intelligent data storage mechanism:

[0009] The two-level storage mechanism includes near-end storage and far-end storage;

[0010] A plurality of the proximal storages form a single proximal storage node, and the proximal storage is a high-performance storage medium mounted on each service node;

[0011] The remote storage is files that have not been accessed by the user for a long time and are of low importance;

[0012] The intelligent data storage mechanism includes file scheduling and file storage;

[0013] In the file scheduling, users with different regional IP addresses access different application service nodes, and the application service can access multiple proximal storage nodes and one remote storage node; the proximal node in the file scheduling has a write mark function, a priority function, a forced remaining space function, and a file migration time recording function;

[0014] The file storage has the functions of writing files to near-end storage and synchronizing them to far-end storage, reading files, migrating files from near-end storage, compressing files from far-end storage, triggering high-definition repair and operation, and storing multiple files.

[0015] Preferably, a third-party API is called to obtain the user's current geographic location, and the proximal storage node of the user's current IP is obtained according to the pre-planned division control of the access area.

[0016] Preferably, the same data file is backed up in at least two remote storages, and data consistency verification is performed regularly by calculating the file MD5 value.

[0017] Preferably, the proximal node marking function: the write mark is either "yes" or "no". When the proximal storage node is marked as "yes", the application service node allows data to be stored in the marked proximal storage node; when the proximal storage node is marked as "no", the application service node does not allow data to be stored in the marked proximal storage node, but can read the data.

[0018] Preferably, the proximal node priority function:

[0019] The proximal node priority includes high priority, medium priority and low priority;

[0020] The application service stores data to accessible, higher priority storage nodes.

[0021] Preferably, the near-end node forced remaining space function: when the available space of the storage node is less than the forced remaining space, the application service node will not continue to store data to the storage node even if the write mark is normal.

[0022] Preferably, the file storage function of writing files to local storage and synchronizing to remote storage includes the following steps:

[0023] S1, file data writing;

[0024] S2, near-end storage node selection:

[0025] S3. Take the proximal storage node with a write mark of "yes". When there are multiple proximal storage nodes, take the proximal storage node with the highest priority: when there are multiple proximal storage nodes, take the proximal storage node with a large forced remaining space value; when there is one proximal storage node, select the proximal storage node as the stored proximal storage node; when there are zero proximal storage nodes, write the file data to the remote storage;

[0026] S4: Determine whether the proximal storage node selected in S3 has enough space.

[0027] When the selected near-end storage node in S3 has enough space, write the data and synchronize it to the remote storage. The remote storage will then synchronize and update the remote storage's backup node:

[0028] When the selected near-end storage node in S3 has enough space but no data has been written, data writing to the storage fails and the content is written to the remote storage:

[0029] When the selected proximal storage node in S3 does not have enough space, return to S3 and select a proximal storage node with a lower priority;

[0030] When all the proximal storage nodes in S3 do not have enough space, the proximal storage is not satisfied and the content is written to the remote storage.

[0031] Preferably, the file storage function of reading files includes the following steps:

[0032] S1. Read the near-end storage first. If there is data in the near-end storage, the reading is successful:

[0033] S2: If there is no data in the local storage, the data in the associated remote storage is read successfully, and the data is synchronized to the local storage;

[0034] S3: If there is no data in the associated remote storage, read other remote storage and synchronize the data to the local storage.

[0035] Preferably, the near-end storage migration file function in the file storage includes the following steps:

[0036] S1. The task of deleting local storage data is started, and local storage data is deleted and retrieved;

[0037] S2: If there is data in the remote storage and it is not accessed for a set period, the data will be deleted and the task will end. If there is no access within the set period, the task will end.

[0038] S3: There is no data in the remote storage, and the remote data backup is deleted.

[0039] Preferably, the remote storage compressed file function in the file storage includes the following steps:

[0040] S1, remote storage data compression task starts, and remote storage data is retrieved;

[0041] S2. When the remote storage does not exceed the set period, the task ends; when the remote storage exceeds the set period, the content of the storage file is compressed and the task ends.

[0042] The beneficial effects of the present invention are:

[0043] The cloud-based distributed storage management-level acceleration solution of the present invention demonstrates significant technical advantages and application effects. The two-level storage mechanism can effectively solve the performance differences when accessing files in different regions and networks, reduce the ambiguity caused by collaborative communication, and effectively improve the user experience; the data hierarchical management design effectively solves the storage management needs of files in different periods. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0045] Figure 1 It is a flow chart of a cloud distributed storage management and acceleration system provided by an embodiment of the present invention;

[0046] Figure 2 It is a flowchart of the function of writing files into near-end storage and synchronizing them to remote storage in a cloud distributed storage management and acceleration system provided by an embodiment of the present invention;

[0047] Figure 3 It is a flowchart of a file reading function in a cloud distributed storage management and acceleration system provided by an embodiment of the present invention;

[0048] Figure 4 It is a flowchart of a near-end storage migration file function in a file storage in a cloud distributed storage management and acceleration system provided by an embodiment of the present invention;

[0049] Figure 5 It is a flow chart of a remote storage compressed file function in a file storage in a cloud distributed storage management and acceleration system provided by an embodiment of the present invention;

[0050] Figure 6 It is a flow chart of triggering high-definition repair and operation functions in a cloud-based distributed storage management and acceleration system provided by an embodiment of the present invention;

[0051] Figure 7It is a flow chart of multi-file storage function in a cloud distributed storage management and acceleration system provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0052] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0053] Example

[0054] Reference Figure 1-Figure 7 , a cloud-based distributed storage management and acceleration system, including a two-level storage mechanism and an intelligent data storage mechanism:

[0055] The two-level storage mechanism includes near-end storage and far-end storage, and the two-level storage mechanism solves the differences in reading files in different areas and different nodes.

[0056] The description of local storage is as follows:

[0057] Near-end storage is mainly high-performance storage media mounted on each service node, such as SSD, SAN, etc. Multiple near-end storages constitute a single near-end storage node.

[0058] Near-end storage is mainly used to store files that users have recently accessed. Storage management is performed based on the creation time and access time of the files.

[0059] The management system calls a third-party API to obtain the user's current geographic location based on the user's current network IP information, and obtains the proximal storage node of the user's current IP based on the pre-planned access area division control. At the same time, it uses machine learning technology to train a large amount of IP address and geographic location data to avoid the problem that the third-party API cannot normally resolve the geographic location.

[0060] The description of remote storage is as follows:

[0061] Remote storage is mainly used to store files that users have not accessed for a long time and are of low importance. It mainly uses cloud storage from major cloud vendors. The integrity of remote storage data is mainly guaranteed by cloud vendors.

[0062] To ensure data reliability, the same data file is backed up in at least two remote storage locations. At the same time, data consistency is checked regularly by calculating the file MD5 value.

[0063] Wherein, the intelligent data storage mechanism includes file scheduling and file storage;

[0064] The file scheduling mechanism is managed by:

[0065] First, the storage nodes are isolated. Users in different regions access different application service nodes. Application services can access multiple near-end storage nodes and one far-end storage node. With this idea, each storage node is configured to achieve storage node access isolation.

[0066] Secondly, the near-end node in the file scheduling has a write mark function, a priority function, a forced remaining space function, and a file migration time recording function, and each function is described as follows:

[0067] The write mark function is used to configure whether there is an abnormality in the proximal storage node. The application service node is not allowed to store data in the abnormal storage node. The write mark is either "yes" or "no". When the proximal storage node is marked as "yes", the application service node is allowed to store data in the marked proximal storage node; when the proximal storage node is marked as "no", the application service node is not allowed to store data in the marked proximal storage node, but can read data.

[0068] The near-end node priority includes high priority, medium priority and low priority. In addition, in the priority function, the application service should store data in accessible storage nodes with higher priority. When the storage node priorities are consistent, the data will be stored in the storage node with larger remaining available space.

[0069] In the forced remaining space function, when the available space of a storage node is less than the forced remaining space, the application service node will not continue to store data to the storage node even if the write mark is normal.

[0070] In the file migration time recording function, the recording method is a positive integer, where (current date - file creation time / update time) > migration time, triggering migration, that is, the local storage deletes the file, but before deletion it will determine that there is at least one copy of the file in the remote storage.

[0071] The file storage mechanism is managed as follows:

[0072] Method 1: Write files to local storage and synchronize them to remote storage. Figure 2 , this function includes the following steps:

[0073] S1, file data writing;

[0074] S2, near-end storage node selection;

[0075] S3. Take the proximal storage node with a write mark of "yes". When there are multiple proximal storage nodes, take the proximal storage node with the highest priority; when there are multiple proximal storage nodes, take the proximal storage node with a large forced remaining space value: when there is one proximal storage node, select the proximal storage node as the proximal storage node to be stored; when there are zero proximal storage nodes, write the file data to the remote storage:

[0076] S4: Determine whether the proximal storage node selected in S3 has enough space.

[0077] When the selected near-end storage node in S3 has enough space, the data is written and synchronized to the remote storage, and the remote storage synchronously updates the remote storage's backup node;

[0078] When the selected near-end storage node in S3 has enough space but no data has been written, data writing to storage fails and the content is written to the remote storage;

[0079] When the selected proximal storage node in S3 does not have enough space, return to S3 and select a proximal storage node with a lower priority;

[0080] When all the proximal storage nodes in S3 do not have enough space, the proximal storage is not satisfied and the content is written to the remote storage.

[0081] Method 2: Read file function, refer to Figure 3 , this function includes the following steps:

[0082] S1, read the near-end storage first. If there is data in the near-end storage, the reading is successful;

[0083] S2: If there is no data in the local storage, the data in the associated remote storage is read successfully, and the data is synchronized to the local storage;

[0084] S3: If there is no data in the associated remote storage, read other remote storage and synchronize the data to the local storage.

[0085] Method 3: File storage in the local storage migration file function, refer to Figure 4 , this function includes the following steps:

[0086] S1. The task of deleting local storage data is started, and local storage data is deleted and retrieved;

[0087] S2: If there is data in the remote storage and it is not accessed for a set period, the data will be deleted and the task will be completed. If there is no access within the set period, the task will be completed.

[0088] S3: There is no data in the remote storage, and the remote data backup is deleted.

[0089] Method 4: Remote storage of compressed files in file storage, refer to Figure 5 , this function includes the following steps:

[0090] S1, remote storage data compression task starts, and remote storage data is retrieved;

[0091] S2. When the remote storage does not exceed the set period, the task ends; when the remote storage exceeds the set period, the content of the storage file is compressed and the task ends.

[0092] Method 5: Triggering HD repair and operation functions, refer to Figure 6 , this function includes the following steps:

[0093] S1, read remote storage data;

[0094] S2: When the remote storage data in S1 meets the usage requirements, the task ends; when the remote storage data in S1 does not meet the usage requirements, the user initiates a high-definition repair service, and the repaired data is saved to the local storage, and the task ends.

[0095] Method 6: Multi-file storage function, refer to Figure 7 , this function includes the following steps:

[0096] S1, retrieve remote storage data;

[0097] S2. When there are more than three copies of the same file, keep the files of the three nodes that have been accessed most recently, and delete the files of other nodes. When there are no more than three copies of the same file, the task ends.

[0098] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A cloud distributed storage management and acceleration system, characterized in that: Including two-level storage mechanism and intelligent data storage mechanism: The two-level storage mechanism includes near-end storage and far-end storage; A plurality of the proximal storages form a single proximal storage node, and the proximal storage is a high-performance storage medium mounted on each service node; The remote storage is files that have not been accessed by the user for a long time and are of low importance; The intelligent data storage mechanism includes file scheduling and file storage; In the file scheduling, users with different regional IP addresses access different application service nodes, and the application service can access multiple proximal storage nodes and one remote storage node; the proximal node in the file scheduling has a write mark function, a priority function, a forced remaining space function, and a file migration time recording function; The file storage has the functions of writing files to near-end storage and synchronizing them to far-end storage, reading files, migrating files from near-end storage, compressing files from far-end storage, triggering high-definition repair and operation, and storing multiple files.

2. A cloud-based distributed storage management and acceleration system according to claim 1, characterized in that: Call a third-party API to obtain the user's current geographic location, and obtain the proximal storage node of the user's current IP based on the pre-planned access area division control.

3. A cloud-based distributed storage management and acceleration system according to claim 2, characterized in that: The same data file is backed up in at least two remote storage locations, and data consistency is checked regularly by calculating the file's MD5 value.

4. A cloud-based distributed storage management and acceleration system according to claim 3, characterized in that: The proximal node marking function: write the mark as either "yes" or "no". When the proximal storage node is marked as "yes", the application service node allows data to be stored in the marked proximal storage node; when the proximal storage node is marked as "no", the application service node does not allow data to be stored in the marked proximal storage node, but can read the data.

5. A cloud-based distributed storage management and acceleration system according to claim 4, characterized in that: The proximal node priority function: The proximal node priority includes high priority, medium priority and low priority; The application service stores data to accessible, higher priority storage nodes.

6. A cloud-based distributed storage management and acceleration system according to claim 5, characterized in that: The near-end node forced remaining space function: when the available space of the storage node is less than the forced remaining space, the application service node will not continue to store data to the storage node even if the write mark is normal.

7. A cloud-based distributed storage management and acceleration system according to claim 6, characterized in that: The file storage function of writing files to local storage and synchronizing them to remote storage includes the following steps: S1. File data writing: S2, near-end storage node selection; S3, take the proximal storage node with a write mark of "yes", and when there are multiple proximal storage nodes, take the proximal storage node with the highest priority; When there are multiple proximal storage nodes, the proximal storage node with the largest forced remaining space value is selected; when there is only one proximal storage node, the proximal storage node is selected as the stored proximal storage node; When there are zero local storage nodes, the file data is written to the remote storage: S4: Determine whether the proximal storage node selected in S3 has enough space. When the selected near-end storage node in S3 has enough space, the data is written and synchronized to the remote storage, and the remote storage synchronously updates the remote storage's backup node; When the selected near-end storage node in S3 has enough space but no data has been written, data writing to storage fails and the content is written to the remote storage; When the selected proximal storage node in S3 does not have enough space, return to S3 and select a proximal storage node with a lower priority; When all the proximal storage nodes in S3 do not have enough space, the proximal storage is not satisfied and the content is written to the remote storage.

8. A cloud-based distributed storage management and acceleration system according to claim 6, characterized in that: The file storage function of reading files includes the following steps: S1, read the near-end storage first. If there is data in the near-end storage, the reading is successful; S2: If there is no data in the local storage, the data in the associated remote storage is read successfully, and the data is synchronized to the local storage; S3: If there is no data in the associated remote storage, read other remote storage and synchronize the data to the local storage.

9. A cloud-based distributed storage management and acceleration system according to claim 6, characterized in that: The file storage function of migrating files at the near end includes the following steps: S1. The task of deleting local storage data is started, and local storage data is deleted and retrieved; S2: If there is data in the remote storage that has not been accessed for a set period of time, the data will be deleted and the task will be completed; If there is no visit within the set period, the task ends; S3: There is no data in the remote storage, and the remote data backup is deleted.

10. A cloud-based distributed storage management and acceleration system according to claim 6, characterized in that: The remote storage compressed file function in the file storage includes the following steps: S1, remote storage data compression task starts, and remote storage data is retrieved; S2. When the remote storage does not exceed the set period, the task ends; when the remote storage exceeds the set period, the content of the storage file is compressed and the task ends.

Images