File saving method and device, electronic equipment and storage medium

Abstract

The invention discloses a file saving method and device, electronic equipment and a storage medium, which are applied to the technical field of cloud storage, and the method comprises the following steps: determining the execution sequence of compression operation and fragmentation operation according to the file storage scale of a to-be-stored file; respectively executing the fragmentation operation and the compression operation on the to-be-stored file according to the execution sequence to obtain at least one file fragment; and storing the file fragments to storage positions corresponding to the storage nodes. According to the embodiment of the invention, the to-be-stored file is subjected to storage management according to the file storage scale, the file storage method adaptive to the file scale of the to-be-stored file is determined, the storage space occupation of the file data can be reduced, and the file storage efficiency is improved.

Description

Technical Field

[0001] The present invention relates to the field of cloud storage technology, and in particular to a file saving method, device, electronic device and storage medium. Background Art

[0002] Cloud computing infrastructure platforms contain a large amount of static archival data, such as image files and backup files. This data typically has the following characteristics: large file sizes, read-only file attributes, and frequent file access. The file sizes of these static archival data vary significantly, varying across operating systems. For example, a Linux system image is approximately 2 gigabytes; for Windows, around 20 gigabytes. For some customized system images, the size can even reach hundreds of gigabytes. Image files are often in the gigabyte range. However, backup files often derive from business production data, and their size varies widely, ranging from a few megabytes to terabytes. Image files are used during the cloud server creation phase, while backup files are regularly backed up and generated during cloud server operation. Obviously, static archival data will increase with business development, and this data will continue to grow in size. Managing the storage of static archival data and improving its storage efficiency will directly impact the service quality and stability of the cloud platform. Summary of the Invention

[0003] The present invention provides a file saving method, device, electronic device and storage medium to store and manage static archive data in a cloud platform, reduce the storage space occupied by static archive data, and improve the storage efficiency of static archive data.

[0004] According to one aspect of the present invention, a file storage method is provided, wherein the method is applied to a storage node of a distributed system, and the method comprises:

[0005] Determine the order of executing compression and sharding operations based on the file storage size of the files to be stored;

[0006] Perform the sharding operation and the compression operation on the file to be stored respectively according to the execution order to obtain at least one file shard;

[0007] The file segments are stored in the storage locations corresponding to the storage nodes.

[0008] According to another aspect of the present invention, a file storage device is provided, wherein the device is applied to a storage node of a distributed system, and the device includes:

[0009] An execution sequence module is used to determine the execution sequence of compression operations and fragmentation operations according to the file storage size of the file to be stored;

[0010] A file sharding module, configured to perform the sharding operation and the compression operation on the file to be stored respectively according to the execution order, to obtain at least one file shard;

[0011] The fragment saving module is used to store the file fragments in the storage location corresponding to the storage node.

[0012] According to another aspect of the present invention, an electronic device is provided, comprising:

[0013] at least one processor; and

[0014] a memory communicatively connected to the at least one processor; wherein,

[0015] The memory stores a computer program that can be executed by the at least one processor. The computer program is executed by the at least one processor so that the at least one processor can execute the file saving method described in any embodiment of the present invention.

[0016] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the file saving method described in any embodiment of the present invention when executed.

[0017] The technical solution of the embodiment of the present invention obtains the file storage scale of the file to be stored, determines the execution order of the compression operation and the sharding operation corresponding to the file to be stored according to the file storage scale, and performs the sharding operation and the compression operation on the file to be stored in sequence according to the execution order, thereby obtaining at least one file shard, and saving the file shard to the storage location corresponding to the storage node. The embodiment of the present invention can select an appropriate compression and sharding operation mode for files of different data storage scales, thereby facilitating the improvement of the file compression rate and the speed of file saving, realizing the storage management of static archive data in the cloud platform, reducing the storage space occupied by static archive data, improving the reading and writing efficiency of stored data, and improving the service quality of the cloud platform.

[0018] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present invention, nor is it intended to limit the scope of the present invention. Other features of the present invention will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0020] Figure 1 This is a schematic diagram of the architecture of a distributed system provided according to an embodiment of the present invention;

[0021] Figure 2 This is a flowchart of a file saving method provided according to the first embodiment of the present invention;

[0022] Figure 3 is a flowchart of another file saving method provided according to embodiment 2 of the present invention;

[0023] Figure 4 is a flowchart of another file saving method provided according to embodiment 3 of the present invention;

[0024] Figure 5 This is an example diagram of a computing node architecture provided according to the fourth embodiment of the present invention;

[0025] Figure 6 This is an example diagram of a file saving method provided according to a fourth embodiment of the present invention;

[0026] Figure 7 This is an example diagram of a file reading method provided according to a fourth embodiment of the present invention;

[0027] Figure 8 This is an example diagram of another file saving method provided according to the fourth embodiment of the present invention;

[0028] Figure 9 This is an example diagram of another file reading method provided according to embodiment 4 of the present invention.

[0029] Figure 10 This is a structural diagram of a file storage device provided according to a fifth embodiment of the present invention;

[0030] Figure 11 It is a structural diagram of an electronic device for implementing the file saving method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0031] In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of the present invention.

[0032] It should be noted that the terms "first", "second", etc. in the description and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0033] Figure 1 This is an architectural diagram of a distributed system provided according to an embodiment of the present invention. The distributed system may include a region, an availability zone, and a store node. A region may be a geographically independent cloud service area that may cover a city or multiple adjacent cities. Each region has independent computing, storage, network, and other infrastructure, and is interconnected through a high-speed network. An availability zone (AZ) may be a logical partition within a region. Each AZ may represent a group of physically isolated data center clusters and may have independent power, cooling, and network facilities. Users may deploy across AZs to avoid single points of failure. Multiple AZs within the same region are interconnected through high-speed optical fibers, and resources within the AZs are interconnected. The above resources include but are not limited to computing, storage, and network, but resource scheduling can be configured for AZs. A store node may be a physical or logical node in a storage service, responsible for the actual storage and access of data. A storage node may belong to a certain AZ and may be part of a distributed storage cluster. Data may be distributed and stored among multiple storage nodes, supporting horizontal expansion and high concurrent access. See Figure 1In Region 1, there are three availability zones (AZs), and in availability zone 3, there are four storage nodes. There is no limit to the number of storage nodes within an availability zone. Each storage node has at least one folder for storing data, which can be used as a storage location in embodiments of the present invention. Therefore, each storage node has at least one storage location. To simplify management, some embodiments of the present invention employ a deployment method where each storage node is configured with one storage location.

[0034] Example 1

[0035] Figure 2 This is a flowchart of a file storage method provided according to the first embodiment of the present invention. This embodiment is applicable to the static archiving data storage management in the cloud platform. The method can be executed by a file storage device. The file storage device can be implemented in the form of hardware and / or software. The device can be configured in the storage node of the distributed system. Figure 2 As shown, the method includes:

[0036] Step 110: Determine the execution order of the compression operation and the fragmentation operation according to the file storage size of the file to be stored.

[0037] The files to be stored may be file data generated as business progresses in a distributed system. The files to be stored may specifically be static archiving data. The files to be stored may include mirror files, backup files, etc. The file storage scale may be the file size of the files to be stored. The file storage scale may be the space required to store the files in a distributed system. The file storage scale may be bytes, megabytes, gigabytes, terabytes, etc. The compression operation may be an operation used to reduce the volume of the files to be stored in order to save storage space and network transmission overhead for the files to be stored. The compression operation may be implemented using the Gzip compression algorithm, the Snappy compression algorithm, the LZ4 compression algorithm, the Zstandard compression algorithm, etc. The sharding operation may be to split the files to be stored into multiple small shards for distributed storage. The sharding operation may be based on size, key value, time, or business.

[0038] In an embodiment of the present invention, a file to be stored can be obtained, and the file storage scale of each file to be stored can be determined. The order in which the compression operation and the sharding operation are performed on the file to be stored can be determined according to the size of the file storage scale, so that the file to be stored can be processed according to the execution order. For example, when the file storage scale is greater than a threshold value, the execution order is determined to first perform the sharding operation and then perform the compression operation. When the file storage scale is less than or equal to the threshold value, the execution order is determined to first perform the compression operation and then perform the sharding operation. It can be understood that performing the compression operation on the file to be stored first and then the sharding operation can improve the compression rate of the file to be stored and reduce the storage space of the file to be stored. If the sharding operation is performed on the file to be stored first and then the compression operation is performed, the compression operation can be performed in parallel within each file shard, which can improve data storage efficiency.

[0039] Step 120: Perform a sharding operation and a compression operation on the file to be stored in the execution order to obtain at least one file shard.

[0040] In an embodiment of the present invention, the sharding operation and the compression operation can be performed on the file to be stored in sequence according to the determined execution order, so that the file to be stored is processed into multiple file shards, and the file data in each file shard can be generated through compression. For example, for a file to be stored with a larger file storage scale, the execution order is to first perform the sharding operation and then perform the compression operation. The file to be stored can be first divided into multiple file shards according to this execution order, and then the compression operation can be performed in each file shard respectively, thereby reducing the space occupied by the file data in each file shard. For a file to be stored with a smaller file storage scale, the execution order is to first perform the compression operation and then perform the sharding operation. The file to be stored can be first processed into a compressed data file through the compression operation according to this execution order, and then the compressed data file can be divided into multiple file shards through the sharding operation.

[0041] Step 130: Store the file fragments in the storage location corresponding to the storage node.

[0042] Among them, the storage location can be a location within the storage node for storing the file to be stored. The storage location can be a folder or designated storage space for storing file shards. The storage location can be located at the storage node for obtaining the file to be stored, or the storage node corresponding to the storage node for obtaining the file to be stored can be a computing node in the distributed system that is in the same availability area as the storage node.

[0043] In embodiments of the present invention, the storage location of a storage node can be determined within a distributed system, and the file shards can be stored in the corresponding storage location. Furthermore, in some embodiments of the invention, to facilitate reading of the file shards, file information corresponding to the file shards can be stored in the storage location. For example, folder names for one or more layers of folders can be generated based on the shard identifiers of the file shards.

[0044] In an embodiment of the present invention, by obtaining the file storage scale of the file to be stored, determining the execution order of the compression operation and the sharding operation corresponding to the file to be stored according to the file storage scale, and sequentially performing the sharding operation and the compression operation on the file to be stored according to the execution order, at least one file shard is obtained, and the file shard is saved to the storage location corresponding to the storage node. In an embodiment of the present invention, an appropriate compression and sharding operation mode can be selected for files of different data storage scales, thereby facilitating the improvement of the file compression rate and the speed of file saving, realizing the storage management of static archive data in the cloud platform, reducing the storage space occupied by static archive data, improving the reading and writing efficiency of stored data, and improving the service quality of the cloud platform.

[0045] Example 2

[0046] Figure 3 This is a flowchart of another file saving method provided by the second embodiment of the present invention. The embodiment of the present invention is a specific embodiment based on the above embodiment. The embodiment of the present invention describes the process of performing a compression operation and then a fragmentation operation on the stored file. Figure 3 The method provided in the embodiment of the present invention specifically includes the following steps:

[0047] Step 210: Determine the execution order of the compression operation and the fragmentation operation according to the file storage size of the file to be stored.

[0048] Step 220: Read the file to be stored; and perform compression on the file to be stored according to a preset streaming compression rule.

[0049] The storage node may be a component of a distributed system. The storage node may obtain static archive files generated by the distributed system during cloud host creation or business execution. The storage node may use the obtained static archive files as files to be stored. The preset streaming compression rules may be pre-configured rules for compressing data. The preset streaming compression rules may be configured on the storage nodes of the distributed system. Each storage node in the distributed system may be individually configured with the preset streaming compression rules. The preset streaming compression rules may be implemented using the LZ4 compression algorithm, the Zstandard compression algorithm, the Snappy compression algorithm, the Gzip compression algorithm, or the like.

[0050] In an embodiment of the present invention, when a file to be stored is obtained, a pre-configured preset streaming compression rule can be called to perform a compression operation on the file to be stored, thereby achieving compression of the file to be stored, thereby reducing the storage space occupied by the file to be stored. It can be understood that the compression operation of the file to be stored can be performed in a streaming manner, that is, the file to be stored is gradually compressed.

[0051] Step 230: When the compressed file data generated by compression reaches a data volume threshold, a fragmentation operation is performed to treat the currently generated compressed file data as a file fragment.

[0052] The compressed file data may be generated by processing the file to be stored according to a preset streaming compression rule, and the compressed file data may be at least a portion of the compressed file to be stored.

[0053] In an embodiment of the present invention, compressed file data that has undergone a compression operation can be monitored to determine the data volume of the compressed file data. When the data volume is greater than or equal to a data volume threshold, a slicing operation can be performed on the compressed file data that currently meets the data volume threshold, thereby generating a file slicing operation. That is, whenever the data volume of a group of compressed file data reaches the data volume threshold, the group of compressed file data that reaches the data volume threshold is processed as a file slicing operation. Furthermore, when the data volume of the compressed file generated by the final compression of the file to be stored is less than the data volume threshold, all the remaining compressed file data can also be processed as a file slicing operation.

[0054] Step 240: Find at least one other storage node in the distributed system that belongs to the same availability zone as the storage node, and determine the storage locations of the storage node and the other storage nodes.

[0055] The available zone may be an availability zone (AZ) within each region in the distributed system, the current storage node may belong to the available zone, other storage nodes may be storage nodes belonging to the same storage zone as the current storage node, other storage nodes may be backup storage nodes, and the other storage nodes. The storage location may be an area within the storage node where file shards are stored, and the storage location may specifically include other storage structures such as folders and file databases.

[0056] In an embodiment of the present invention, an available area to which a storage node belongs can be determined within a distributed system, and other storage nodes can be identified within the available area. The other storage nodes can be one or more. Storage locations can be determined within the current storage node and each other storage node. The number of storage locations can be one or more, and the number of storage locations can be determined based on configuration or actual business needs. The storage location can be a storage structure within the other storage node, and the storage structure can include a file database or folder, etc.

[0057] Step 250: Write the sparse file of the file to be stored into the storage location.

[0058] Among them, sparse files can be used to optimize storage space in distributed systems. Sparse files can save and record the storage location of each file segment. The storage location can be implemented in the form of a starting position and length. Each storage location can have its own corresponding sparse file.

[0059] In an embodiment of the present invention, a sparse file can be written in each storage location. The content of the sparse file can be empty. The sparse file currently written to the storage file is only a placeholder. The actual data size of the sparse file can be 0. It can be understood that the sparse file corresponding to each storage location can respectively save its corresponding compressed file fragments.

[0060] Step 260: Write each file fragment into the sparse file according to the offset information.

[0061] The offset information may be the offset between data in each file shard, and the offset information may identify the offset between the starting positions of data in the file shard.

[0062] In an embodiment of the present invention, the offset information between one or more segments of data in each file shard is determined, and each segment of data in the file shard can be added to the corresponding sparse file according to its corresponding offset information, thereby realizing the preservation of the file shard in the storage location.

[0063] An embodiment of the present invention determines the execution order of compression operations and sharding operations according to the file storage scale of the file to be stored, reads the file to be stored, performs compression operations on the file to be stored according to preset streaming compression rules, and performs sharding operations on the compressed file data generated by compression whenever the data volume of the compressed file data reaches a data volume threshold, processes the compressed file data into a file shard, obtains a storage location in a distributed system, and the storage location is located in other storage nodes in the same available area as the storage node, generates a sparse file at the storage location, and saves the file shard to the sparse file according to offset information to achieve the preservation of the file shard. In an embodiment of the present invention, the execution order of compression operation and fragmentation operation is determined based on the file storage scale of the file to be stored, and appropriate compression and fragmentation operation methods can be selected for files of different data storage scales, so as to improve the compression rate of the file and improve the file saving speed; streaming operation is performed on the stored file according to the streaming compression rules, and the fragmentation operation can be performed after part of the data of the file to be stored is compressed, which can improve the storage efficiency of the file; based on the sparse file, the offset information of the file fragment at the storage location is saved, and there is no need to traverse all the saved information at the storage location, which can improve the file reading efficiency.

[0064] Example 3

[0065] Figure 4 This is a flowchart of another file saving method provided by embodiment 3 of the present invention. This embodiment of the present invention is a refinement of the above embodiment, and describes the process of first performing a fragmentation operation and then performing a compression operation on the stored file. Figure 4 The method provided in the embodiment of the present invention specifically includes the following steps:

[0066] Step 310: Determine the execution order of the compression operation and the fragmentation operation according to the file storage size of the file to be stored.

[0067] Step 320: Read the file information of the file to be stored, and generate at least one logical shard information according to the file information, wherein the logical shard information at least includes the file size and the file name.

[0068] The file information may be information indicating the attributes of the file to be stored, and may include file attribute parameters such as file size and file name. The logical shard information may be information used to perform a sharding operation on the file to be stored, and may include shard file information of a file shard of the file to be stored. The logical shard information may be determined by the shard size of each file shard.

[0069] In an embodiment of the present invention, when a file to be stored is obtained, file information of the file to be stored can be extracted, and one or more logical shard information can be generated according to the file information. Each logical shard information can include the shard size of the corresponding file shard and identification information of the file shard. The generation process of the logical shard information can be determined based on the file information. For example, the file to be stored can be logically divided into multiple file shards with the same shard size according to the file size of the file information, and corresponding identification information can be configured for each file shard. The identification information can reflect that each file shard belongs to the corresponding file part of the file to be stored. For example, the identification information can include the shard serial number of the file shard, the resource identification of the file shard, etc.

[0070] Step 330: Determine the storage location of the file to be stored, and set at least one processing thread according to the logical sharding information and the storage location.

[0071] Among them, the processing thread can be the smallest computing unit for processing the file to be stored, multiple processing threads can be executed in parallel, and the processing thread can execute the assigned tasks. For example, the processing thread can perform compression operations and / or sharding operations according to the assigned logical sharding information.

[0072] In an embodiment of the present invention, the storage location of the file to be stored can be obtained, one or more processing threads can be created, and each processing thread can be initialized according to the determined logical sharding information and storage location, so that the processing thread processes the file to be stored according to the corresponding storage location and logical sharding information. It can be understood that when there are multiple processing threads, the logical sharding information and storage location configured in each processing thread can be different, and each processing thread can process a part of the file data corresponding to the logical sharding information in the file to be stored, and different processing threads can be responsible for processing different file data.

[0073] Step 340: Based on the sparse file of the file to be stored written by the processing thread at the storage location, the file fragments corresponding to the logical fragment information in the file to be stored are read according to the fragmentation operation, and the streaming compression rule is called through the compression operation to compress the file fragments.

[0074] In an embodiment of the present invention, each processing thread can write a sparse file in its respective configured storage location. The sparse file can be an empty file. The processing thread can perform a sharding operation, and the processing thread reads the file shards in the file to be stored according to the logical sharding information configured by the processing thread, and performs a compression operation through the processing thread, so that the processing thread compresses the read file shards according to the streaming compression rules. It can be understood that when there are multiple processing threads, each processing thread can read the file shards in the file to be stored according to its corresponding logical sharding information. When the logical sharding information configured by the processing thread is different, the file shards read by each processing thread may be different.

[0075] Step 350: Each processing thread transfers the corresponding file slice to a storage location, and adds the file slice to the sparse file according to the offset information.

[0076] Specifically, each processing thread can save the generated file shards to their respective configured storage locations, and the processing thread can add the sparse files pre-written with the shard data of the file shards according to the offset information of the file shards, so that the storage locations can save the file shards.

[0077] The embodiment of the present invention determines the execution order of compression and sharding operations according to the file storage scale of the file to be stored, reads file information in the file to be stored, and divides the file information into at least one logical sharding information, obtains the storage location of the file to be stored, creates at least one processing thread according to the storage location and the logical sharding information, writes a sparse file to the storage location through the processing thread, and reads the file shards corresponding to the logical sharding information in the file to be stored according to the sharding operation, executes compression operations according to the processing thread, compresses the file shards according to the streaming compression rules, and calls each processing thread to save the compressed file shards to the sparse file at the storage location according to the corresponding offset information. The embodiment of the present invention determines the execution order of compression and sharding operations based on the file storage scale of the file to be stored, and can select appropriate compression and sharding operation methods for files of different data storage scales, thereby improving the compression rate of the file; by determining multiple logical sharding information of the file to be stored and performing parallel processing on the file to be stored corresponding to each logical sharding information based on the processing thread, the compression efficiency of the file to be stored can be improved, which helps to improve the file storage efficiency of the file to be stored.

[0078] Furthermore, based on the above-mentioned embodiments of the invention, resource identification information of the file to be stored is obtained, and the storage path corresponding to the storage location is determined according to the resource identification information; at least one layer of folders and file names of the file fragments are generated according to the storage path, and the file fragments are saved to the folder with the file name.

[0079] The resource identification information may be a character string indicating a unique storage resource of the file to be stored. The storage path may be information indicating the location of the storage location in the distributed system, and the storage path may be determined by the resource identification information.

[0080] In an embodiment of the present invention, the resource identification information of the file to be stored can be extracted, and a storage path corresponding to the storage location can be generated according to the resource identification information. For example, the resource identification information can be used to construct a storage path for storing file fragments. The storage path can be composed of the storage locations of one or more layers of folders. A corresponding folder can be created according to the storage path and a file name for the file fragment can be generated. The fragment file can be saved to the created folder according to the file name, thereby realizing the preservation of the fragment file. Exemplarily, the resource ID of the file to be stored is 00112233, and the fragment number of the file fragment is 1. The file name of the fragment file created according to the resource ID is 00112233.lz4.1, wherein 'lz4' can indicate that the fragment file is compressed with lz4, '.1' can indicate the fragment sequence number of the fragment file, and the storage path can be / storage / 00 / 00112233.lz4.1, wherein / storage can indicate the storage location. / 00 may be a folder generated based on a resource ID, and 00112233.lz4.1 may represent the file name of a fragment file.

[0081] In some embodiments of the invention, it also includes: determining a target storage location in the distributed system according to the resource identification information of the file to be stored; reading and decompressing the file fragments in sequence according to the file name at the target storage location; and combining the decompressed file fragments into a target file.

[0082] In an embodiment of the present invention, the target storage location for storing file slices can be searched within a distributed system according to the resource identification information of the file to be stored. Each file slice can be read sequentially according to the file name at the target storage location and the file slices can be decompressed. The decompressed file slices can be combined into a target file, thereby achieving reading of the stored file. It is understood that the above-mentioned reading and decompression process of the file slices can be implemented in parallel through threads. For example, a separate thread can be set for each file slice stored in the target storage location, and each file slice can be read and decompressed by the thread, which can improve the reading efficiency of the file slices.

[0083] Example 4

[0084] The embodiment of the present invention provides a file storage method for static archiving data in a distributed system. The storage nodes in the distributed system can be pre-configured. The storage nodes can describe the storage location of the storage nodes through a configuration file. The configuration file can be specifically: root@delta01n00:~#cat / etc / iaas / storage_location.yaml

[0085] - / storage, that is, to create a / storage directory for the storage node delta01n00's storage location, which can be accessed through root@delta01n00:~#df-h / storage

[0086] Filesystem Size Used Avail Use%Mounted on

[0087] / dev / sdf1 3.6T 1.4T 2.1T 41% / storage

[0088] Query the total storage space, remaining storage space, and used storage space in the / storage directory.

[0089] In an embodiment of the present invention, the folder name in the storage location can be generated according to a specified rule. The specified rule may include determining the folder name to be stored based on the resource identifier. For example, the first N characters of the resource identifier can be used as the folder name. Taking the first two characters of the resource identifier as an example, assuming that the resource identifier of a resource is 00112233, then 00 can be used as the folder name where the resource is located. The number of folder layers can be expanded. Taking a two-layer folder structure as an example, the first a+b characters of the resource identifier of the above resource can be removed to construct the folder name. Assuming that the resource identifier of a resource is ab0143fd, then ab and 01 can be used as folder names respectively. The first-layer folder name can be ab, and the second-layer folder name can be 01. The resource will be saved in / storage / ab / 01 / . Furthermore, the specific rules for generating the folder structure and folder name for saving resources based on the resource identifier can be customized as needed. By establishing a unified folder name rule through the specific rule, it is convenient to quickly find the saved resources by the folder name, which can improve the query efficiency and reduce the resource query volume.

[0090] See also Figure 5In this distributed system, compute node 1 can create a virtual machine. When the corresponding system image file is not available, it will pull the image file data from the storage node, prioritizing storage nodes in the same availability zone to avoid cross-zone transmission (which has high network latency and low bandwidth). The virtual machine on compute node 1 has been running for a period of time and has generated a lot of business data. The customer needs to back up the data. The backup file will be compressed, fragmented, and then distributed and stored in the storage nodes in the same availability zone.

[0091] In an embodiment of the present invention, the storage sizes of the mirror file and the backup file are different, especially the size difference of the backup file is more obvious. For example, during the first backup, the generated backup file 1 is relatively large, which may be 100G, but during the second backup, the generated backup file 2 is an incremental backup, which only backs up the difference. Therefore, the size of backup file 2 may be only 10M. In order to cope with the situation where the file size difference is large, smaller files can obtain a higher compression ratio and larger files can obtain higher read and write efficiency. In this embodiment of the present invention, different storage methods can be adopted for files of different storage sizes. Method 1 can be used to process files smaller than 2G, and method 2 can be used to process files greater than or equal to 2G.

[0092] In an exemplary embodiment, see Figure 6 An embodiment of the present invention provides a file saving method. In this embodiment of the present invention, the file reading, compression, and writing process can be implemented based on Linux pipelines. This process can be streamed and no additional intermediate files are generated to occupy storage space. The file saving method can save the fragmented data of the storage file to the storage node. The file saving method can save the fragmented file read by the storage node to a local target file. The file saving method may include the following steps:

[0093] S1.1: Read files, read original data files, for example, read the backup original files of the virtual machine in the computing node.

[0094] S1.2: Stream compression, compressing the data. In the embodiment of the present invention, lz4 can be used for compression.

[0095] S1.3: Get the shard number. This step starts the process of writing data, streaming the compressed data. The shard size of the embodiment of the present invention is 1G. Determine the first shard, read 1G of data, and transmit the compressed data to the storage node; then determine the second shard, continue to read 1G of data, and transmit the compressed data to the storage node; repeat this process until all the data is read. The last shard may not read 1G of data. Note that this is streaming reading and writing. The solution of the present invention is only for example and not limited. The shards can be read in the form of 'one-time reading' or 'streaming reading'. After each shard of 1G is read, the next shard will be processed.

[0096] S1.4: Obtain the storage location for the shard. Specifically, obtain the remaining free space of all storage nodes in the availability zone, sort them, and select the one with the largest remaining free space. If there are multiple, select a random one. In principle, cross-zone storage should not be used. If the storage space of the local storage node is insufficient, expand the storage space.

[0097] S1.5: Write sparse files to the storage location. This step is used to take up space. The actual data size of the sparse file is 0. After using the sparse file to occupy the storage space, the "statistical data of the remaining space" can be more accurate, allowing other parallel tasks to obtain more accurate information. Sparse files can store compressed data of shard files. The name of the shard file is the suffix of the resource id plus '.digit'. For example, if the resource id is 00112233 and the shard number is 1, then the shard file name is 00112233.lz4.1, where '.lz4' indicates lz4 compression and '.1' is the shard sequence number. The file path of the resource in the storage node is / storage / 00 / 00112233.lz4.1.

[0098] S1.6: Write the shard data to the storage location. The compressed data is transmitted to the storage node and written to the sparse file in S5 according to the shard offset information. The transmission channel can be HTTP protocol, SSH protocol, etc., and is not limited by the embodiments of the present invention. The shard offset information means that the shard data may be transmitted in stages, such as the first stage transmitting data content from 0 to 10240 (excluding 10240), and the second stage transmitting data content from 10240 to 20480. The writing of different stages must be written to the sparse file according to the starting offset. The first stage writes the corresponding amount of data starting from offset 0, and the second stage writes the corresponding amount of data starting from offset 10240. As described in S3, after reading a shard size of 1GB, the next shard process will be entered and continue from S3. When the data is read, the process ends; otherwise, it continues from S3 to the next shard. In addition, due to network or disk load reasons, this step may fail and retransmission will be retried. After multiple failures, the task fails and the files related to the resource will be deleted from the storage node.

[0099] On this basis, see Figure 7 , you can also read the file data saved in the local storage location, the reading process may include the following steps:

[0100] S2.1: Obtain the storage locations of all shards. This can be done from the availability zone where the target resource resides. If data is insufficient in that zone, it can be retrieved from storage locations in other availability zones. The principle is to avoid pulling data across zones, but to ensure successful task execution. Because resource folders are defined using fixed naming conventions, finding the corresponding shard file path is straightforward and convenient.

[0101] S2.2: Read data in ascending order of sequence number, reading data from smallest to largest shard sequence number. The data transmission channel can be a protocol such as HTTP or SSH. This embodiment of the present invention does not limit the implementation of the data transmission channel. Read the shard data of the resource from the storage node.

[0102] S2.3: Streaming decompression: Decompress the data. In this embodiment, lz4 is used for decompression.

[0103] S2.4: Write to file. Write the decompressed data to the target file. After completion, you can use the target file.

[0104] Figure 8 This is an example diagram of another file saving method provided according to the fourth embodiment of the present invention. Figure 8, it first determines the logical shards, then performs concurrent processing on the data in each shard. Because concurrency is involved, there are two parts: the main program and the thread pool. The main program determines the relevant information of the logical shards and converts these logical shards into thread data, which is then handed over to the thread pool for consumption. The thread pool will start a new thread based on the number of concurrent connections and begin processing the shard data. The processing of shard data in the thread process uses Linux pipeline operations, and no intermediate files are generated during the process. The embodiments of the present invention can use Linux "pipelines" to implement the 'read-compress-write' process of shard data. This process is streaming processing and does not generate additional intermediate files to occupy storage space. The write process refers to the process of saving the data of each shard of the 'file to be stored' to the storage node, and the read process refers to the process of reading the shard file from the storage node and writing it to the local 'target file'. The thread pool in the write process and the thread pool in the read process are two different thread pools: one is responsible for writing shard data to the storage node, and the other is responsible for reading shard data from the storage node.

[0105] The main process of the file saving method may include the following steps:

[0106] S3.1. Read file information. This file is the file to be stored. Calculate logical shard information, create thread data for each shard, and place it in the thread pool. The file information read here primarily includes file size and file name information. The file name corresponds to the resource identifier. Subsequently, the shard information is determined based on the file size, the resource identifier is determined based on the file name, and the shard file name is determined based on the resource identifier.

[0107] S3.2. Generate thread data for each logical shard and store it in the thread pool. Assuming the file to be stored is 7.3GB in size and the shard size is 1GB, there will be eight logical shards, the last of which is 0.3GB. The actual calculations in the code are performed in bytes, but for ease of understanding, we will use gigabytes here. These eight shards will generate eight copies of thread data, which will be stored in the thread pool and await consumption.

[0108] Subsequently, the thread pool will consume the thread data of these 8 shards according to the number of concurrent connections. The thread data is similar to:

[0109]

[0110]

[0111] Where resource_id is the resource identifier, shard_size is the shard size (1GB), file_size is the original size of the file to be stored (7.3GB), shard_index is the shard number 1, which is the first shard, and shard_range is the data range of the shard (0, 1073741824), excluding the 1073741824th byte of data, which is 1GB. The starting number 0 indicates the start of the file to be stored at byte 0, and the ending number 1073741824 indicates the end of the file to be stored at (1073741824-1), which is 1073741823rd byte. Data byte numbering starts at 0.

[0112] S3.3, wait for consumption to complete, and recycle if failure occurs. The main process will wait for the data of these shards to be transferred. If a shard fails to be transferred, the task of saving the file to be stored will be abandoned, and the relevant shard data of the resource will be deleted at the storage node.

[0113] The concurrency of the thread pool can come from a preset configuration, such as configuring the concurrency based on the number of storage nodes, or it can be dynamically adjusted based on actual conditions, such as adjusting based on load conditions. This will not be expanded here.

[0114] The thread flow in the file saving method may include the following steps:

[0115] S4.1: Obtain the storage location for the shard. Obtain the remaining free space of all storage nodes in the availability zone, sort, and select the one with the largest remaining free space. If multiple storage nodes exist, select a random one. In principle, cross-zone storage should not be used. If the storage space of the local storage node is insufficient, expand the capacity.

[0116] S4.2: Write a sparse file to the storage location. This serves as a placeholder. The actual data size of the sparse file is 0. Using the sparse file to occupy storage space allows for more accurate statistics of remaining space, allowing other parallel tasks to obtain more precise information. This sparse file is the shard file. The shard file is named by appending a '.number' to the resource ID. For example, if the resource ID is 00112233 and the shard number is 1, the shard file name is 00112233.lz4.1, where '.lz4' indicates LZ4 compression and '.1' is the shard sequence number. The file path of this resource on the storage node is / storage / 00 / 00112233.lz4.1.

[0117] S4.3: Read the shard data, and read the corresponding data range according to the shard information, i.e., thread data. Assume that the shard information is

[0118]

[0119] The command line for reading shard data is similar to "dd if = path of file to be stored skip = 0 count = 1073741824 bs = 1 MB." The dd command reads the file to be stored and determines the skip and count values ​​based on the data range of the shard. The count is calculated by subtracting the start of the range from the end of the range, which is ba in shard_range(a,b).

[0120] S4.4: Stream compression, compressing the data. This embodiment uses lz4 for compression.

[0121] S4.5: Write the shard data to the storage location. The compressed data is transmitted to the storage node and written to the sparse file in S2 according to the shard offset information. The transmission channel can be HTTP, SSH, etc., and this solution has no restrictions. The shard offset information means that the shard data may be transmitted in stages. For example, the first stage transmits data content from 0 to 10240 (excluding 10240), and the second stage transmits data content from 10240 to 20480. The writing of different stages is written to the sparse file according to the starting offset. The first stage writes the corresponding amount of data starting from offset 0, and the second stage writes the corresponding amount of data starting from offset 10240. In addition, due to network or disk load reasons, this step may fail and retransmission will be retried. After multiple failures, the task fails and the transmission of other shard data is also stopped.

[0122] Based on the above embodiments of the invention, see Figure 9 The corresponding file reading process can also be implemented through the main process and thread. The main process in the file reading process may include the following steps:

[0123] S5.1. Create a sparse file. This sparse file is the target file used to receive data. The shard data read from the storage node will be written to this sparse file according to the data range of the shard. The file size and resource identifier information comes from the cloud platform's resource management service. The cloud platform has a relational database to record the management of resource lifecycles. This is not the focus of this solution, so it will not be discussed here. For example:

[0124]

[0125] S5.2. Generate thread data for each logical shard and place it in the thread pool. Assuming the file to be stored is 7.3GB in size and the shard size is 1GB, there will be eight logical shards, the last of which is 0.3GB. The actual calculations in the code are performed in bytes, but for ease of understanding, we will use gigabytes here. These eight shards will generate eight copies of thread data, which will be stored in the thread pool and await consumption. Subsequently, the thread pool will consume the thread data from these eight shards based on the number of concurrent connections. The thread data will look like this:

[0126]

[0127] Where resource_id is the resource identifier, shard_size is the shard size (1GB), file_size is the target file size (7.3GB), shard_index is the shard number (1), which is the first shard, and shard_range is the data range (0, 1073741824) for that shard, excluding the 1073741824th byte (i.e., the size is 1GB). The starting number 0 indicates the start of the target file at byte 0, and the ending number 1073741824 indicates the end of the target file at (1073741824-1), which is byte 1073741823. Byte numbering starts at 0.

[0128] S5.3, Waiting for consumption to complete. The main process will wait for the data transmission of these shards to be completed. If the data transmission of a shard fails, the task fails and the threads of other shards will also stop.

[0129] The concurrency of the thread pool can come from a preset configuration, such as configuring the concurrency based on the number of storage nodes, or it can be dynamically adjusted based on actual conditions, such as adjusting based on load conditions. This will not be expanded here.

[0130] The thread flow of the file reading process may include the following steps:

[0131] S6.1. Obtain the storage location of the shard. This step prioritizes obtaining the storage location from the availability zone where the target resource is located. If the zone does not have sufficient data, or if the shard file cannot be found, the shard file will be retrieved from storage locations in other availability zones. The principle is to avoid pulling data across zones, but the goal is to ensure the success of the task. Because the resource folder is determined by a fixed naming convention, the shard file path corresponding to the resource can be easily and directly found. Command line reference:

[0132] root@delta01n00:~#find / storage / 8a-type f-name'8aznrm6l*'

[0133] / storage / 8a / 8aznrm6l.lz4.2

[0134] / storage / 8a / 8aznrm6l.lz4.1

[0135] To find the fragment file associated with the resource ID 8aznrm6l, confirm that the folder name is 8a and then directly find the relevant fragment file in the folder. The fragment number in this step is confirmed, and the fragment file can be found.

[0136] S6.2. Read the shard data. The data transmission channel can be http, ssh, and other protocols, and this solution has no restrictions. Read the shard data of the resource from the storage node. Taking the ssh protocol as an example, the command line is similar to "ssh node cat 'shard file path'", such as: ssh delta01n00'cat

[0137] / storage / 8a / 8aznrm6l.lz4.1' reads shard data 8aznrm6l.lz4.1 from storage node delta01n00.

[0138] S6.3, Streaming decompression: Decompress the read fragmented data. In this embodiment, lz4 is used for streaming decompression.

[0139] S6.4. Write to the offset corresponding to the sparse file. The shard information contains a shard range, shard_range(a,b). The first number a represents the starting number of the target file, and the second number b represents the ending number of the target file, which is (b-1). The data size of the shard is (ba). a is counted starting from 0. For example, if the shard range of the target file is (0,1073741824), then starting from byte 0 of the target file, write (1073741824-0), or 1073741824 bytes of data, which is the original data of the shard file. The command line is similar to 'seek_write_file --ifile --seek starting number --ofile target file path'. '—ifile-' indicates reading data from the pipe, '—seek starting number' indicates starting writing data from that position, and '—ofile target file path' indicates writing to the target file. This is the significance of creating a sparse file before. After the sparse file is created, the file size is confirmed, and then the offset can be used to determine where to start writing data.

[0140] Example 5

[0141] Figure 10 This is a structural diagram of a file storage device provided according to embodiment 5 of the present invention. Figure 10 , the device is applied to a storage node of a distributed system, and the device may include:

[0142] The execution sequence module 410 is used to determine the execution sequence of the compression operation and the fragmentation operation according to the file storage size of the file to be stored.

[0143] The file sharding module 420 is configured to execute the sharding operation and the compression operation on the file to be stored respectively according to the execution order to obtain at least one file shard.

[0144] The fragment saving module 430 is used to store the file fragments in the storage location corresponding to the storage node.

[0145] In an embodiment of the present invention, the file storage scale of the file to be stored is obtained by executing a sequence module, and the execution order of the compression operation and the sharding operation corresponding to the file to be stored is determined according to the file storage scale. The file sharding module performs sharding and compression operations on the file to be stored in sequence according to the execution order, thereby obtaining at least one file shard. The shard saving module saves the file shard to the storage location corresponding to the storage node. In an embodiment of the present invention, the appropriate compression and sharding operation mode can be selected for files of different data storage scales, thereby facilitating the improvement of the file compression rate and the file saving speed, realizing the storage management of static archive data in the cloud platform, reducing the storage space occupied by static archive data, improving the reading and writing efficiency of stored data, and improving the service quality of the cloud platform.

[0146] Based on the above embodiments of the invention, the file segmentation module 420 includes:

[0147] The file reading unit is used to read the file to be stored.

[0148] The compression processing unit is used to perform the compression operation on the file to be stored according to a preset streaming compression rule.

[0149] The fragmentation execution unit is used to execute the fragmentation operation and use the currently generated compressed file data as a file fragment when the compressed file data generated by the compression reaches a data amount threshold.

[0150] Based on the above embodiments of the invention, the file segmentation module 420 further includes:

[0151] The logical sharding unit is configured to read the file information of the file to be stored and generate at least one logical sharding information according to the file information, wherein the logical sharding information at least includes the file size and the file name.

[0152] A thread setting unit is used to determine the storage location of the file to be stored and set at least one processing thread according to the logical sharding information and the storage location.

[0153] A thread processing unit is used to write the sparse file of the file to be stored in the storage location based on the processing thread, read the file fragment corresponding to the logical fragment information in the file to be stored according to the fragmentation operation, and compress the file fragment by calling the streaming compression rule through the compression operation.

[0154] Based on the above embodiments of the invention, the fragment storage module 430 includes:

[0155] A location determination unit is configured to locate at least one other storage node in the distributed system that belongs to the same available area as the storage node, and determine the storage locations of the storage node and the other storage nodes.

[0156] The sparse placeholder unit is used to write the sparse file of the file to be stored into the storage location.

[0157] The file storage unit is configured to write each of the file fragments into the sparse file according to the offset information.

[0158] Based on the above embodiment of the invention, the fragment storage module 430 further includes:

[0159] The thread saving unit is used to transfer the corresponding file fragments to the storage location through each processing thread, and add the file fragments to the sparse file according to the offset information.

[0160] Based on the above-mentioned embodiment of the invention, it also includes: a saving unit, used to obtain the resource identification information of the file to be stored, and determine the storage path corresponding to the storage location according to the resource identification information; generate at least one layer of folders and the file name of the file fragment according to the storage path, and save the file fragment to the folder with the file name.

[0161] In some embodiments of the invention, it also includes: a file reading module, used to determine the target storage location in the distributed system according to the resource identification information of the file to be stored; read the file fragments in sequence according to the file name at the target storage location and decompress them; and combine the decompressed file fragments into a target file.

[0162] The file storage device provided in the embodiment of the present invention can execute the file storage method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0163] Example 6

[0164] Figure 11 Schematic diagram of the structure of an electronic device that implements the file saving method of an embodiment of the present invention. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0165] like Figure 11 As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., which is communicatively connected to the at least one processor 11. The memory stores a computer program that can be executed by the at least one processor. The processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 into the random access memory (RAM) 13. Various programs and data required for the operation of the electronic device 10 can also be stored in the RAM 13. The processor 11, ROM 12, and RAM 13 are connected to each other via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0166] Multiple components in the electronic device 10 are connected to the I / O interface 15, including an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a magnetic disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0167] The processor 11 may be any general-purpose and / or specialized processing component with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various specialized artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The processor 11 executes the various methods and processes described above, such as the file saving method.

[0168] In some embodiments, the file saving method can be implemented as a computer program that is tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program can be loaded and / or installed on electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the file saving method described above can be performed. Alternatively, in other embodiments, processor 11 can be configured to perform the file saving method in any other suitable manner (e.g., by means of firmware).

[0169] Various embodiments of the systems and techniques described herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system-on-chip systems (SOCs), programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system that includes at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0170] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when the computer program is executed by the processor, the functions / operations specified in the flowcharts and / or block diagrams are implemented. The computer program may be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0171] In the context of the present invention, computer-readable storage media can be tangible media that can contain or store a computer program for use with an instruction execution system, device or equipment or used in combination with an instruction execution system, device or equipment. Computer-readable storage media can include but are not limited to electronic, magnetic, optical, electromagnetic, infrared or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, computer-readable storage media can be machine-readable signal media. More specific examples of machine-readable storage media can include electrical connections based on one or more lines, portable computer disks, hard disks, random access memories (RAM), read-only memories (ROM), erasable programmable read-only memories (EPROM or flash memory), optical fibers, portable compact disk read-only memories (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0172] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0173] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0174] A computing system may include clients and servers. The clients and servers are typically remote from each other and typically interact via a communication network. This client-server relationship arises through computer programs running on the respective computers, creating a client-server relationship. The server may be a cloud server, also known as a cloud computing server or cloud host. This server is a hosting product within the cloud computing service ecosystem that addresses the management difficulties and limited scalability of traditional physical hosting and VPS services.

[0175] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in the present invention can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solution of the present invention can be achieved. This is not limited herein.

[0176] The above specific embodiments do not limit the scope of protection of the present invention. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention are intended to be included within the scope of protection of the present invention.

Claims

1. A file saving method, characterized in that: Applied to a storage node of a distributed system, the method comprises: Determine the order of executing compression and sharding operations based on the file storage size of the files to be stored; Perform the sharding operation and the compression operation on the file to be stored respectively according to the execution order to obtain at least one file shard; The file segments are stored in the storage locations corresponding to the storage nodes.

2. The method according to claim 1, characterized in that The performing the sharding operation and the compression operation on the file to be stored respectively in the execution order to obtain at least one file shard includes: Reading the file to be stored; Performing the compression operation on the file to be stored according to a preset streaming compression rule; When the compressed file data generated by the compression reaches a data amount threshold, the fragmentation operation is performed to use the currently generated compressed file data as a file fragment.

3. The method according to claim 1, characterized in that The performing the sharding operation and the compression operation on the file to be stored respectively in the execution order to obtain at least one file shard includes: Reading file information of the file to be stored, and generating at least one logical shard information according to the file information, wherein the logical shard information includes at least a file size and a file name; Determine the storage location of the file to be stored, and set at least one processing thread according to the logical sharding information and the storage location; Based on the sparse file of the file to be stored written by the processing thread at the storage location, the file fragment corresponding to the logical fragment information in the file to be stored is read according to the fragmentation operation, and the streaming compression rule is called through the compression operation to compress the file fragment.

4. The method according to claim 2, characterized in that The storing the file fragment in a storage location corresponding to the storage node according to the resource identification information of the file fragment includes: at least one other storage node in the distributed system that belongs to the same availability zone as the storage node, and determining the storage locations of the storage node and the other storage nodes; Writing the sparse file of the file to be stored into the storage location; Each of the file fragments is written into the sparse file according to the offset information.

5. The method according to claim 3, characterized in that: The storing the file fragment in a storage location corresponding to the storage node according to the resource identification information of the file fragment includes: The corresponding file slices are respectively transmitted to the storage location by each processing thread, and the file slices are added to the sparse file according to the offset information.

6. The method according to claim 4 or 5, characterized in that The storage of the file fragments at the storage location includes: Obtain resource identification information of the file to be stored, and determine a storage path corresponding to the storage location according to the resource identification information; At least one layer of folders and file names of the file segments are generated according to the storage path, and the file segments are saved in the folders with the file names.

7. The method according to claim 6, characterized in that Also includes: Determining a target storage location in the distributed system according to the resource identification information of the file to be stored; Read the file segments in sequence according to the file names at the target storage location and decompress them; The decompressed file segments are combined into a target file.

8. A file storage device, characterized in that: A storage node applied to a distributed system, the device comprising: An execution sequence module is used to determine the execution sequence of compression operations and fragmentation operations according to the file storage size of the file to be stored; A file sharding module, configured to perform the sharding operation and the compression operation on the file to be stored respectively according to the execution order, to obtain at least one file shard; The fragment saving module is used to store the file fragments in the storage location corresponding to the storage node.

9. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor. The computer program is executed by the at least one processor so that the at least one processor can execute the file saving method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the file saving method according to any one of claims 1 to 7 when executed.

Images