A fragmented upload method, device and medium for a distributed storage system

Abstract

The present application discloses a fragment upload method, device and medium for a distributed storage system, which relates to the field of distributed storage, improves the upload efficiency of fragmented small files, receives a fragment upload request issued by a client, creates a fragmented small file, writes the fragment data of the fragment upload request in the fragmented small file, records the fragment number and fragment size information of the fragmented small file in a temporary index file, and after the fragment data upload is completed, creates an empty large file, the size of the large file is the actual size of the fragment data, reads the fragment information recorded in the temporary index file and updates it to the metadata of the large file. The uploaded fragment data is stored in the corresponding fragmented small file, and the fragment number and fragment size information of the corresponding fragmented small file are recorded in the metadata of the large file. There is no need to combine small files, which reduces the number of copies and improves the efficiency of fragment upload.

Description

Technical Field

[0001] The present application relates to the field of distributed storage, and in particular to a fragmented uploading method, device and medium for a distributed storage system. Background Art

[0002] In distributed storage systems, in order to adapt to the object protocol shard upload function and achieve lossless compatibility with the object protocol, a sharded small file merging solution has been implemented to solve the problem that the shard size is not fixed and large files cannot be appended. That is, after all shard data is uploaded, the sharded small files are read one by one according to the shard number from small to large, and then appended to the large file until the large file data is complete. Then, the sharded small files are deleted to support the object shard upload function.

[0003] Since the files need to be copied again when combining the fragmented small files into a large file, the efficiency of fragmented upload is reduced. When the fragmented small files are combined into a large file, normal reading / writing cannot be performed, resulting in poor performance.

[0004] It can be seen that how to solve the problem of low efficiency in uploading fragmented small files is a technical problem that needs to be solved urgently by people in this field. Summary of the Invention

[0005] The purpose of this application is to provide a fragmented uploading method, device and medium for a distributed storage system that improves the efficiency of uploading fragmented small files.

[0006] To solve the above technical problems, the present application provides a fragmented upload method for a distributed storage system, comprising:

[0007] Receive the multi-segment upload request sent by the client;

[0008] Create sharded small files;

[0009] Write the fragment data of the fragment upload request into the fragment file;

[0010] Record the fragment number and fragment size information of the fragmented small file in the temporary index file;

[0011] When the shard data is uploaded, an empty large file is created. The size of the large file is the actual size of the shard data.

[0012] Read the shard information recorded in the temporary index file and update it to the metadata of the large file.

[0013] Preferably, in the above-mentioned fragmented upload method of the distributed storage system, creating fragmented small files includes:

[0014] Parse the upload path information and upload number information of the multi-part upload request;

[0015] Create a hidden directory named after the upload number in the corresponding path according to the upload path information;

[0016] Create small fragment files in a hidden directory.

[0017] Preferably, in the above-mentioned multi-slice upload method of the distributed storage system, after parsing the upload path information and the upload number information of the multi-slice upload request, and before creating a hidden directory named with the upload number information under the corresponding path according to the upload path information, the method further includes:

[0018] Determine whether there are fragmented small files in the corresponding path of the upload path information;

[0019] If not, proceed to the step of creating a hidden directory named with the upload number information under the corresponding path according to the upload path information.

[0020] Preferably, in the above-mentioned fragmented uploading method of the distributed storage system, if the upload path information indicates that there are fragmented small files under the corresponding path, the method further includes:

[0021] Returns an error code indicating that the file exists to the client.

[0022] Preferably, in the above-mentioned fragmented upload method of the distributed storage system, after reading the fragmented information recorded in the temporary index file and updating it into the metadata of the large file, the method further includes:

[0023] Persistent metadata.

[0024] Preferably, in the above-mentioned fragmented upload method of the distributed storage system, after reading the fragmented information recorded in the temporary index file and updating it into the metadata of the large file, the method further includes:

[0025] Receive read / write requests for large files;

[0026] Open the large file and apply for read / write permissions for the large file;

[0027] Find the fragment number of the corresponding small file in the metadata of the large file based on the read / write start position and read / write length;

[0028] Concatenate the paths corresponding to the fragmented small files according to the fragment numbers;

[0029] Open the sharded small file in read / write mode;

[0030] When the data reading / writing of the sharded small file is completed, the read / write result data is returned and the large file and the sharded small file are closed.

[0031] Preferably, in the above-mentioned fragmented upload method of the distributed storage system, opening the fragmented small file in a read / write mode includes:

[0032] After the current shard file is read / written, determine whether the next shard file needs to be read / written;

[0033] If yes, close the current fragment file, connect the path of the next fragment file and open it, and read / write the next fragment file;

[0034] If not, close the current sharded small file, proceed to the step of returning the read / write result data, and close the large file and the sharded small file.

[0035] To solve the above technical problems, the present application further provides a fragment upload device for a distributed storage system, comprising:

[0036] The receiving module is used to receive the fragment upload request sent by the client;

[0037] Create sharded small file module, used to create sharded small files;

[0038] The writing module is used to write the fragment data of the fragment upload request into the fragment small file;

[0039] The recording module is used to record the fragment number and fragment size information of the fragmented small file into the temporary index file;

[0040] The large file creation module is used to create an empty large file after the fragment data is uploaded. The size of the large file is the actual size of the fragment data.

[0041] The update module is used to read the shard information recorded in the temporary index file and update it to the metadata of the large file.

[0042] The module for creating fragmented small files includes: a parsing subunit for parsing the upload path information and upload number information of the fragment upload request;

[0043] Create a directory subunit, which is used to create a hidden directory named with the upload number information under the corresponding path according to the upload path information;

[0044] Create a fragment file subunit, which is used to create fragment small files in a hidden directory.

[0045] The first judgment subunit is used to judge whether there are fragmented small files under the corresponding path of the upload path information;

[0046] If not, the directory creation subunit is triggered to create a hidden directory named with the upload number information under the corresponding path according to the upload path information.

[0047] If so, the return subunit is triggered to return an error code indicating that the file exists to the client.

[0048] A read / write request receiving module is used to receive read / write requests for large files;

[0049] The permission application module is used to open large files and apply for read / write permissions for large files;

[0050] The search module is used to find the fragment number of the corresponding fragmented small file in the metadata of the large file according to the read and write start position and read and write length;

[0051] The splicing module is used to splice the paths corresponding to the fragmented small files according to the fragment numbers;

[0052] Open module, used to open small fragment files in read / write mode;

[0053] The data return module is used to return the read / write result data and close the large file and the sharded small file after the data reading / writing of the sharded small file is completed.

[0054] The opening module includes: a second judgment subunit, which is used to judge whether it is necessary to read / write the next fragmented file after the reading / writing of the current fragmented file is completed;

[0055] If so, the continuation sub-unit is triggered to close the current fragment file, splice the path of the next fragment file and open it, and read / write the next fragment file;

[0056] If not, the return data module is triggered to close the current sharded small file, enter the return read / write result data, and close the large file and sharded small file.

[0057] To solve the above technical problems, the present application further provides a fragment upload device for a distributed storage system, comprising:

[0058] memory for storing computer programs;

[0059] The processor is used to implement the steps of the above-mentioned distributed storage system segmented uploading method when executing the computer program.

[0060] In order to solve the above technical problems, the present application also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the sharded uploading method of the above distributed storage system are implemented.

[0061] The fragment upload method of the distributed storage system provided by the present application includes: receiving a fragment upload request sent by a client; creating a fragmented small file; writing the fragment data of the fragment upload request in the fragmented small file; recording the fragment number and fragment size information of the fragmented small file in a temporary index file; when the fragment data is uploaded, creating an empty large file, the size of the large file is the actual size of the fragment data; reading the fragment information recorded in the temporary index file and updating it to the metadata of the large file. The uploaded fragment data is stored in the corresponding fragmented small file, and the fragment number of the corresponding fragmented small file and the fragment size information of the fragment data are recorded in the metadata of the large file. There is no need to combine the fragmented small files, which reduces the number of copies and improves the efficiency of fragment upload. The method of recording fragment metadata instead of fragment merging can solve the performance problems caused by the combination of fragmented small files. At the same time, this implementation method helps to improve the reading and writing concurrency efficiency of large files and improve the reading and writing performance of large files.

[0062] In addition, the present application also provides a fragmented uploading device and medium for a distributed storage system, which corresponds to the fragmented uploading method of the above-mentioned distributed storage system and has the same effect as above. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] In order to more clearly illustrate the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0064] Figure 1 A flowchart of a fragment uploading method for a distributed storage system provided in an embodiment of the present application;

[0065] Figure 2 A structural diagram of a shard upload device for a distributed storage system provided in an embodiment of the present application;

[0066] Figure 3 A structural diagram of another distributed storage system fragment upload device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0067] The following will be combined with the accompanying drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0068] The core of this application is to provide a fragmented uploading method, device and medium for a distributed storage system.

[0069] In order to enable those skilled in the art to better understand the present application, the present application is further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0070] In the era of the Internet of Things, cloud storage and rapid growth of intelligent data, under the mature fusion technology architecture, a data file storage, object storage and big data storage protocols have been realized to share access, solving the problems of data operation efficiency and maintenance difficulty; in order to solve the problems of poor upload performance of large files, file retransmission efficiency in poor network environment or object upload performance when the file size is unknown, the object protocol adopts the fragmented upload method to solve the above three problems. Since the fusion method is based on file storage, a fragmented data uploaded by the object corresponds to a fragmented small file in the distributed system, and the size of each fragmented small file is determined by the client using the object protocol. The fragment size is not fixed. When all the fragmented data are uploaded, the storage in the distributed system is independent. Independent fragmented small files, that is, uploading partial data of a large file in fragments, cannot support operations such as reading, writing, and deleting the large file. To adapt to the object protocol fragmented upload function and achieve lossless compatibility with the object protocol, a fragmented small file merging solution has been implemented to solve the problem of inability to append large files due to unstable fragment sizes. Specifically, after all fragmented data is uploaded, the fragmented small files are read one by one in ascending fragment number order and then appended to the large file until the large file data is complete. The fragmented small files are then deleted to support the object fragmented upload function. However, because the fragmented small files need to be copied when they are combined into a large file, the efficiency of fragmented upload is reduced. When the fragmented small files are combined into a large file, normal read / write operations cannot be performed, resulting in poor performance. In addition, the concurrent access performance of multiple protocols is poor when the fragments are combined, resulting in low concurrent access efficiency.

[0071] Based on the above problems, the present invention proposes a fragmented upload method for a distributed storage system. Figure 1 A flowchart of a fragment upload method for a distributed storage system provided in an embodiment of the present application is shown as follows: Figure 1 As shown, including:

[0072] S11: Receive the multi-segment upload request sent by the client;

[0073] S12; create fragmented small files;

[0074] S13: Write the fragment data of the fragment upload request into the fragment file;

[0075] S14: Record the fragment number and fragment size information of the fragmented small file into the temporary index file;

[0076] S15: After the shard data is uploaded, an empty large file is created. The size of the large file is the actual size of the shard data.

[0077] S16: Read the fragment information recorded in the temporary index file and update it to the metadata of the large file.

[0078] In this embodiment, the server can receive the fragment upload request. The server may include a hardware device with a data information processing function and the necessary software required to drive the hardware device to work. The server can analyze the received fragment upload request to obtain the upload path information and upload number information of the fragment file contained in the file upload request. The server can provide a predetermined port and receive the fragment upload request sent by the client through the predetermined port. In this embodiment, the client can be a terminal device that can access a communication network based on a network protocol. The client can also be an upper-layer application with a file upload function, such as a cloud storage application, an enterprise office automation system, etc. The client fragments the file, and step S11 receives the fragment upload request issued by the client, and creates a fragment small file according to the fragment upload request for storing the fragment to be uploaded. Preferably, step S12 creates the fragment small file, including:

[0079] Parse the upload path information and upload number information of the multi-part upload request;

[0080] Create a hidden directory named after the upload number in the corresponding path according to the upload path information;

[0081] Create small fragment files in a hidden directory.

[0082] According to the upload path information and upload number information contained in the multipart upload request, a hidden directory named after the upload number information is created under the corresponding path, and small multipart files are created under the hidden directory.

[0083] After the sharded small file is created, step S13 writes the sharded data of the sharded upload request into the sharded small file. When receiving and writing the uploaded sharded data, step S14 records the shard number and shard size information of the sharded small file into the temporary index file, that is, simultaneously records the shard size information uploaded to the sharded small file and the corresponding shard number into the temporary index file.

[0084] After all the fragmented data is uploaded, an empty large file is created. The size of the large file is the actual size of the fragmented data. In this embodiment, the large file refers to a file with a large file size, such as several GB or dozens of GB. The specific size of the large file can be set as needed. The small fragmented file refers to a file with a smaller file size. The specific setting range of the small fragmented file size can also be set by the user.

[0085] When the large file is created, step S16 reads the fragment information recorded in the temporary index file and updates it to the metadata of the large file, and the fragment upload is completed.

[0086] Specifically, the fragment upload method of the distributed storage system provided in this embodiment includes: receiving a fragment upload request sent by a client; creating a fragmented small file; writing the fragment data of the fragment upload request in the fragmented small file; recording the fragment number and fragment size information of the fragmented small file in a temporary index file; when the fragment data is uploaded, creating an empty large file, the size of the large file is the actual size of the fragment data; reading the fragment information recorded in the temporary index file and updating it to the metadata of the large file. The uploaded fragment data is stored in the corresponding fragmented small file, and the fragment number of the corresponding fragmented small file and the fragment size information of the fragment data are recorded in the metadata of the large file. There is no need to combine the small files, which reduces the number of copies and improves the efficiency of fragment upload. The method of recording fragment metadata instead of fragment merging can solve the performance problems caused by the combination of fragmented small files. At the same time, this implementation method helps to improve the read and write concurrency efficiency of large files and improve the read and write performance of large files.

[0087] According to the above embodiment, before creating a fragmented small file, it is possible to first determine whether the fragmented file to be uploaded already exists in the server. If the server informs the client that the fragmented file already exists, it means that another client has already uploaded the fragment, and the client does not need to upload it again, which can improve the upload efficiency of the target fragment, and the server does not need to repeatedly create the fragmented small file, which can also save the user's network traffic. Preferably, in the fragmented upload method of the above distributed storage system, after parsing the upload path information and upload number information of the fragmented upload request, before creating a hidden directory named with the upload number information under the corresponding path according to the upload path information, it also includes:

[0088] Determine whether there are fragmented small files in the corresponding path of the upload path information;

[0089] If it does not exist, proceed to the step of creating a hidden directory named with the upload number information under the corresponding path according to the upload path information.

[0090] If it exists, an error code indicating that the file exists is returned to the client.

[0091] The server determines whether a small fragment file exists under the corresponding path based on the upload path information. When the server determines whether a small fragment file exists or does not exist under the corresponding path based on the upload path information, it can return a reply message, which includes information about whether a small fragment file exists or does not exist under the corresponding path based on the upload path information.

[0092] After the server determines that the upload path information does not contain a small fragment file in the corresponding path, it creates a hidden directory named after the upload number information in the corresponding path based on the upload path information, and creates the small fragment file in the hidden directory. After receiving the digital reply message indicating that the upload path information determines that the small fragment file does not exist in the corresponding path, the client can send the fragment data to the server. After receiving the fragment data, the server stores the fragment data in the small fragment file in the corresponding path of the upload path information.

[0093] If it is determined that a small fragment file exists under the corresponding path of the upload path information, an error code indicating that the file exists is returned to the client to indicate that the corresponding small fragment file already exists, thereby avoiding repeated uploading.

[0094] Before creating a fragmented file, the server first determines whether the fragmented file to be uploaded already exists on the server. If the server informs the client that the fragmented file already exists, it means that another client has uploaded the fragment. The client does not need to upload it again, which can improve the upload efficiency of the target fragment. The server does not need to repeatedly create fragmented files, and it can also save the user's network traffic.

[0095] According to the above embodiment, preferably, the shard upload method of the distributed storage system, after reading the shard information recorded in the temporary index file and updating it into the metadata of the large file, further includes:

[0096] Persistent metadata.

[0097] Persistence refers to the mechanism for converting program data between a persistent state and a transient state. In layman's terms, it means that transient data (such as data in memory, which cannot be permanently stored) is persisted as persistent data (such as data persisted in a database, which can be stored for a long time). Using data persistence has the following advantages: program code is highly reusable. Even if the database is changed, only the configuration file needs to be changed, and there is no need to rewrite the program code. Business logic code is highly readable, and there is no large amount of Structured Query Language (SQL) in the code, which improves the readability of the program. Persistence technology can automatically optimize to reduce the number of database accesses and improve program operation efficiency.

[0098] According to the above embodiment, after the shard data upload is completed, normal read / write operations can be performed on the large file. Therefore, this embodiment provides a specific implementation scheme, a shard upload method for a distributed storage system, which reads the shard information recorded in the temporary index file and updates it to the metadata of the large file, and further includes:

[0099] Receive read / write requests for large files;

[0100] Open the large file and apply for read / write permissions for the large file;

[0101] Find the fragment number of the corresponding small file in the metadata of the large file based on the read / write start position and read / write length;

[0102] Concatenate the paths corresponding to the fragmented small files according to the fragment numbers;

[0103] Open the sharded small file in read / write mode;

[0104] When the data reading / writing of the sharded small file is completed, the read / write result data is returned and the large file and the sharded small file are closed.

[0105] After receiving the read / write request for a large file sent by the client, the corresponding large file is opened and the read / write permission of the large file is applied. After obtaining the read / write request permission, the fragment number of the corresponding fragmented small file in the metadata of the large file is found according to the read / write starting position and read / write length of the read / write request, the path corresponding to the fragmented small file is spliced ​​according to the fragment number, and the fragmented small file is opened in read / write mode. Until the data reading / writing of the fragmented small file is completed, the read / write result data is returned to the client, and the large file and the fragmented small file are closed. The reading / writing of the large file is completed.

[0106] In addition, it should be noted that in this embodiment, if a read / write request includes reading / writing multiple sharded small files, the multiple sharded small files can be read / written sequentially or simultaneously, improving the read / write performance of sharded uploaded files. Multiple sharded small files can be read / written concurrently, improving the concurrency of large files. By recording shard information in the large file metadata and persisting it, the performance overhead and concurrent access issues caused by shard merging are replaced, achieving lossless support for the object protocol sharded upload feature and improving the read / write performance of sharded uploaded files.

[0107] According to the above embodiment, preferably, multiple shard files are read / written in sequence to ensure data accuracy, and the shard files are opened in a read / write mode, including:

[0108] After the current shard file is read / written, determine whether the next shard file needs to be read / written;

[0109] If yes, close the current fragment file, connect the path of the next fragment file and open it, and read / write the next fragment file;

[0110] If not, close the current sharded small file, proceed to the step of returning the read / write result data, and close the large file and the sharded small file.

[0111] After a small shard file is read or written, a check is performed to determine whether the next small shard file needs to be read or written. If so, the current small shard file is closed, the path to the next small shard file is concatenated and opened, and the next small shard file is read or written. If not, the current small shard file is closed, and the read / write result data is returned, and the large file and small shard files are closed to complete the read / write request. By recording and persisting shard information in the large file metadata, the performance overhead and concurrent access issues associated with shard merging are eliminated, achieving lossless support for the object protocol's shard upload feature and improving the read and write performance of uploaded shard files.

[0112] In the above embodiments, a method for uploading shards to a distributed storage system is described in detail. This application also provides corresponding embodiments of a device for uploading shards to a distributed storage system. It should be noted that this application describes the embodiments of the device from two perspectives: one is based on the functional module perspective, and the other is based on the hardware perspective.

[0113] Figure 2 A structural diagram of a fragment upload device for a distributed storage system provided in an embodiment of the present application is shown as follows: Figure 2 As shown, including:

[0114] Receiving module 21, used to receive the fragment upload request sent by the client;

[0115] A fragmented small file creation module 22 is used to create fragmented small files;

[0116] The writing module 23 is used to write the fragment data of the fragment upload request into the fragment small file;

[0117] Recording module 24, used to record the fragment number and fragment size information of the fragmented small file into the temporary index file;

[0118] A large file creation module 25 is used to create an empty large file after the fragment data is uploaded. The size of the large file is the actual size of the fragment data.

[0119] The updating module 26 is used to read the fragmentation information recorded in the temporary index file and update it into the metadata of the large file.

[0120] Specifically, the receiving module 21 receives the fragment upload request sent by the client; the fragment creation module 22 creates the fragment small file; the writing module 23 writes the fragment data of the fragment upload request into the fragment small file; the recording module 24 records the fragment number and fragment size information of the fragment small file into the temporary index file; the large file creation module 25 creates an empty large file after the fragment data upload is completed, and the size of the large file is the actual size of the fragment data; the updating module 26 reads the fragment information recorded in the temporary index file and updates it to the metadata of the large file. The uploaded fragment data is stored in the corresponding fragment small file, and the fragment number of the corresponding fragment small file and the fragment size information of the fragment data are recorded in the metadata of the large file. There is no need to combine the small files, and the large file can be read / written normally, which reduces the number of copies and improves the efficiency of fragment upload. Using the method of recording fragment metadata instead of fragment merging can solve the performance problems caused by the combination of fragment small files. At the same time, this implementation method helps to improve the read and write concurrency efficiency of large files and improve the read and write performance of large files.

[0121] Preferably, it also includes:

[0122] The module for creating fragmented small files includes: a parsing subunit for parsing the upload path information and upload number information of the fragment upload request;

[0123] Create a directory subunit, which is used to create a hidden directory named with the upload number information under the corresponding path according to the upload path information;

[0124] Create a fragment file subunit, which is used to create fragment small files in a hidden directory.

[0125] The first judgment subunit is used to judge whether there are fragmented small files under the corresponding path of the upload path information;

[0126] If not, the directory creation subunit is triggered to create a hidden directory named with the upload number information under the corresponding path according to the upload path information.

[0127] If so, the return subunit is triggered to return an error code indicating that the file exists to the client.

[0128] A read / write request receiving module is used to receive read / write requests for large files;

[0129] The permission application module is used to open large files and apply for read / write permissions for large files;

[0130] The search module is used to find the fragment number of the corresponding fragmented small file in the metadata of the large file according to the read and write start position and read and write length;

[0131] The splicing module is used to splice the paths corresponding to the fragmented small files according to the fragment numbers;

[0132] Open module, used to open small fragment files in read / write mode;

[0133] The data return module is used to return the read / write result data and close the large file and the sharded small file after the data reading / writing of the sharded small file is completed.

[0134] The opening module includes: a second judgment subunit, which is used to judge whether it is necessary to read / write the next fragmented file after the reading / writing of the current fragmented file is completed;

[0135] If so, the continuation sub-unit is triggered to close the current fragment file, splice the path of the next fragment file and open it, and read / write the next fragment file;

[0136] If not, the return data module is triggered to close the current sharded small file, enter the return read / write result data, and close the large file and sharded small file.

[0137] Since the embodiments of the apparatus part correspond to the embodiments of the method part, please refer to the description of the embodiments of the method part for the embodiments of the apparatus part, and they will not be repeated here.

[0138] Figure 3 A structural diagram of another distributed storage system fragment upload device provided in an embodiment of the present application, such as Figure 3 As shown, the fragment upload device of the distributed storage system includes: a memory 30 for storing computer programs;

[0139] The processor 31 is configured to implement the steps of the method for obtaining user operation habit information in the above embodiment (sharded uploading method for a distributed storage system) when executing a computer program.

[0140] The fragment upload device of the distributed storage system provided in this embodiment may include but is not limited to a smart phone, a tablet computer, a laptop computer, or a desktop computer.

[0141] Among them, the processor 31 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 31 can be implemented in at least one hardware form of a digital signal processor (DSP), a field programmable gate array (FPGA), and a programmable logic array (PLA). The processor 31 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a central processing unit (CPU); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 31 may be integrated with a graphics processing unit (GPU), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 31 may also include an artificial intelligence (AI) processor, which is used to process computing operations related to machine learning.

[0142] The memory 30 may include one or more computer-readable storage media, which may be non-transitory. The memory 30 may also include a high-speed random access memory, and a non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In this embodiment, the memory 30 is at least used to store the following computer program 301, wherein, after the computer program is loaded and executed by the processor 31, it can implement the relevant steps of the fragmented upload method of the distributed storage system disclosed in any of the aforementioned embodiments. In addition, the resources stored in the memory 30 may also include an operating system 302 and data 303, etc., and the storage method may be temporary storage or permanent storage. Among them, the operating system 302 may include Windows, Unix, Linux, etc. The data 303 may include but is not limited to data involved in implementing the fragmented upload method of the distributed storage system, etc.

[0143] In some embodiments, the shard upload device of the distributed storage system may further include a display screen 32 , an input / output interface 33 , a communication interface 34 , a power supply 35 , and a communication bus 36 .

[0144] Those skilled in the art will understand that Figure 3 The structure shown in does not constitute a limitation on the segmented upload device of the distributed storage system, and may include more or fewer components than shown in the figure.

[0145] The fragment upload device of the distributed storage system provided by the embodiment of the present application includes a memory and a processor. When the processor executes the program stored in the memory, it can implement the following method: the fragment upload method of the distributed storage system includes: receiving a fragment upload request sent by the client; creating a fragment small file; writing the fragment data of the fragment upload request in the fragment small file; recording the fragment number and fragment size information of the fragment small file in a temporary index file; when the fragment data is uploaded, creating an empty large file, the size of the large file is the actual size of the fragment data; reading the fragment information recorded in the temporary index file and updating it to the metadata of the large file. The uploaded fragment data is stored in the corresponding fragment small file, and the fragment number and fragment size information of the corresponding fragment small file are recorded in the metadata of the large file. There is no need to combine the small files, and the large file can be read and written normally, which reduces the number of copies and improves the efficiency of fragment upload. Using the method of recording fragment metadata instead of fragment merging can solve the performance problems caused by the combination of fragment small files. At the same time, this implementation method helps to improve the read and write concurrency efficiency of large files and improve the read and write performance of large files.

[0146] Finally, the present application also provides an embodiment corresponding to a computer-readable storage medium. The computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps described in the embodiment of the sharded upload method for the distributed storage system.

[0147] It is understandable that if the method in the above embodiment is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and executes all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0148] The computer-readable storage medium provided in this embodiment stores a computer program. When the processor executes the program, the following method can be implemented: a fragmented upload method for a distributed storage system, comprising: receiving a fragmented upload request sent by a client; creating a fragmented small file; writing the fragmented data of the fragmented upload request in the fragmented small file; recording the fragment number and fragment size information of the fragmented small file in a temporary index file; after the fragmented data is uploaded, creating an empty large file, the size of the large file is the actual size of the fragmented data; reading the fragment information recorded in the temporary index file and updating it to the metadata of the large file. The uploaded fragmented data is stored in the corresponding fragmented small file, and the fragment number and fragment size information of the corresponding fragmented small file are recorded in the metadata of the large file. There is no need to combine small files, which reduces the number of copies and improves the efficiency of fragmented upload.

[0149] The above is a detailed introduction to the shard upload method, device and medium of the distributed storage system provided by the present application. The various embodiments in the specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same and similar parts between the various embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part description. It should be pointed out that for ordinary technicians in this technical field, without departing from the principles of this application, several improvements and modifications can be made to the present application, and these improvements and modifications also fall within the scope of protection of the claims of this application.

[0150] It should also be noted that, in this specification, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variants thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article, or apparatus comprising the element.

Claims

1. A fragmented upload method for a distributed storage system, characterized in that: include: Receive the multi-segment upload request sent by the client; Create sharded small files; Write the fragment data of the fragment upload request into the fragment small file; Record the fragment number and fragment size information of the fragmented small file in a temporary index file; When the fragment data is uploaded, an empty large file is created, and the size of the large file is the actual size of the fragment data; Read the fragment information recorded in the temporary index file and update it into the metadata of the large file; After reading the fragment information recorded in the temporary index file and updating it into the metadata of the large file, the method further includes: receiving a read / write request for the large file; Open the large file and apply for read / write permission for the large file; According to the read / write start position and the read / write length, the fragment number of the corresponding fragmented small file is searched in the metadata of the large file; Concatenate the paths corresponding to the fragmented small files according to the fragment numbers; Open the fragmented small file in read / write mode; When the data reading / writing of the fragmented small files is completed, the read / write result data is returned, and the large file and the fragmented small files are closed.

2. The fragmented uploading method of the distributed storage system according to claim 1 is characterized in that: The step of creating a fragmented small file includes: Parsing the upload path information and upload number information of the segment upload request; Creating a hidden directory named after the upload number information under the corresponding path according to the upload path information; The fragmented small files are created in the hidden directory.

3. The fragmented uploading method of the distributed storage system according to claim 2 is characterized in that: After parsing the upload path information and upload number information of the multipart upload request, and before creating a hidden directory named with the upload number information under the corresponding path according to the upload path information, the method further includes: Determine whether the fragmented small file exists under the corresponding path of the upload path information; If not, enter the step of creating a hidden directory named with the upload number information under the corresponding path according to the upload path information.

4. The fragmented uploading method of the distributed storage system according to claim 3 is characterized in that: If the upload path information indicates that the fragmented small file exists in the corresponding path, the method further includes: An error code indicating that the file exists is returned to the client.

5. The fragmented uploading method of the distributed storage system according to claim 1 is characterized in that: After reading the fragment information recorded in the temporary index file and updating it into the metadata of the large file, the method further includes: The metadata is persisted.

6. The fragmented upload method of the distributed storage system according to claim 5, characterized in that: The step of opening the small file in a read / write mode includes: If the reading / writing of the current fragmented small file is completed, determine whether it is necessary to read / write the next fragmented small file; If so, close the current fragment file, splice the path of the next fragment file and open it, and read / write the next fragment file; If not, close the current fragmented small file, enter the step of returning the read / write result data, and close the large file and the fragmented small file.

7. A fragment upload device for a distributed storage system, characterized in that: include: The receiving module is used to receive the fragment upload request sent by the client; Create sharded small file module, used to create sharded small files; A writing module, used for writing the fragment data of the fragment upload request into the fragment small file; A recording module is used to record the fragment number and fragment size information of the fragmented small file in a temporary index file; A large file creation module is used to create an empty large file after the fragment data is uploaded. The size of the large file is the actual size of the fragment data; An update module is used to read the fragment information recorded in the temporary index file and update it into the metadata of the large file; After reading the fragment information recorded in the temporary index file and updating it into the metadata of the large file, the method further includes: receiving a read / write request for the large file; Open the large file and apply for read / write permission for the large file; According to the read / write start position and the read / write length, the fragment number of the corresponding fragmented small file is searched in the metadata of the large file; Concatenate the paths corresponding to the fragmented small files according to the fragment numbers; Open the fragmented small file in read / write mode; When the data reading / writing of the fragmented small files is completed, the read / write result data is returned, and the large file and the fragmented small files are closed.

8. A fragment upload device for a distributed storage system, characterized in that: include: Memory for storing computer programs; A processor, configured to implement the steps of the sharded uploading method for a distributed storage system as described in any one of claims 1 to 6 when executing the computer program.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the sharded uploading method of the distributed storage system according to any one of claims 1 to 6.

Images