A file integrated upload method and device, a file download method and device

Abstract

The present invention discloses a method and device for integrated uploading of files, and a method and device for downloading files, wherein the integrated uploading method comprises: obtaining multiple first files sent by a client, and metadata information of the first files, wherein the metadata information of the first files comprises name information, length and content information; calculating and generating offsets corresponding to each first file according to the length of each first file; integrating and generating a second file according to the offset and metadata information of each first file; uploading and writing the second file to a disk for storage. By implementing the present invention, the problems of poor disk read / write performance and low efficiency of uploading small files caused by concurrent uploading of multiple small files in the related art are solved, and the metadata information and content information of the uploaded multiple small files are combined to integrate into a large file, thereby reducing the additional disk read / write overhead and improving the disk read / write performance.

Description

Technical Field

[0001] The present invention relates to the field of data storage, and in particular to a file integration uploading method and device, and a file downloading method and device. Background Art

[0002] Blu-ray storage is a cold storage technology that stores data through an object storage interface. When a file Blu-ray storage system reads and writes multiple small files, due to the metadata operation of the file system and the fact that mechanical disks are more suitable for sequential reading and writing, the results of testing the read and write performance of files show that when multiple small files are uploaded concurrently, the read and write performance is poor, resulting in poor disk read and write performance and low efficiency in uploading small files. Summary of the invention

[0003] In view of this, an embodiment of the present invention provides a file integration upload method and device, and a file download method and device to solve the problem of poor disk read and write performance and low efficiency of uploading small files caused by concurrent uploading of multiple small files.

[0004] According to the first aspect, an embodiment of the present invention provides a method for integrated uploading of files, including: obtaining multiple first files sent by a client, and metadata information of the first files, the metadata information of the first files including name information, length and content information; calculating and generating an offset corresponding to each of the first files according to the length of each of the first files; integrating and generating a second file according to the offset and metadata information of each of the first files; uploading the second file and writing it to a disk for storage.

[0005] In combination with the first aspect, in the first implementation of the first aspect, the offset corresponding to each first file is calculated and generated based on the length of each first file, including: based on the length of each first file and a preset unit storage capacity, the ratio of the length of the first file to the preset unit storage capacity is calculated, and the ratio is used as the offset of the first file.

[0006] In combination with the first aspect, in the second implementation of the first aspect, the method also includes: the generation of the second file based on the integration of the offset and metadata information of each of the first files, including: updating the offset of the pre-stored second file linked list according to the offset of the first file, and generating the offset of the updated second file linked list; storing the content information of each of the first files in a preset memory space, generating a memory pointer, and the memory pointer is used to represent the storage path information of the first file in the preset memory space; generating a first file linked list item for each of the first files based on the memory pointer, the metadata information of the first file, the offset of the first file and the offset of the updated second file linked list; inserting each of the first file linked list items into the second file linked list, integrating to generate a second file linked list item, and integrating each of the first files and the metadata information to generate the second file.

[0007] In combination with the second implementation of the first aspect, in the third implementation of the first aspect, uploading and writing the second file to the disk for storage includes: uploading and writing the second file to the disk for storage according to the second file linked list item.

[0008] In combination with the second implementation of the first aspect, in the fourth implementation of the first aspect, after the step of inserting the first file link list item into the second file link list, it also includes: updating the length value of the second file link list; when the updated length value of the second file link list is greater than a preset threshold, adding an end flag to the second file link list to generate a second file link list item.

[0009] In combination with the second implementation of the first aspect, in the fifth implementation of the first aspect, the second file linked list is generated by the following process: creating a random second file linked list, the name of the random second file linked list is a unique identification code name generated by a random algorithm; initializing the random second file linked list to generate a second file linked list, in an initial state, the length value of the second file linked list is zero, the number of first files included in the second file linked list is zero, and the offset of the second file linked list is zero.

[0010] In combination with the fifth implementation of the first aspect, in the sixth implementation of the first aspect, the method also includes: writing the name of the second file link list and the metadata information of each first file in sequence into a preset storage database, the metadata information of the first file includes: folder name, file name, creation time information, offset of the first file, length of the first file and check value of the first file, the check value of the first file is calculated and generated based on the first file and the MD5 message digest algorithm.

[0011] In combination with the first aspect, in the seventh implementation of the first aspect, the method also includes: after uploading the second file and writing it to the disk for storage, generating a successful upload prompt message, and sending the successful upload prompt message, the name information of the first file, and the name information of the second file back to the client.

[0012] According to the second aspect, an embodiment of the present invention provides a method for downloading a file, comprising: obtaining a file read request, wherein the file read request includes name information of each first file; based on the name information of each first file, querying a preset storage database, determining the length, offset and name information of each first file and the corresponding second file, and generating a file read pointer; based on the file read pointer and the file length, reading the corresponding second file on the disk, and sending the read file to a client, wherein the second file is uploaded to the disk according to the method described in the first aspect or any embodiment of the first aspect.

[0013] According to the third aspect, an embodiment of the present invention provides a file integration upload device, including: a first acquisition module, used to acquire multiple first files sent by a client, and metadata information of the first files, the metadata information of the first files including name information, length and content information; an offset calculation module, used to calculate and generate the offset corresponding to each of the first files according to the length of each of the first files; an integration module, used to integrate and generate a second file according to the offset and metadata information of each of the first files; an upload module, used to upload the second file and write it to the disk for storage.

[0014] According to the fourth aspect, an embodiment of the present invention provides a file downloading device, comprising: a second acquisition module, used to obtain a file reading request, wherein the file reading request includes the name information of each first file; a query module, used to query a preset storage database according to the name information of each first file, determine the length and offset of each first file and the name information of the corresponding second file, and generate a file reading pointer; a reading module, used to read the corresponding second file on the disk according to the file reading pointer and the file length, and send the read file to the client, wherein the second file is uploaded to the disk according to the method described in the first aspect or any embodiment of the first aspect.

[0015] According to the fifth aspect, an embodiment of the present invention provides a computer device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the steps of the file integration upload method described in the first aspect or any one embodiment of the first aspect, or the steps of the file download method described in the second aspect.

[0016] According to the sixth aspect, an embodiment of the present invention provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of the file integration upload method described in the first aspect or any one of the embodiments of the first aspect are implemented, or the steps of the file download method described in the second aspect are implemented.

[0017] The technical solution of the present invention has the following advantages:

[0018] 1. The present invention provides a method and device for integrated uploading of files, wherein the integrated uploading method comprises: obtaining multiple first files sent by a client, and metadata information of the first files, wherein the metadata information of the first files comprises name information, length and content information; calculating and generating the offset corresponding to each first file according to the length of each first file; integrating and generating a second file according to the offset and metadata information of each first file; uploading and writing the second file to a disk for storage. By implementing the present invention, the problems of poor disk read / write performance and low efficiency of uploading small files caused by concurrent uploading of multiple small files in the related art are solved, and the metadata information and content information of the uploaded multiple small files are combined to integrate into a large file, thereby reducing the additional disk read / write overhead, improving the disk read / write performance, and improving the speed and space utilization of Blu-ray burning.

[0019] 2. The present invention provides a method and device for downloading files, wherein the download method includes: obtaining a file read request, wherein the file read request includes the name information of each first file; according to the name information of each first file, querying a preset storage database, determining the length, offset and name information of the corresponding second file of each first file, and generating a file read pointer; according to the file read pointer and the file length, reading the corresponding second file on the disk, and sending the read file to the client. By implementing the present invention, combining the file read request and the preset database, generating a file read pointer, the content information of a small file can be quickly and accurately read on the disk, and returned to the user terminal in a timely manner. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0021] Figure 1 A flowchart of a specific example of a method for integrated uploading of files in an embodiment of the present invention;

[0022] Figure 2 A flowchart of integrating and generating a second file in a method for integrating and uploading files in an embodiment of the present invention;

[0023] Figure 3 is a flowchart of a method for downloading a file in an embodiment of the present invention;

[0024] Figure 4 A schematic diagram of a method for integrating uploading files and a method for downloading files in an embodiment of the present invention;

[0025] Figure 5 It is a principle block diagram of a specific example of a device for integrating and uploading files in an embodiment of the present invention;

[0026] Figure 6 is a principle block diagram of a specific example of a device for downloading files in an embodiment of the present invention;

[0027] Figure 7 FIG. 4 is a specific example diagram of a computer device in an embodiment of the present invention. DETAILED DESCRIPTION

[0028] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0029] In the related art, various common network attached storage systems have good performance when reading and writing large files, but for small files, due to the metadata operation of the file system (concurrent upload) and the fact that the disk is more suitable for sequential reading and writing, the disk reading and writing performance is usually poor.

[0030] Since the user terminal can upload multiple small files concurrently through the client through the HyperText Transfer Protocol (http) or HTTP channel (https) interface, when uploading files using the object storage interface, such as s3, or open swift interface, if the server can immediately store these small files in a large file, it can save the additional disk IO bandwidth for reading and writing data when compressing and packaging the small files later, and can also make full use of the various performance characteristics of the file system.

[0031] Based on this, an embodiment of the present invention provides a file integration upload method and device, as well as a file download method and device, the purpose of which is to enable multiple small files to share a large file in the file system, reduce metadata operations, and thus improve disk reading and writing performance.

[0032] The object storage file system includes a client, a storage server, and a disk. In actual applications, the client (user terminal) in the object storage file system is used to actually upload or read files; the storage server may include a preset memory space and an arbitrator, wherein the preset memory space is used to cache file data, and the arbitrator is used to integrate the uploaded small files into a large file; the disk is used to stably store the files uploaded by the client.

[0033] The embodiment of the present invention provides a method for integrating and uploading files, such as Figure 1 As shown, the arbitrator used in the storage server includes:

[0034] Step S11: Acquire multiple first files sent by the client, and metadata information of the first files, where the metadata information of the first files includes name information, length, and content information. In this embodiment, the client may be any computer device in the file storage system. The first file may be a small file, such as a file smaller than 10k. The metadata information of the first file may be information used to characterize file attributes, such as name information, length, and content information, where the name information of the first file may be the file name information carried by the user terminal when uploading the small file, and may also include the file bucket name in the object storage file system. The length information of the first file may be the actual length of the small file, which may be in bytes, KB, MB, GB, etc. The content information of the first file may be the data information actually stored in the small file.

[0035] Specifically, when the user terminal receives a file upload request sent by the client, it obtains multiple small upload files sent by the client according to each upload file request, that is, the client sends multiple small files to the storage server, and the arbitrator obtains metadata information corresponding to each small file based on the received small files.

[0036] Step S12: Calculate and generate the offset corresponding to each first file according to the length of each first file; in this embodiment, since the first file will be cached in the preset memory space after being sent to the storage server, the offset corresponding to the first file can be an offset within the segment of the preset memory space, which is used to characterize the distance between the actual address of the storage unit and the segment address of the segment where it is located, and can also be called an effective address. Specifically, in the actual application scenario of the object storage file system, the offset corresponding to the first file can be the ratio of the length of each first file to the preset unit storage capacity, and the preset unit storage capacity can be 512 bytes or 1024 bytes, which is not limited by the present invention.

[0037] Step S13: Generate a second file based on the offset and metadata information of each first file; in this embodiment, the second file can be a large file integrated from multiple first files, specifically, the second file can be a large file, and the first file can be a small file; the arbitrator allocates a storage location to each small file based on the calculated offset of multiple small files, and integrates the small files to generate a large file.

[0038] Step S14: Upload and write the second file to the disk for storage. In this embodiment, according to the generated second file, the arbitrator uploads the received multiple first files to the disk in sequence for stable storage.

[0039] The present invention provides a method for integrated uploading of files, comprising: obtaining multiple first files sent by a client, and metadata information of the first files, wherein the metadata information of the first files includes name information, length, and content information; calculating and generating offsets corresponding to each first file according to the length of each first file; integrating and generating a second file according to the offsets and metadata information of each first file; and uploading and writing the second file to a disk for storage.

[0040] By implementing the present invention, the problem of poor disk read / write performance and low efficiency of uploading small files caused by concurrent uploading of multiple small files in the related art is solved. The metadata information and content information of multiple uploaded small files are combined into a large file, which reduces the additional disk read / write overhead, improves the disk read / write performance, and improves the speed and space utilization of Blu-ray burning.

[0041] As an optional implementation of the present invention, the above-mentioned step S12 calculates and generates the offset corresponding to each first file according to the length of each first file, including: according to the length of each first file and the preset unit storage capacity, calculating the ratio of the length of the first file to the preset unit storage capacity, and using the ratio as the offset of the first file.

[0042] In this embodiment, the length of the first file may be the length information of the content information of the first file; the preset unit storage amount may be the size of a file block preset in the storage space; the offset of the first file may be an offset within a segment of the preset memory space, which is used to characterize the distance between the actual address of the storage unit and the segment address of the segment in which it is located; specifically, the length of the first file may be 1KB, and the preset unit storage amount may be 512 bytes. At this time, the offset of the first file may be the ratio of the length of the first file to the preset unit storage amount, which is 2 preset file blocks. The size of the preset file block may be determined according to the actual application scenario, for example, 512 bytes.

[0043] As an optional implementation of the present invention, the above step S13, integrating and generating the second file according to the offset and metadata information of each first file, includes:

[0044] Step S21: according to the offset of the first file, update the offset of the pre-stored second file linked list, and generate the offset of the updated second file linked list; in this embodiment, the pre-stored large file linked list can be an initial linked list pre-set in the arbitrator, and the offset of the large file linked list can be the position where the small file is inserted into the large file at the current moment; specifically, the offset of the initial large file linked list can be zero, and the offset of the first uploaded small file can be 2 preset file blocks. At this time, after the small file is uploaded, the offset of the large file linked list is updated. At this time, the offset of the updated large file linked list is also 512×2 bytes.

[0045] Step S22: Store the content information of each first file in a preset memory space respectively, and generate a memory pointer, where the memory pointer is used to represent the storage path information of the first file in the preset memory space; in this embodiment, the preset memory space is a storage unit pre-set in the storage server, which is used to temporarily cache data; temporarily cache each small file in the preset memory space, and generate a memory pointer based on the storage path information of the small file stored in the preset memory space, and the data in the preset memory space can be read quickly and accurately based on the memory pointer.

[0046] Step S23: Generate first file linked list items for each first file according to the memory pointer, the metadata information of the first file, the offset of the first file and the offset of the updated second file linked list; in this embodiment, the client uploads multiple small files to the storage server. When the storage server receives each small file and calculates the offset of each small file, it generates a linked list item for each small file according to the storage path information of each small file in the preset memory space, the offset of each small file, the content information of each small file, the length information of each small file, and the name information of each small file to represent the characteristic information of each small file.

[0047] Step S24: insert each first file link list item into the second file link list, integrate and generate the second file link list item, and integrate each first file and metadata information to generate the second file. In this embodiment, the generated small file link list items are sequentially inserted into the pre-stored large file link list to generate a large file link list item, which is used to store the storage path information and name information of each small file in the preset memory space; the large file can be generated by integrating and arranging each small file according to the order in which each small file is uploaded.

[0048] Optionally, the small files are integrated to generate a large file, the small file list items are inserted into the pre-stored large file list in sequence to generate large file list items, and then the large file list items are attached to the integrated large file, which can further ensure the security of the content information of each small file.

[0049] The file integration and uploading method provided by the embodiment of the present invention is combined with integrating the small files into a large file and then reading it, so that the size of the burned file can be kept within a reasonable range when the file is burned on a Blu-ray disc, and the number of inode nodes that need to be read when the file is burned can be reduced, thereby improving the burning speed; combined with integrating the small files uploaded in parallel into a large file uploaded in serial, the information of all the small files contained in the large file (i.e., the small file link list items) is added to the end of the large file generated by integration, thereby improving the reliability of the data, and the user can completely restore all file contents using a single disk cartridge.

[0050] Optionally, the above step S14, uploading and writing the second file to a disk for storage, includes:

[0051] According to the second file linked list item, the second file is uploaded and written to the disk for storage. In this embodiment, the linked list items in the second file include each small file linked list item, and a collection of storage path information of each small file arranged in series is generated according to each small file linked list item, that is, the storage path information in the preset memory space. When the second file is uploaded to the disk, each small file is read in sequence according to the storage position of each small file in the preset memory space, that is, the memory pointer of each small file, and then uploaded to the disk in sequence.

[0052] Optionally, a collection of storage path information of each small file arranged in series may be generated based on the storage path information of each small file in a preset memory space, and each small file may be read in sequence based on the storage position of each small file in the preset memory space, that is, the memory pointer of each small file, and firstly be spliced ​​into a large file and then uploaded to the disk.

[0053] As an optional embodiment of the present invention, after the step of inserting the first file linked list item into the second file linked list in step S24, it also includes: updating the length value of the second file linked list; when the updated length value of the second file linked list is greater than a preset threshold, adding an end mark to the second file linked list to generate a second file linked list item.

[0054] In this embodiment, after the small file list item is inserted into the second file list, the length value of the second file list is updated according to the length value in the metadata information of the small file; the preset threshold value can be a length value determined in an actual application scenario, for example, 5GB. Specifically, when the total length of the integrated second file exceeds 5GB, the writing of new small files should be stopped, an end mark should be marked, and the allocation of the arbitrator should be removed.

[0055] The embodiment of the present invention provides a method for integrating and uploading files. By combining the length value of the second file generated by the integration with the control, files with a length of 1-10KB can be merged into a 5GB file, greatly improving the disk writing rate.

[0056] As an optional implementation of the present invention, the second file linked list is generated by the following process:

[0057] First, create a random second file list, the name of the random second file list is a unique identification code name generated by a random algorithm; in this embodiment, the unique identification code name generated by the random algorithm can be a randomly generated UUID, which can ensure that the name of the integrated and generated second file will not be repeated, thereby ensuring the uniqueness of the second file name information.

[0058] Secondly, the random second file linked list is initialized to generate the second file linked list. In the initial state, the length value of the second file linked list is zero, the number of first files included in the second file linked list is zero, and the offset of the second file linked list is zero. In this embodiment, the process of initializing the random second file linked list may be to set the length information of the second file linked list to zero, set the offset of the second file linked list to zero, set the number of small file linked list items and the number of small files included in the second file linked list to zero, and set the pointers to the first small file and the last small file to null.

[0059] Optionally, when the first file is an initial first file, a second file linked list entry is generated according to a memory pointer, name information, length, offset of the initial first file, offset of the second file linked list, and a preset head pointer. In this embodiment, the preset head pointer may be a pointer to the second file header.

[0060] As an optional implementation of the present invention, the integrated upload method of the file further includes:

[0061] The name of the second file list and the metadata information of each first file are written into the preset storage database in sequence. The metadata information of the first file includes: folder name, file name, creation time information, offset of the first file, length of the first file and check value of the first file. The check value of the first file is calculated and generated based on the first file and the MD5 message digest algorithm.

[0062] In this embodiment, the preset storage database may be an object database in an object file storage system; the folder name of the first file may be the bucket name of each small file; and the checksum of the first file may be the etag value of each small file. After uploading all small files and the large file generated by integration to the disk, the name of the second file link list and the metadata information of each first file are sequentially written into the preset storage database, marking the end of the integration upload of each small file.

[0063] An integrated file upload method provided by an embodiment of the present invention stores each data in an object database only after each small file has been successfully uploaded to a disk, that is, only after the object database has been updated. This avoids the problem of losing the files uploaded by the user before being written to the disk due to a sudden power outage of the machine through caching in the memory. This ensures that the file upload is completed only after the file has been written to the disk and the database content has been updated, thereby improving the stability and security of file storage.

[0064] As an optional implementation of the present invention, the integrated upload method of the file further includes: after the second file is uploaded and written to the disk for storage, a successful upload prompt message is generated, and the successful upload prompt message, the name information of the first file, and the name information of the second file are returned to the client. In this embodiment, after executing the step of writing the second file to the disk, the second file and the second file linked list item are recorded in the database, and then the information of the file object written to the disk is returned to the client, including the check value of each small file, the check value of the large file, the name information of each small file, and the name information of the large file.

[0065] The embodiment of the present invention also provides a method for downloading a file. Figure 3 As shown, the arbitrator used in the storage server includes:

[0066] Step S31: obtaining a file reading request, wherein the file reading request includes the name information of each first file; in this embodiment, receiving a file reading request sent by a client, and determining the name information of the file that the client needs to read.

[0067] Step S32: According to the name information of each first file, query the preset storage database, determine the length and offset of each first file and the name information of the corresponding second file, and generate a file read pointer; in this embodiment, the preset storage database can be the object storage database described in the above embodiment. Specifically, according to the name information of each small file in the file read request, the name information, offset and length information of the corresponding large file are determined in the object storage database, and then the file read pointer is generated.

[0068] Step S33: Read the corresponding second file on the disk according to the file read pointer and the file length, and send the read file to the client. The second file is uploaded to the disk according to the file integration upload method described in any of the above embodiments. In this embodiment, the storage location and offset of each small file on the disk are determined according to the name information and offset of the large file in the file read pointer and the length information of the first file. The large file is read at the corresponding position on the disk according to the file read pointer. The corresponding file block is read according to the offset. The read end position is determined according to the length information of the first file, and the small files that the user needs to obtain are generated and then sent back to the client.

[0069] The present invention provides a method for downloading a file, comprising: obtaining a file reading request, wherein the file reading request includes name information of each first file; querying a preset storage database according to the name information of each first file, determining the length and offset of each first file and the name information of the corresponding second file, and generating a file reading pointer; reading the corresponding second file on a disk according to the file reading pointer and the file length, and sending the read file to a client.

[0070] By implementing the present invention, combining the file reading request and the preset database, a file reading pointer is generated, and the content information of the small file can be quickly and accurately read from the disk and returned to the user terminal in time.

[0071] In combination with a specific embodiment, Figure 4 As shown, the integrated upload method of files and the download method of files described in the above embodiment are described in detail and applied to the arbitrator in the storage server.

[0072] Specifically, each storage server in the file system of the object storage includes an arbitrator for quickly responding to instructions; a large file list is pre-stored in the storage server, and the large file name is a UUID generated according to a random algorithm. At this time, the file size of the pre-stored large file list is zero, the offset is zero, the number of small files contained in it is zero, and the pointers to the first file and the last file are empty;

[0073] When the storage server receives multiple small files uploaded by the client, for example, including uploaded file 1, with a length of A; uploaded file 2, with a length of B; uploaded file 3, with a length of C; the arbitrator calculates the offset of the small file based on the length information of uploaded file 1, uploaded file 2, and uploaded file 3 and the preset unit storage capacity (for example, 512 bytes), wherein if there is a remainder, one is added. Generate each small file linked list item based on the folder name, small file name, and creation time of each small file, for example, file block 1, file block 2, and file block 3 can be generated, and then insert each generated small file linked list into the tail of the pre-stored large file linked list in sequence until the total length of the integrated large file exceeds 5GB, and stop writing new small files. At this time, a mark is marked to end the large file operation, and the allocation of the arbitrator is removed.

[0074] The uploaded small file first applies for a linked list item through the arbitrator and is inserted into the linked list of the large file. Then the content information of the small file is uploaded to the memory, a memory pointer is generated, and the memory pointer is stored in the linked list of each small file.

[0075] Specifically, when the length of the uploaded small file (the uploaded file 4, with a length of D) exceeds a preset threshold (e.g., 256MB), a large file linked list can be allocated separately for storage for this file, and the file block 4 generated by the above-mentioned file integration upload method is inserted into the tail of the pre-stored large file linked list, and then an end mark is added; the preset threshold can be specifically determined according to the actual application scenario, and the present invention does not limit this. The preset threshold can be modified by modifying the configuration file of the software, that is, the configuration file storing the preset threshold needs to be stored separately. At this time, the large file can be uploaded in the same format as the integrated small files, which is convenient for the unified operation of the management personnel.

[0076] Specifically, when a large file list item and a large file are generated, an upload signal is generated and sent to the disk write manager in the disk, and the disk write manager writes the generated large file list item and the large file to the disk. The large file list item is written after the large file, that is, after the metadata information of each small file is written into the integrated large file, the block boundary of the large file and the metadata is a multiple of 512 bytes. If it is not an exact multiple of 512 bytes, it should be filled with 0 at the end. After the large file is written to the disk, it is immediately recorded in the database, and finally the information of the written object is returned to the client. The S3 client requires the Etag value of the file to be returned.

[0077] Specifically, when reading a small file, for example, when reading file 5, based on the name information of the small file, a table entry is queried in the preset object storage database to determine the name information, offset value and length information of the large file corresponding to the small file, and then the corresponding large file is read, the offset of the file pointer is set, the file length data is read, and the corresponding part of the large file is returned to the client.

[0078] The file integration upload method and file download method provided in the embodiments of the present invention can merge multiple concurrently uploaded small files into one large file, and ensure that the contents of the small files can be read correctly, and also reduce the additional overhead of disk reading and writing, and improve the disk writing rate. Combined with merging in memory and serial writing when writing to the disk, the disk reading and writing overhead is reduced, the object storage file system's reading and writing operations and performance indicators for small files are optimized, and indicators such as Blu-ray storage's burning of small files and space utilization are improved.

[0079] The embodiment of the present invention also provides a file integration upload device, such as Figure 5 As shown, including:

[0080] The first acquisition module 41 is used to acquire multiple first files sent by the client, and metadata information of the first files, where the metadata information of the first files includes name information, length and content information; for detailed implementation content, please refer to the relevant description of step S11 in the above method embodiment.

[0081] The offset calculation module 42 is used to calculate and generate the offset corresponding to each first file according to the length of each first file; the detailed implementation content can refer to the relevant description of step S12 in the above method embodiment.

[0082] The integration module 43 is used to integrate and generate a second file according to the offset and metadata information of each first file; the detailed implementation content can refer to the relevant description of step S13 in the above method embodiment.

[0083] The upload module 44 is used to upload the second file to the disk for storage. For detailed implementation, please refer to the relevant description of step S14 in the above method embodiment.

[0084] The present invention provides a file integration upload device, including: a first acquisition module 41, used to acquire multiple first files sent by a client, and metadata information of the first files, the metadata information of the first files including name information, length and content information; an offset calculation module 42, used to calculate and generate the offset corresponding to each first file according to the length of each first file; an integration module 43, used to integrate and generate a second file according to the offset and metadata information of each first file; an upload module 44, used to upload and write the second file to a disk for storage. By implementing the present invention, the problem of poor disk read and write performance and low efficiency of uploading small files caused by concurrent uploading of multiple small files in the related art is solved. The metadata information and content information of the uploaded multiple small files are combined to integrate into a large file, and the serial writing of multiple small files is realized, which reduces the additional disk read and write overhead, improves the disk read and write performance, and improves the speed and space utilization of Blu-ray burning.

[0085] The embodiment of the present invention also provides a device for downloading files, such as Figure 6 As shown, including:

[0086] The second acquisition module 51 is used to acquire a file reading request, wherein the file reading request includes the name information of each first file; for detailed implementation content, please refer to the relevant description of step S21 in the above method embodiment.

[0087] The query module 52 is used to query the preset storage database according to the name information of each first file, determine the length, offset and name information of the corresponding second file of each first file, and generate a file reading pointer; the detailed implementation content can be found in the relevant description of step S22 in the above method embodiment.

[0088] The reading module 53 is used to read the corresponding second file from the disk according to the file reading pointer and the file length, and send the read file to the client, and the second file is uploaded to the disk according to the method of the above embodiment. The detailed implementation content can be found in the relevant description of step S23 in the above method embodiment.

[0089] The present invention provides a file downloading device, comprising: a second acquisition module 51, used to acquire a file reading request, wherein the file reading request includes the name information of each first file; a query module 52, used to query a preset storage database according to the name information of each first file, determine the length and offset of each first file and the name information of the corresponding second file, and generate a file reading pointer; a reading module 53, used to read the corresponding second file on the disk according to the file reading pointer and the file length, and send the read file to the client. By implementing the present invention, combining the file reading request and the preset database, generating a file reading pointer, the content information of a small file can be quickly and accurately read on the disk, and returned to the user terminal in time.

[0090] The embodiment of the present invention provides a file integration upload device, such as Figure 7 As shown, including:

[0091] The embodiment of the present invention also provides a computer device, such as Figure 7 As shown, the computer device may include a processor 61 and a memory 62, wherein the processor 61 and the memory 62 may be connected via a bus 60 or other means. Figure 7 The connection via bus 60 is taken as an example.

[0092] The processor 61 may be a central processing unit (CPU). The processor 61 may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or a combination of the above chips.

[0093] The memory 62 is a non-transitory computer-readable storage medium that can be used to store non-transitory software programs, non-transitory computer executable programs and modules, such as program instructions / modules corresponding to the integrated upload method of files in the embodiment of the present invention. The processor 61 executes various functional applications and data processing of the processor by running the non-transitory software programs, instructions and modules stored in the memory 62, that is, implementing the integrated upload method of files in the above method embodiment.

[0094] The memory 62 may include a program storage area and a data storage area, wherein the program storage area may store an operating system and applications required for at least one function; the data storage area may store data created by the processor 61, etc. In addition, the memory 62 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage devices. In some embodiments, the memory 62 may optionally include a memory remotely arranged relative to the processor 61, and these remote memories may be connected to the processor 61 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0095] The one or more modules are stored in the memory 62, and when executed by the processor 61, the following is performed: Figure 1 The integrated upload method of files in the illustrated embodiment.

[0096] For details of the above computer equipment, please refer to Figure 1 The corresponding related descriptions and effects in the illustrated embodiments can be understood and will not be repeated here.

[0097] An embodiment of the present invention further provides a non-transitory computer-readable medium, which stores computer instructions, and the computer instructions are used to enable a computer to execute the file integration and uploading method described in any one of the above embodiments, wherein the storage medium may be a magnetic disk, an optical disk, a read-only memory (ROM), a random access memory (RAM), a flash memory (Flash Memory), a hard disk (HardDisk Drive, abbreviated as: HDD) or a solid-state drive (SSD), etc.; the storage medium may also include a combination of the above-mentioned types of memories.

[0098] Obviously, the above embodiments are merely examples for the purpose of clear explanation, and are not intended to limit the implementation methods. For those skilled in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the implementation methods here. The obvious changes or modifications derived therefrom are still within the scope of protection of the invention.

Claims

1. A method for integrated uploading of files, characterized in that: include: Acquire multiple first files sent by the client, and metadata information of the first files, where the metadata information of the first files includes name information, length, and content information; According to the length of each of the first files and the preset unit storage amount, respectively calculate the ratio of the length of the first file to the preset unit storage amount, and use the ratio as the offset of the first file, the preset unit storage amount is the size of the file block preset in the storage space, and the offset is the offset of the first file in the segment of the preset memory space; According to the offset of the first file, updating the offset of the pre-stored second file linked list to generate an updated offset of the second file linked list; storing content information of each of the first files in a preset memory space respectively, and generating a memory pointer, wherein the memory pointer is used to represent storage path information of the first file in the preset memory space; Generate a first file linked list item for each of the first files according to the memory pointer, the metadata information of the first file, the offset of the first file, and the offset of the updated second file linked list; insert each of the first file linked list items into the second file linked list to integrate and generate a second file linked list item, and integrate each of the first files and the metadata information to generate the second file; The second file linked list is generated by the following process: Creating a random second file linked list, wherein the name of the random second file linked list is a unique identification code name generated by a random algorithm; Initializing the random second file linked list to generate a second file linked list, in which, in an initial state, the length value of the second file linked list is zero, the number of first files included in the second file linked list is zero, and the offset of the second file linked list is zero; Writing the name of the second file link list and metadata information of each first file into a preset storage database in sequence, wherein the metadata information of the first file includes: a folder name, a file name, a creation time information, an offset of the first file, a length of the first file, and a check value of the first file, wherein the check value of the first file is calculated and generated according to the first file and an MD5 message digest algorithm; The second file is uploaded and written to disk for storage.

2. The method according to claim 1, characterized in that The uploading and writing the second file into a disk for storage includes: According to the second file linked list item, the second file is uploaded and written to the disk for storage.

3. The method according to claim 1, characterized in that After the step of inserting the first file linked list item into the second file linked list, the method further includes: Update the length value of the second file linked list; When the updated length value of the second file linked list is greater than a preset threshold, an end mark is added to the second file linked list to generate a second file linked list item.

4. The method according to claim 1, characterized in that: Also includes: After the second file is uploaded and written to the disk for storage, a successful upload prompt message is generated, and the successful upload prompt message, the name information of the first file, and the name information of the second file are sent back to the client.

5. A method for downloading a file, characterized in that: include: Obtaining a file reading request, wherein the file reading request includes name information of each first file; According to the name information of each of the first files, query a preset storage database to determine the length and offset of each of the first files and the name information of the corresponding second file, and generate a file reading pointer; According to the file read pointer and the file length, the corresponding second file is read from the disk, and the read file is sent to the client, wherein the second file is uploaded to the disk according to the method described in any one of claims 1-4.

6. A file integration upload device, characterized in that: include: A first acquisition module, used to acquire multiple first files sent by the client, and metadata information of the first files, where the metadata information of the first files includes name information, length, and content information; an offset calculation module, configured to calculate, according to the length of each of the first files and a preset unit storage amount, a ratio of the length of the first file to the preset unit storage amount, and use the ratio as an offset of the first file, wherein the preset unit storage amount is a size of a file block preset in a storage space, and the offset is an offset of the first file in a segment of a preset memory space; An integration module, used for updating the offset of the pre-stored second file linked list according to the offset of the first file, and generating an updated offset of the second file linked list; storing content information of each of the first files in a preset memory space respectively, and generating a memory pointer, wherein the memory pointer is used to represent storage path information of the first file in the preset memory space; Generate a first file linked list item for each of the first files according to the memory pointer, the metadata information of the first file, the offset of the first file, and the offset of the updated second file linked list; insert each of the first file linked list items into the second file linked list to integrate and generate a second file linked list item, and integrate each of the first files and the metadata information to generate the second file; The second file linked list is generated by the following process: Creating a random second file linked list, wherein the name of the random second file linked list is a unique identification code name generated by a random algorithm; Initializing the random second file linked list to generate a second file linked list, in which, in an initial state, the length value of the second file linked list is zero, the number of first files included in the second file linked list is zero, and the offset of the second file linked list is zero; Writing the name of the second file link list and metadata information of each first file into a preset storage database in sequence, wherein the metadata information of the first file includes: a folder name, a file name, a creation time information, an offset of the first file, a length of the first file, and a check value of the first file, wherein the check value of the first file is calculated and generated according to the first file and an MD5 message digest algorithm; The uploading module is used to upload the second file and write it to the disk for storage.

7. A file downloading device, characterized in that: include: A second acquisition module, used to acquire a file reading request, wherein the file reading request includes name information of each first file; A query module, used to query a preset storage database according to the name information of each of the first files, determine the length and offset of each of the first files and the name information of the corresponding second file, and generate a file reading pointer; A reading module is used to read the corresponding second file on the disk according to the file reading pointer and the file length, and send the read file to the client, wherein the second file is uploaded to the disk according to the method described in any one of claims 1-4.

8. A computer device, characterized in that: include: at least one processor; And a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the one processor, and the instructions are executed by the at least one processor so that the at least one processor executes the steps of the file integration upload method described in any one of claims 1 to 4 or the file download method described in claim 5.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the file integration upload method according to any one of claims 1 to 4 or the file download method according to claim 5 are implemented.

Images