Method and system for storing multiple concurrent large files

Abstract

The invention relates to the technical field of file storage, and discloses a method and a system for storing multiple concurrent large files. According to the method and the system, the format and the size of a file are verified, a unique file name is generated and used for identifying metadata of a storage file and outputting the file, each small block of the file is divided into a fixed size and is divided into small blocks, a unique identifier is generated for each small block, a database is connected, and the metadata of the file is stored in the database. Then traversing each small block, selecting a storage node for each small block, uploading the small blocks by using multiple threads, setting a request queue, processing file uploading and downloading requests in sequence, avoiding concurrent write-in conflicts of the same file by using a lock mechanism for file uploading, and reading and controlling a read request through a read-write lock; and acquiring the information of the storage file from the database, acquiring each small block by utilizing a concurrent downloading mode, recombining the small blocks into an original file in sequence after downloading is completed, and performing integrity verification after file recombining.

Description

Technical Field

[0001] The present invention relates to the technical field of file storage, and in particular to a method and system for storing multiple concurrent large files. Background Art

[0002] In modern application environments, especially in cloud computing and distributed systems, allowing multiple users to upload or download large files at the same time can greatly improve resource utilization. This concurrent processing can better support large-scale user access and effectively improve user experience. Supporting concurrent storage allows the system to be more easily expanded to adapt to the growing storage needs in the future, which is beneficial to the scalability of the system.

[0003] Currently, most files are not divided into small blocks when they are stored, which increases the throughput of the storage system. At the same time, the lack of request queues and lock mechanisms leads to concurrent access conflicts, resulting in data loss and corruption when writing to the same file. Summary of the invention

[0004] In view of the shortcomings of the prior art, the present invention provides a method and system for storing multiple concurrent large files, which can generate unique files by verifying the file format and size. Metadata used to identify stored files and output files, including files , file name and file size, divide each small block of the file into 4MB size, and split it to generate N small blocks, generate a unique identifier for each small block, then connect to the database, store the file metadata in the database, and calculate the average storage time and storage success rate, then traverse each small block, select a storage node for each small block, use multi-threading to upload the small block, and record the upload status, and calculate the actual number of threads used and the average thread usage time, set up a request queue, and process file upload and download requests in sequence. For file upload, use a lock mechanism to avoid concurrent write conflicts of the same file. For read requests, read control through a read-write lock and calculate the number of lock conflicts. Obtain all small block information of the stored file from the database, use concurrent download to obtain each small block, and reassemble the small blocks into the original file in order after the download is completed, and verify the integrity of the reassembled file. By dividing large files into small blocks for storage, the pressure on a single storage unit can be significantly reduced, and the throughput of the storage system can be improved. At the same time, the introduction of request queues and lock mechanisms can effectively control concurrent access conflicts, ensuring that there will be no data loss or damage when writing to the same file, solving the above problems.

[0005] To achieve the above object, the present invention provides the following technical solution: a method for storing multiple concurrent large files, comprising the following steps: S1. Verify file format and size and generate a unique file Metadata used to identify stored files and output files, including files , file name and file size; S2, divide each small block of the file into 4MB in size, and split it into N small blocks, and generate a unique identifier for each small block; S3, connect to the database and store the file metadata in the database, including the file , file name, total number of blocks, each block and storage location, and calculate the average storage time and storage success rate; S4, traverse each small block, select a storage node for each small block, use multiple threads to upload the small blocks, record the upload status, and calculate the actual number of threads used and the average thread usage time; S5. Set up a request queue to process file upload and download requests in sequence. For file upload, use a lock mechanism to avoid concurrent write conflicts of the same file. For read requests, read control is performed through a read-write lock, and the number of lock conflicts is calculated. S6. Obtain all small block information of the stored file from the database, obtain each small block by concurrent downloading, reassemble the small blocks into the original file in order after the download is completed, and perform integrity verification on the reassembled file.

[0006] Preferably, the S1 generates a unique file The formula is as follows:

[0007] In the formula, Indicates the unique name of the file. Indicates the original name of the file, Indicates the number of bytes in the file. Indicates the creation time of the file, obtained through the system timestamp, Indicates that a hash operation is performed on the concatenation result of the above information.

[0008] Preferably, the formula for S2 to split the file is as follows:

[0009] In the formula, Indicates the total number of small blocks generated, rounded up. Represents the bytes of each small block, Indicates the total size of the file in bytes.

[0010] Preferably, the formula for calculating the average deposit time in S3 is as follows:

[0011] In the formula, represents the average deposit time, Indicates The time it takes to store a file block in the database, Indicates the total number of small file blocks. Indicates a counting subscript.

[0012] Preferably, the formula for calculating the storage success rate of S3 is as follows:

[0013] In the formula, Indicates the storage success rate. Indicates the number of blocks successfully deposited, Indicates the total number of small blocks generated.

[0014] Preferably, the formula for calculating the number of threads actually used in S4 is as follows:

[0015] In the formula, Indicates the number of threads actually used. Indicates the number of small blocks currently being uploaded. Indicates the maximum number of threads allowed by the system. Indicates the minimum limit function, the number of small blocks currently being uploaded Less than or equal to , then the actual number of active threads is , whereas the actual number of active threads is .

[0016] Preferably, the formula for calculating the average thread usage time in S4 is as follows:

[0017] In the formula, represents the average thread usage time, k represents the number of small blocks uploaded, Indicates Upload time for small chunks, Indicates a counting subscript.

[0018] Preferably, the formula for calculating the number of lock conflicts in S5 is as follows:

[0019] In the formula, Indicates the number of lock conflicts. Indicates the number of initial lock conflicts. When multiple threads try to acquire the same resource at the same time, a conflict will occur. Each time a conflict occurs, The value of is increased by 1.

[0020] Preferably, the formula for file reorganization integrity verification in S6 is as follows:

[0021] In the formula, Indicates the value of the received file reassembly integrity verification. Represents the original file, represents the reorganized file, H represents the hash function, which is used to calculate the hash value of the file. When the above two hash values ​​are equal, it means that the reorganized file is valid and the content is consistent with the original file. When they are not equal, it means that the reorganized file has data errors or is damaged.

[0022] A multi-concurrent large file storage system includes a metadata generation module, a small block segmentation module, a data storage module, a multi-threaded upload module, a lock mechanism module and a concurrent download reorganization verification module; The metadata generation module verifies the file format and size and generates a unique file Metadata used to identify stored files and output files, including files , file name and file size; The small block segmentation module divides each small block of the file into 4MB in size, and segments it to generate N small blocks, and generates a unique identifier for each small block; The data storage module stores the metadata of the file in the database, including the file , file name, total number of blocks, each block and storage location, and calculate the average storage time and storage success rate; The multi-threaded upload module traverses each small block, selects a storage node for each small block, uploads the small blocks using multiple threads, records the status of the upload, and calculates the number of threads actually used and the average thread usage time; The lock mechanism module sets a request queue to process file upload and download requests in sequence. For file upload, a lock mechanism is used to avoid concurrent write conflicts of the same file. For read requests, read control is performed through a read-write lock and the number of lock conflicts is calculated. The concurrent download reassembly verification module obtains all small block information of the stored file from the database, obtains each small block by concurrent download, reassembles the small blocks into the original file in sequence after the download is completed, and performs integrity verification on the reassembled file.

[0023] Compared with the prior art, the present invention provides a method and system for storing multiple concurrent large files, which has the following beneficial effects: The present invention generates a unique file by checking the file format and size Metadata used to identify stored files and output files, including files , file name and file size, divide each small block of the file into 4MB size, and split it to generate N small blocks, generate a unique identifier for each small block, then connect to the database, store the file metadata in the database, and calculate the average storage time and storage success rate, then traverse each small block, select a storage node for each small block, use multi-threading to upload small blocks, and record the upload status, and calculate the actual number of threads used and the average thread usage time, set up a request queue, and process file upload and download requests in sequence. For file upload, use a lock mechanism to avoid concurrent write conflicts of the same file. For read requests, read control through read-write locks and calculate the number of lock conflicts. Obtain all small block information of the stored file from the database, use concurrent downloads to obtain each small block, and reassemble the small blocks into the original file in order after downloading, and verify the integrity of the reassembled file. By dividing large files into small blocks for storage, the pressure on a single storage unit can be significantly reduced, and the throughput of the storage system can be improved. At the same time, the introduction of request queues and lock mechanisms can effectively control concurrent access conflicts and ensure that there will be no data loss or damage when writing to the same file. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 It is a schematic diagram of the steps of the method of the present invention; Figure 2 It is a schematic diagram of the system flow of the present invention. DETAILED DESCRIPTION

[0025] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0026] Most of the current files are not divided into small blocks for storage, which increases the throughput of the storage system. At the same time, the lack of request queues and lock mechanisms leads to concurrent access conflicts. When writing to the same file, data loss and corruption occur. Therefore, a multi-concurrency large file storage method is proposed. Figure 1 , the method comprises the following steps: S1. Verify file format and size and generate a unique file Metadata used to identify stored files and output files, including files , file name and file size; Generate unique file The formula is as follows:

[0027] Generate unique files Ensure that each file has a unique identifier in the system to avoid file name conflicts and duplicate storage problems. In the formula, Indicates the unique name of the file. Indicates the original name of the file, Indicates the number of bytes in the file. Indicates the creation time of the file, obtained through the system timestamp, Indicates that the hash operation is performed on the connection result of the above information, and the unique Makes file retrieval and access more efficient, especially when dealing with large numbers of files, enabling quick location of required files; S2, divide each small block of the file into 4MB in size, and split it into N small blocks, and generate a unique identifier for each small block; The formula for file segmentation is as follows:

[0028] Splitting large files into small blocks can be processed on multiple storage nodes at the same time, significantly improving the performance of concurrent access. In the formula, Indicates the total number of small blocks generated, rounded up. Represents the bytes of each small block, Indicates the total bytes of the file. If a small block fails to be uploaded, only the small block needs to be re-uploaded instead of the entire file. This operation improves the fault tolerance of the system. S3, connect to the database and store the file metadata in the database, including the file , file name, total number of blocks, each block and storage location, and calculate the average storage time and storage success rate; The calculation formula for average deposit time is as follows:

[0029] By calculating the average storage time, the performance of the storage system can be monitored in real time and potential performance bottlenecks can be discovered in time. In the formula, represents the average deposit time, Indicates The time it takes to store a file block in the database, Indicates the total number of small file blocks. Indicates the counting subscript. Analyzing the changing trend of storage time helps optimize storage strategy and resource allocation and improve overall storage efficiency. The formula for calculating the storage success rate is as follows:

[0030] The storage success rate reflects the stability and reliability of the system and helps identify possible problems in the storage process. In the formula, Indicates the storage success rate. Indicates the number of blocks successfully deposited, Indicates the total number of small blocks generated. By analyzing the storage success rate, corresponding improvement measures can be formulated to reduce the failure rate and improve the reliability of the system; S4, traverse each small block, select a storage node for each small block, use multiple threads to upload the small blocks, record the upload status, and calculate the actual number of threads used and the average thread usage time; The calculation formula for the actual number of threads used is as follows:

[0031] Monitoring the number of threads actually used can evaluate the utilization efficiency of system resources and ensure the stability of the system under high concurrency conditions. In the formula, Indicates the number of threads actually used. Indicates the number of small blocks currently being uploaded. Indicates the maximum number of threads allowed by the system. Indicates the minimum limit function. The number of small blocks currently being uploaded is less than or equal to , then the actual number of active threads is m, otherwise the actual number of active threads is ,By analyzing thread usage, reasonable performance tuning can be performed to avoid resource waste or excessive competition, and at the same time, it helps to achieve better load balancing and ensure uniform load on each storage node; The formula for calculating the average thread usage time is as follows:

[0032] Calculating the average usage time of threads helps analyze the performance of each thread and identify potential bottlenecks. In the formula, represents the average thread usage time, k represents the number of small blocks uploaded, Indicates Upload time for small chunks, Indicates the counting subscript. By evaluating the average thread usage time, you can optimize the concurrency strategy, reasonably configure the thread pool size, and improve the overall performance. S5. Set up a request queue to process file upload and download requests in sequence. For file upload, use a lock mechanism to avoid concurrent write conflicts of the same file. For read requests, read control is performed through a read-write lock, and the number of lock conflicts is calculated. The formula for calculating the number of lock conflicts is as follows:

[0033] The statistics of lock conflicts can reflect the performance of the system during concurrent access. By analyzing the lock conflicts, we can optimize the lock usage strategy, reduce lock competition, and improve concurrent performance. In the formula, Indicates the number of lock conflicts. Indicates the number of initial lock conflicts. When multiple threads try to acquire the same resource at the same time, a conflict will occur. Each time a conflict occurs, Adding 1 to the value can reduce lock conflicts, reduce request processing delays, and improve system response speed. S6. Obtain all small block information of the stored file from the database, obtain each small block by concurrent downloading, reassemble the small blocks into the original file in order after the download is completed, and perform integrity verification on the reassembled file; The formula for file reorganization integrity verification is as follows:

[0034] The integrity verification after file reorganization ensures that all small blocks can be correctly combined after downloading, ensuring the consistency and integrity of the data. During the verification process, errors in the download process can be discovered and corrected in a timely manner, reducing the risk of data corruption. In the formula, Indicates the value of the received file reassembly integrity verification. Represents the original file, Represents a reorganized file, and H represents a hash function used to calculate the hash value of a file. When the two hash values ​​are equal, it means that the reorganized file is valid and the content is consistent with the original file. When they are not equal, it means that the reorganized file has data errors or is damaged. By providing integrity verification, users' trust in the system is enhanced, ensuring that the data obtained by users is accurate and reliable. A multi-concurrent large file storage system includes a metadata generation module, a small block segmentation module, a data storage module, a multi-threaded upload module, a lock mechanism module and a concurrent download reorganization verification module; The metadata generation module verifies the file format and size and generates a unique file Metadata used to identify stored files and output files, including files , file name and file size; The small block segmentation module divides each small block of the file into 4MB in size, and segments it to generate N small blocks, and generates a unique identifier for each small block; The data storage module stores the metadata of the file in the database, including the file , file name, total number of blocks, each block and storage location, and calculate the average storage time and storage success rate; The multi-threaded upload module traverses each small block, selects a storage node for each small block, uploads the small blocks using multiple threads, records the status of the upload, and calculates the number of threads actually used and the average thread usage time; The lock mechanism module sets a request queue to process file upload and download requests in sequence. For file upload, a lock mechanism is used to avoid concurrent write conflicts of the same file. For read requests, read control is performed through a read-write lock and the number of lock conflicts is calculated. The concurrent download reassembly verification module obtains all small block information of the stored file from the database, obtains each small block by concurrent download, reassembles the small blocks into the original file in sequence after the download is completed, and performs integrity verification on the reassembled file.

[0035] Through the application of the above method and system, dividing large files into small blocks for storage can significantly reduce the pressure on a single storage unit and improve the throughput of the storage system. At the same time, the introduction of request queues and lock mechanisms can effectively control concurrent access conflicts and ensure that no data loss or damage will occur when writing to the same file.

[0036] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A method for storing multiple concurrent large files, characterized in that: The following steps are involved: S1. Verify file format and size and generate a unique file Metadata used to identify stored files and output files, including files , file name and file size; S2, divide each small block of the file into 4MB in size, and split it into N small blocks, and generate a unique identifier for each small block; S3, connect to the database and store the file metadata in the database, including the file , file name, total number of blocks, each block and storage location, and calculate the average storage time and storage success rate; S4, traverse each small block, select a storage node for each small block, use multiple threads to upload the small blocks, record the upload status, and calculate the actual number of threads used and the average thread usage time; S5. Set up a request queue to process file upload and download requests in sequence. For file upload, use a lock mechanism to avoid concurrent write conflicts of the same file. For read requests, read control is performed through a read-write lock, and the number of lock conflicts is calculated. S6. Obtain all small block information of the stored file from the database, obtain each small block by concurrent downloading, reassemble the small blocks into the original file in order after the download is completed, and perform integrity verification on the reassembled file.

2. A method for storing multiple concurrent large files according to claim 1, characterized in that: The S1 generates a unique file The formula is as follows: In the formula, Indicates the unique name of the file. Indicates the original name of the file, Indicates the number of bytes in the file. Indicates the creation time of the file, obtained through the system timestamp, Indicates that a hash operation is performed on the concatenation result of the above information.

3. A method for storing multiple concurrent large files according to claim 2, characterized in that: The formula for S2 to split the file is as follows: In the formula, Indicates the total number of small blocks generated, rounded up. Represents the bytes of each small block, Indicates the total size of the file in bytes.

4. A method for storing multiple concurrent large files according to claim 3, characterized in that: The formula for calculating the average deposit time in S3 is as follows: In the formula, represents the average deposit time, Indicates The time it takes to store a file block in the database, Indicates the total number of small file blocks. Indicates a counting subscript.

5. A method for storing multiple concurrent large files according to claim 4, characterized in that: The formula for calculating the storage success rate of S3 is as follows: In the formula, Indicates the storage success rate. Indicates the number of blocks successfully deposited, Indicates the total number of small blocks generated.

6. A method for storing multiple concurrent large files according to claim 5, characterized in that: The formula for calculating the actual number of threads used by S4 is as follows: In the formula, Indicates the number of threads actually used. Indicates the number of small blocks currently being uploaded. Indicates the maximum number of threads allowed by the system. Indicates the minimum limit function, the number of small blocks currently being uploaded Less than or equal to , then the actual number of active threads is , whereas the actual number of active threads is .

7. A method for storing multiple concurrent large files according to claim 6, characterized in that: The formula for calculating the average thread usage time in S4 is as follows: In the formula, represents the average thread usage time, K represents the number of small blocks uploaded, Indicates Upload time for small chunks, Indicates a counting subscript.

8. A method for storing multiple concurrent large files according to claim 7, characterized in that: The formula for calculating the number of lock conflicts in S5 is as follows: In the formula, Indicates the number of lock conflicts. Indicates the number of initial lock conflicts. When multiple threads try to acquire the same resource at the same time, a conflict will occur. Each time a conflict occurs, The value of is increased by 1.

9. A method for storing multiple concurrent large files according to claim 8, characterized in that: The formula for file reorganization integrity verification in S6 is as follows: In the formula, Indicates the value of the received file reassembly integrity verification. Represents the original file, represents the reorganized file, H represents the hash function, which is used to calculate the hash value of the file. When the above two hash values ​​are equal, it means that the reorganized file is valid and the content is consistent with the original file. When they are not equal, it means that the reorganized file has data errors or is damaged.

10. A multi-concurrent large file storage system, characterized by: It includes metadata generation module, small block segmentation module, data storage module, multi-threaded upload module, lock mechanism module and concurrent download reorganization verification module; The metadata generation module verifies the file format and size and generates a unique file Metadata used to identify stored files and output files, including files , file name and file size; The small block segmentation module divides each small block of the file into 4MB in size, and segments it to generate N small blocks, and generates a unique identifier for each small block; The data storage module stores the metadata of the file in the database, including the file , file name, total number of blocks, each block and storage location, and calculate the average storage time and storage success rate; The multi-threaded upload module traverses each small block, selects a storage node for each small block, uploads the small blocks using multiple threads, records the status of the upload, and calculates the number of threads actually used and the average thread usage time; The lock mechanism module sets a request queue to process file upload and download requests in sequence. For file upload, a lock mechanism is used to avoid concurrent write conflicts of the same file. For read requests, read control is performed through a read-write lock and the number of lock conflicts is calculated. The concurrent download reassembly verification module obtains all small block information of the stored file from the database, obtains each small block by concurrent download, reassembles the small blocks into the original file in sequence after the download is completed, and performs integrity verification on the reassembled file.

Images