Method and system for distributed video file processing based on raid disk FTP

Abstract

This invention relates to a distributed video file processing method and system based on RAID disks via FTP. The method includes: calculating the size, content hash value, and block content hash value of the video file to be uploaded via an FTP client and uploading it to an FTP scheduling module; the FTP scheduling module searches the FTP server for a file with the same size and hash value as the video file to be uploaded; if a file exists, the FTP file address of the same file is returned to the client; if no file exists, an optimal FTP server address is selected and returned to the client, and the client uploads the video file to the optimal FTP address. This invention utilizes the fast read / write characteristics of RAID 0 and also adds an idle backup and recovery file module to ensure data security. If a file is corrupted, a backup file is used for download, and the file can be recovered in the background.

Description

Technical Field

[0001] This invention relates to the storage of video files in video security monitoring, specifically to an FTP distributed video file storage method based on RAID disks. Background Technology

[0002] In the field of video security surveillance, the storage of video files presents the following problems:

[0003] 1. Based on the existing FTP server storage system with ordinary hard drives, disk I / O read / write operations are high, resulting in slow file response.

[0004] As the storage center for local area network information resources, the FTP server handles a very large throughput of video data, placing extremely high demands on the speed and stability of hard drive read and write operations. In addition, the data center lacks dedicated storage devices, and all data is stored on the FTP server's local hard drive, which increases the server's burden. The original storage system can no longer handle such high performance requirements.

[0005] When multiple users concurrently access an FTP server or access the same file simultaneously, the hard drive's response time and seek time increase significantly. This reduces the server's response speed and transfer speed to each client's request, and can even cause interruptions. More often than not, both situations occur simultaneously, meaning that storage performance largely determines the FTP server's performance. Table 1 shows the upload speed of an FTP server based on a standard hard drive.

[0006] 2. Security and integrity of files stored via FTP.

[0007] Traditional FTP servers prevent unauthorized file access by disabling anonymous login, enabling strong password security policies, implementing FTP directory isolation mechanisms, and specifying the source IP address. However, if files are corrupted, these mechanisms cannot be recovered or accessed.

[0008] 3. HDFS distributed storage system has slow upload speed for large video files.

[0009] Distributed storage systems (such as Hadoop's HDFS distributed file system) solve the problems of file security and integrity, but their upload speed is much slower than FTP (because communication uses the HTTP protocol and files are stored in 64MB blocks by default). Summary of the Invention

[0010] The purpose of this invention is to overcome the shortcomings of the prior art and provide a distributed FTP video file processing method and system based on RAID disks. This invention utilizes the fast file reading and writing characteristics of RAID0, and adds idle backup and recovery file modules to ensure data security. If a file is corrupted, the backup file is used for download, and the file can be recovered in the background.

[0011] The technical solution adopted to achieve the purpose of this invention is an FTP distributed video file processing method based on RAID disks, which includes:

[0012] S1. Configure the corresponding RAID0 disk drive letter and storage directory parameters on the FTP server, start the FTP service, and then upload the running status of the FTP server to the FTP scheduling module in real time.

[0013] S2. The FTP client calculates the size of the video file to be uploaded, the hash value of the file content, and the hash value of the file block content, and then uploads it to the FTP scheduling module.

[0014] S3. The FTP scheduling module searches the FTP server for a file with the same size and hash value as the video file to be uploaded. If a file exists, the module returns the FTP file address of the file to the client. If no file exists, the module selects the best FTP server address and returns it to the client, which then uploads the video file to that address.

[0015] S4. After the video file is successfully uploaded, the FTP scheduling module stores the unique ID, name, size, hash value, and FTP address of the video file in the database.

[0016] S5. Download the video file. The FTP client communicates with the FTP scheduling module to find the FTP server download address corresponding to the video file. If the file corresponding to the main FTP server download address is inaccessible, the client finds the corresponding backup FTP server download address. Based on the FTP server address, the FTP client communicates with the FTP server to download the video file.

[0017] Furthermore, this invention also provides an FTP distributed video file processing system based on RAID disks, the system comprising:

[0018] An FTP client used to calculate the size of the video file to be uploaded, the hash value of the file content, and the hash value of the file chunk content;

[0019] The FTP server is used to configure the storage directory parameters and storage files for the corresponding RAID0 disk drive letter.

[0020] The FTP scheduling module is used to search the FTP server for a file with the same size and hash value as the video file to be uploaded, calculated by the FTP client based on the file size, file content hash value, and file block content hash value. If a file exists, the FTP file address of the file is returned to the client; otherwise, an optimal FTP server address is selected and returned to the client, and the client uploads the video file to the optimal FTP address.

[0021] The system also includes an idle backup and restore module, used to back up and restore video files stored on an FTP server.

[0022] This invention leverages the fast read / write speeds of RAID 0 disks while adding backup and recovery operations to restore data and prevent data loss in the event of a hard drive failure. Furthermore, for multiple FTP storage services, the optimal FTP server is selected for operation. The FTP service ultimately writes data to or reads data from the disk, thus balancing the storage load on the RAID 0 disks. Attached Figure Description

[0023] Figure 1 This is a structural block diagram of the FTP distributed video file processing system based on RAID disks according to the present invention.

[0024] Figure 2 This is a schematic diagram of the file upload process in the FTP distributed video file processing method based on RAID disks of the present invention.

[0025] Figure 3 This is a schematic diagram of the file integrity judgment process in the FTP distributed video file processing method based on RAID disks of the present invention.

[0026] Figure 4 This is a schematic diagram of the file backup process in the FTP distributed video file processing method based on RAID disks of the present invention.

[0027] Figure 5 This is a schematic diagram of the file recovery process in the FTP distributed video file processing method based on RAID disks of the present invention.

[0028] Figure 6 This is a schematic diagram of the file download process in the FTP distributed video file processing method based on RAID disks of the present invention. Detailed Implementation

[0029] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments.

[0030] like Figure 1As shown, the FTP distributed video file processing system based on RAID disks of this invention includes an FTP client, an FTP server, an FTP scheduling module, and an idle backup and recovery module. The system's process for storing and downloading video files is as follows: Figure 2 As shown, it includes the following steps:

[0031] S1. Configure the FTP server with the corresponding RAID0 disk drive letter and storage directory parameters, start the FTP service, and then upload the FTP server running status to the FTP scheduling module in real time. The FTP server running status includes: disk free space, disk I / O unused rate, network unused rate, CPU unused rate, and memory unused rate.

[0032] S2. The client calculates the size of the video file to be uploaded, the hash value of the file content (64 bits), and the hash value of the file block content (64 bits). The hash value of the file block content is calculated by dividing the file into 1024 blocks, reading the first and last bytes from each block to form a string, and calculating a hash value (64 bits). After the calculation is completed, the client communicates with the FTP scheduling module and passes in the file size, the hash value of the file content, and the hash value of the file block content.

[0033] S3. The FTP scheduling module searches the FTP server for a file with the same size and hash value as the video file to be uploaded. If a file exists, it returns the FTP file address of the same file to the client. If no file exists, it selects the best FTP server address and returns it to the client, and the client uploads the video file to the best FTP address.

[0034] The selection of the best FTP server for this example includes the following steps:

[0035] 1) Based on the size of the file to be uploaded, filter out FTP servers with remaining storage space smaller than the size of the file to be uploaded;

[0036] 2) Select the best FTP server from among those with sufficient storage space to store files; the best FTP server can be selected based on the following methods:

[0037] In a single-server scenario, deploy 2-3 RAID0 disks and corresponding FTP servers. Calculate a weighted score based on the FTP server's disk I / O unutilization, network unutilization, CPU unutilization, and memory unutilization. For example, weight the four parameters by 0.2, 0.6, 0.1, and 0.1 respectively. Finally, sort the scores and select the FTP server with the highest score as the best FTP server.

[0038] In a multi-server scenario, a weighted score is calculated based on the FTP server's disk I / O unutilization rate, network unutilization rate, CPU unutilization rate, and memory unutilization rate. For example, the four parameters are weighted by 0.5, 0.3, 0.1, and 0.1 respectively. Finally, the scores are sorted, and the FTP server with the highest score is selected as the best FTP server.

[0039] Existing FTP storage services are mostly based on ordinary disks or RAID5 large file storage systems. However, the slow disk I / O results in relatively low FTP performance. Many distributed storage systems, while solving file security and integrity issues, have upload speeds significantly slower than FTP. This invention fully utilizes the fast read / write characteristics of RAID0, while employing idle backup and recovery modules to ensure file security and integrity.

[0040] Tests revealed that FTP performance was optimal when using two RAID0 disks and an FTP server configuration. See Table 2 for specific test results.

[0041] In a multi-computer server scenario, a weighted score was calculated based on disk I / O unutilization, network unutilization, CPU unutilization, and memory unutilization. Files with the highest scores were prioritized for FTP storage. See Table 3 for detailed test results.

[0042] 1) One computer server, with weights of 0.2, 0.6, 0.1, and 0.1 respectively, has the best score for FTP file storage and the fastest upload speed.

[0043] 2) With two or more computer servers, weighted at 0.5, 0.3, 0.1, and 0.1 respectively, the server with the best score will store files via FTP and have the fastest upload speed.

[0044] Upload speed test results under the same server conditions: regular disk, single FTP server:

[0045] Number of disks Number of FTP servers Upload speed of 1000 files Upload speed of 3000 files 1 1 99.17 GB / h 95.24GB / h

[0046] Note: Each file is 1GB in size, the network card bandwidth is 1Gb / s, and h represents the hour.

[0047] Table 1: FTP upload speed of a single server's ordinary disk

[0048] Upload speed test results for multiple RAID0 disks and an FTP server on the same server:

[0049]

[0050]

[0051] Note:

[0052] Each file is 1GB in size, the network card bandwidth is 1Gb / s, h represents the hourly rate, and the RAID0 write speed is 450MB / s.

[0053] When the number of disks and FTP servers reaches 3 or 4, there is no significant speed improvement. This is because the bandwidth reaches its maximum (average 100MB / s) in this case, and the average write speed of each RAID0 is 80MB / s (although the hardware write speed of RAID0 is very high, other applications occupy disk I / O, such as video file backup, video file recovery, video file format conversion, etc.). However, the overall RAID0 write speed is faster than the network speed, and the real bottleneck is the network card bandwidth.

[0054] Table 2: FTP upload speed of a single server's RAID0 disk

[0055] Upload speed test results with two RAID0 configurations and two FTP servers on a single server:

[0056] Disk I / O weight Network weight CPU weight Memory weight Upload speed of 1000 files Upload speed of 3000 files 0.1 0.7 0.1 0.1 343.44 GB / h 337.39 GB / h 0.2 0.6 0.1 0.1 349.59 GB / h 342.23 GB / h 0.3 0.5 0.1 0.1 341.17GB / h 333.86GB / h 0.4 0.4 0.1 0.1 333.66 GB / h 323.51 GB / h 0.5 0.3 0.1 0.1 327.14 GB / h 319.47GB / h 0.6 0.2 0.1 0.1 318.89 GB / h 309.18 GB / h 0.7 0.1 0.1 0.1 296.97 GB / h 287.71 GB / h

[0057] Upload speed test results with two servers, each with two RAID0 configurations and two FTP servers:

[0058]

[0059]

[0060] Upload speed test results with three servers, each with two RAID0 configurations and two FTP servers:

[0061] Disk I / O weight Network weight CPU weight Memory weight Upload speed of 1000 files Upload speed of 3000 files 0.1 0.7 0.1 0.1 935.11 GB / h 891.41 GB / h 0.2 0.6 0.1 0.1 950.81 GB / h 873.51 GB / h 0.3 0.5 0.1 0.1 965.36 GB / h 879.63 GB / h 0.4 0.4 0.1 0.1 985GB / h 882.80GB / h 0.5 0.3 0.1 0.1 1034.25GB / h 974.70 GB / h 0.6 0.2 0.1 0.1 954.08 GB / h 870.45GB / h 0.7 0.1 0.1 0.1 933.78 GB / h 847.30 GB / h

[0062] Note:

[0063] 1) Each file is 1GB in size, the network card bandwidth is 1Gb / s, h represents the hourly rate, and the RAID0 write speed is 450MB / s.

[0064] 2) In the case of a single server, the bandwidth reaches its maximum (average 100MB / s), but the average write speed of each RAID0 array is only 80MB / s (although the hardware write speed of RAID0 is very high, other applications consume disk I / O, such as video file backup, video file recovery, and video file format conversion). Therefore, the best performance is achieved with a weighting ratio of (0.2, 0.6, 0.1, 0.1).

[0065] Table 3: FTP upload speed of RAID 0 disks on multiple servers

[0066] S4. After the video file is successfully uploaded, the FTP scheduling module stores the unique ID, name, size, hash value, and FTP address of the video file in the database.

[0067] S5. When downloading a video file, the FTP client communicates with the FTP scheduling module to find the corresponding FTP server download address for the video file. If the file at the main FTP server download address is inaccessible, the client searches for the corresponding backup FTP server download address. Based on the FTP server address, the FTP client communicates with the FTP server to download the video file. Figure 6 As shown.

[0068] In a preferred embodiment of the present invention, after the video file is successfully uploaded, the present invention can also back up and restore the video file during idle time through the idle backup and restore file module. Specifically, the integrity of the file is checked before backup, such as... Figure 3 As shown, it includes the following steps:

[0069] 1) Periodically scan drive letters; if an FTP server cannot access a drive, it is considered to have a corrupted disk. Periodically access files; if an access failure occurs, the file is considered corrupted. Periodically compare file sizes with the original file size; if they do not match, the file is considered corrupted. Record the disk corruption or file corruption status of files and write the data to the database.

[0070] 2) During idle periods, determine file integrity. When disk I / O usage is low, compare the hash values ​​of file block content with the original file. If they do not match, the file is considered corrupted. Record the file's disk corruption or file corruption status and write it to the database.

[0071] The idle backup file is initiated when there are no FTP file storage tasks on the FTP server and disk I / O usage is very low. It then backs up the files to another FTP server. Figure 4 As shown, it includes the following steps:

[0072] 1) Find a usable FTP server to use as a backup FTP server.

[0073] 2) When the original video file is stored on the same computer as the backup FTP server, use the file copy method for backup.

[0074] 3) When the original video file storage FTP server and the backup FTP server are not on the same computer, use the FTP protocol to upload the backup.

[0075] 4) After a successful backup, record the file backup FTP information to the database and update the file backup FTP address information to the FTP scheduling module.

[0076] The idle recovery file is used when disk I / O usage is very low to read damaged or corrupted files from the disk and initiate file recovery. Figure 5 As shown, it includes the following steps:

[0077] 1) If the file is of the type of disk corruption, find the corresponding backup file, then find a usable FTP server and upload the backup file to that FTP server;

[0078] 2) If the file is corrupted, find the corresponding backup file and upload it to another available FTP server: If the available FTP server and the backup file are on the same computer, simply copy the file to restore it; if the available FTP server and the backup file are not on the same computer, use the FTP protocol to upload the file to restore it.

[0079] 3) After the recovery is complete, record the normal FTP server address information into the database, and update the FTP address information to the FTP scheduling module at the same time.

[0080] The following is a comparison of the video file processing effects of the method of this invention with existing technologies:

[0081] 1. FTP file storage, because the file data is divided into multiple storage blocks by RAID 0, supports simultaneous read and write operations on multiple blocks, resulting in low disk I / O and fast file storage response. The write and read speeds for a 1GB file are as follows:

[0082] type IO write speed Total time to complete file writing Regular Disk FTP 89MB / s 11.5 seconds This invention relates to RAID0 disk FTP. 460MB / s 2.23 seconds

[0083] type IO read speed Total time to complete file reading Regular Disk FTP 109MB / s 9.4 seconds This invention relates to RAID0 disk FTP. 990MB / s 1.03 seconds

[0084] 2. FTP file storage adds a file recovery mechanism. When a file is corrupted, it can be recovered using the idle backup and recovery module, making file storage safer and more reliable.

[0085]

[0086] 3. In FTP storage systems, large video files can be uploaded quickly.

[0087] Hadoop's HDFS distributed file system uses the HTTP protocol and stores files in 64MB blocks by default, making file uploads slower than FTP storage.

[0088] RAID0 disk FTP storage systems use the FTP protocol, which offers a significant speed advantage for uploading large files. The reasons are as follows:

[0089] 1) No meta-data was added to the sent data; only the original binary file was transmitted.

[0090] 2) There is no excessive block coding.

[0091] The upload speed test results for uploading 10 files (each file is 1GB in size) are as follows:

[0092]

Claims

1. A method for processing distributed video files via FTP based on RAID disks, characterized in that, include: S1. Configure the corresponding RAID0 disk drive letter and storage directory parameters on the FTP server, start the FTP service, and then upload the running status of the FTP server to the FTP scheduling module in real time. S2. The FTP client calculates the size of the video file to be uploaded, the hash value of the file content, and the hash value of the file block content, and then uploads it to the FTP scheduling module. S3. The FTP scheduling module checks the FTP server for a file with the same size and hash value as the video file to be uploaded. If a file exists, the module returns the FTP address of that file to the client. If no file exists, the module selects an optimal FTP server address and returns it to the client, who then uploads the video file to that address. The selection of the optimal FTP server includes the following steps: 1) Based on the size of the file to be uploaded, filter out FTP servers with remaining storage space smaller than the size of the file to be uploaded; 2) Select the best FTP server from among those with sufficient storage space to store files; the best FTP server can be selected based on the following methods: In a single-server scenario, deploy 2-3 RAID0 disks and corresponding FTP servers. Calculate a weighted score based on the FTP server's disk I / O unutilization, network unutilization, CPU unutilization, and memory unutilization. Sort the scores and select the FTP server with the highest score as the best FTP server. In a multi-server scenario, a weighted score is calculated based on the FTP server's disk I / O unutilization rate, network unutilization rate, CPU unutilization rate, and memory unutilization rate. The scores are then sorted, and the FTP server with the highest score is selected as the best FTP server. S4. After the video file is successfully uploaded, the FTP scheduling module stores the unique ID, name, size, hash value, and FTP address of the video file in the database.

2. The FTP distributed video file processing method based on RAID disks according to claim 1, characterized in that: It also includes creating a backup file task, which, when there are no FTP file storage tasks on the FTP server and disk I / O usage is low, initiates the backup file task to back up the files to another FTP server, including the following steps: 1) Find a usable FTP server and use it as the backup FTP server; 2) When the original FTP server and the backup FTP server are on the same computer, use the file copy method for backup; when the original FTP server and the backup FTP server are not on the same computer, use the FTP protocol upload method for backup. 3) After a successful backup, record the file backup FTP server information to the database, and update the file backup FTP server address information to the FTP scheduling module.

3. The FTP distributed video file processing method based on RAID disks according to claim 2, characterized in that: It also includes idle recovery files, which read damaged or corrupted files from the disk and initiate recovery when disk I / O usage is very low; including the following steps: 1) If the file is of the type of disk corruption, find the corresponding backup file, then find a usable FTP server and upload the backup file to that FTP server; 2) If the file is corrupted, find the corresponding backup file and upload it to another available FTP server: If the available FTP server and the backup file are on the same computer, simply copy the file to restore it; if the available FTP server and the backup file are not on the same computer, use the FTP protocol to upload the file to restore it. 3) After the recovery is complete, record the normal FTP server address information into the database, and update the FTP address information to the FTP scheduling module at the same time.

4. The FTP distributed video file processing method based on RAID disks according to claim 3, characterized in that, Also includes: S5. Download the video file. The FTP client communicates with the FTP scheduling module to find the FTP server download address corresponding to the video file. If the file corresponding to the main FTP server download address is inaccessible, the client finds the corresponding backup FTP server download address. Based on the FTP server address, the FTP client communicates with the FTP server to download the video file.

5. A distributed video file processing system based on RAID disks via FTP, characterized in that, include: An FTP client used to calculate the size of the video file to be uploaded, the hash value of the file content, and the hash value of the file chunk content; The FTP server is used to configure the storage directory parameters and storage files for the corresponding RAID0 disk drive letter. The FTP scheduling module is used to search the FTP server for a file with the same size and hash value as the video file to be uploaded, calculated by the FTP client based on the file size, file content hash value, and file chunk content hash value. If a file exists, the module returns the FTP file address of the file to the client; otherwise, it selects an optimal FTP server address and returns it to the client, which then uploads the video file to that address. The selection of the optimal FTP server includes the following steps: 1) Based on the size of the file to be uploaded, filter out FTP servers with remaining storage space smaller than the size of the file to be uploaded; 2) Select the best FTP server from among those with sufficient storage space to store files; the best FTP server can be selected based on the following methods: In a single-server scenario, deploy 2-3 RAID0 disks and corresponding FTP servers. Calculate a weighted score based on the FTP server's disk I / O unutilization, network unutilization, CPU unutilization, and memory unutilization. Sort the scores and select the FTP server with the highest score as the best FTP server. In a multi-server scenario, a weighted score is calculated based on the FTP server's disk I / O unutilization rate, network unutilization rate, CPU unutilization rate, and memory unutilization rate. The scores are then sorted, and the FTP server with the highest score is selected as the best FTP server.

6. The FTP distributed video file processing system based on RAID disks according to claim 5, characterized in that... Also includes: The idle backup and restore module is used to back up and restore video files stored on the FTP server.

Images