Method and system for optimizing damage file determination

Abstract

The present application relates to a kind of damaged file optimization judgment method and system thereof, including steps S1, whether the type of file is compressed file, when the type of file is compressed file, set the threshold of empty byte proportion in compressed file, when hash check code verification error or the empty byte proportion in compressed file is greater than threshold, then identify the compressed file as damaged file;Step S2, when the type of file is non-compressed file, whether consistent according to resolution, frame rate, duration and video content in metadata is verified, or whether there is exception in parsing process, when verification is inconsistent or parsing exception, then identify compressed file as damaged file;Step S3, the path of being identified as damaged file is saved to damaged record list and saved.The present application can better find damaged file in file, so that early discovery, timely repair means is taken.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of file system, in particular to a method and system for optimizing judgment of damaged files. BACKGROUND

[0002] In a storage system, many files are unknown whether damaged or not, and the files themselves have no check code for self-checking, or some have self-checking code correct, but the files themselves may be damaged, and the check code generated after the file is damaged is also correct. If the damaged file is not found in time, the file is lost, and when it is found, other backups may have been lost. SUMMARY

[0003] To overcome the above problems, the purpose of the present application is to provide a method and system for optimizing judgment of damaged files, which can better find damaged files in files, so as to find them early and take timely repair measures.

[0004] The present application adopts the following scheme: a method for optimizing judgment of damaged files, the method comprising the following steps:

[0005] Step S1, judging whether the type of the file is a compressed file, when the type of the file is a compressed file, setting a threshold value of the empty byte proportion in the compressed file, verifying whether the hash check code in the compressed file is incorrect or the empty byte proportion in the compressed file is greater than the threshold value, when the hash check code is verified to be incorrect or the empty byte proportion in the compressed file is greater than the threshold value, then the compressed file is determined as a damaged file;

[0006] Step S2, when the type of the file is a non-compressed file, verifying whether the resolution, frame rate, time length and video content in the metadata are consistent, or whether there is an exception in the parsing process, when the verification is inconsistent or there is an exception in the parsing process, then the compressed file is determined as a damaged file;

[0007] Step S3, saving the path of the damaged file determined as a damaged file to a damaged record list.

[0008] Further, the step S1 further comprises: when the type of the file is a compressed file, identifying whether the file has a built-in hash value, when the compressed file has a built-in hash value, verifying whether the hash value in the compressed file is incorrect, and marking the file as damaged when the hash value in the compressed file is incorrect; when the file has no built-in hash value, setting a threshold value of the empty byte in the compressed file, and judging whether the empty byte proportion in the file is greater than the threshold value.

[0009] Further, the step S2 "verify whether the resolution, frame rate, duration and video content in the metadata are consistent, or whether there is an exception in the parsing process" is further: when the non-compressed file is a video file, verify whether the resolution, frame rate, duration and video content in the metadata are consistent, or whether there is an exception in the parsing process; when the non-compressed file is a picture file, only verify whether the picture resolution in the metadata is consistent or whether there is an exception in the parsing process.

[0010] Further, the step S1 "verify whether the hash check code in the compressed file is incorrect" is further: read the crc check code stored when the file is compressed by winrar, recalculate the crc check code by crc algorithm, and then compare it with the original recorded crc check code. If they are not the same, it is verified that the hash check code in the compressed file is incorrect. If they are the same, it is verified that the hash check code in the compressed file is correct.

[0011] A system for optimizing the judgment of damaged files, the system comprising a compressed file damage judgment module, a non-compressed file damage judgment module and a damaged file saving module;

[0012] The compressed file damage judgment module: judges whether the type of the file is a compressed file, sets a threshold for the proportion of empty bytes in the compressed file when the type of the file is a compressed file, verifies whether the hash check code in the compressed file is incorrect or the proportion of empty bytes in the compressed file is greater than the threshold, and determines that the compressed file is a damaged file when the hash check code is incorrect or the proportion of empty bytes in the compressed file is greater than the threshold.

[0013] The non-compressed file damage judgment module: when the type of the file is a non-compressed file, verifies whether the resolution, frame rate, duration and video content in the metadata are consistent, or whether there is an exception in the parsing process, and determines that the compressed file is a damaged file when the verification is inconsistent or the parsing is abnormal.

[0014] The damaged file saving module: saves the path of the file determined to be a damaged file to a damaged record list.

[0015] Further, the "when the type of the file is a compressed file, verify whether the hash check code in the compressed file is incorrect or the proportion of empty bytes in the compressed file is greater than the threshold" in the compressed file damage judgment module is further: when the type of the file is a compressed file, identify whether the file has a built-in hash value, verify whether the hash value in the compressed file is incorrect when the compressed file has a built-in hash value, and mark the file as damaged when the hash value in the compressed file is incorrect; when the file does not have a built-in hash value, set a threshold for the empty bytes in the compressed file, and judge whether the proportion of empty bytes in the file is greater than the threshold.

[0016] Further, the "whether consistent according to resolution, frame rate, time length and video content in the metadata, or whether there is an exception in the parsing process" in the non-compressed file damage judgment module is further: when the non-compressed file is a video file, whether consistent according to resolution, frame rate, time length and video content in the metadata, or whether there is an exception in the parsing process, when the non-compressed file is a picture file, only need to verify whether the picture resolution in the metadata is consistent or whether there is an exception in the parsing process.

[0017] Further, the "whether the hash check code in the compressed file is wrong" in the compressed file damage judgment module is further: by reading the crc check code stored when winrar is compressed, the crc algorithm is recalculated, and then compared with the original recorded crc check code, if the comparison is not the same, it is verified that the hash check code in the compressed file is wrong, if the comparison is the same, it is verified that the hash check code in the compressed file is correct.

[0018] The beneficial effects of the present application are that: in the case of normal operation of the file system, the internal metadata of the file is identified to determine whether the file is damaged, the damaged file in the file can be better found through the technical scheme of the present application, different identification and determination methods are matched according to different file types, the damaged file can be found early, and the damaged file found early can be repaired, if the time is long, the possibility of repair will be reduced. BRIEF DESCRIPTION OF DRAWINGS

[0019] Figure 1 It is a method flowchart of the present application.

[0020] Figure 2 It is a system principle block diagram of the present application. DETAILED DESCRIPTION

[0021] The present application will be further described below in combination with the drawings.

[0022] Please refer to Figure 1 The damaged file optimization judgment method of the present application, the method comprises the following steps:

[0023] Step S1, judge whether the type of the file is a compressed file, when the type of the file is a compressed file, set the threshold value of the empty byte proportion in the compressed file, verify whether the hash check code in the compressed file is wrong or the empty byte proportion in the compressed file is greater than the threshold value, when the hash check code is wrong or the empty byte proportion in the compressed file is greater than the threshold value, the compressed file is determined as a damaged file;

[0024] Step S2, when the file type is a non-compressed file, verify whether the resolution, frame rate, duration and video content in the metadata are consistent, or whether there are abnormalities in the parsing process, when the verification is inconsistent or the parsing is abnormal, the compressed file is determined to be a damaged file;

[0025] Step S3, save the path of the damaged file identified as a damaged file to the damaged record list.

[0026] A damaged file optimization judgment method,

[0027] 1. Determine whether the file type is a compressed file, when the file type is a compressed file, set a threshold for the proportion of empty bytes in the compressed file, verify whether the hash check code in the compressed file is incorrect or the proportion of empty bytes in the compressed file is greater than the threshold, when the hash check code verification is incorrect or the proportion of empty bytes in the compressed file is greater than the threshold, then determine that the compressed file is a damaged file;

[0028] 1.1 "When the file type is a compressed file, verify whether the hash check code in the compressed file is incorrect or the proportion of empty bytes in the compressed file is greater than the threshold" is further: when the file type is a compressed file, identify whether the file has a built-in hash value, when the compressed file has a built-in hash value, verify whether the hash value in the compressed file is incorrect, and if the hash value in the compressed file is incorrect, mark the file as damaged; when the file does not have a built-in hash value, set a threshold for the empty bytes in the compressed file, and determine whether the proportion of empty bytes in the file is greater than the threshold.

[0029] 1.2 "Verify whether the hash check code in the compressed file is incorrect" is further: by reading the crc check code stored when winrar is compressed, recalculate through crc algorithm, and then compare with the original recorded crc check code, if the comparison is not the same, it is verified that the hash check code in the compressed file is incorrect, if the comparison is the same, it is verified that the hash check code in the compressed file is correct.

[0030] For example: a.rar file, verify that the crc check code is incorrect, save the path of the file to the damaged record list.

[0031] For example: a.mp4 file, 50% of the file is found to be empty bytes, save the path of the file to the damaged record list.

[0032] 2. When the file type is a non-compressed file, verify whether the resolution, frame rate, duration and video content in the metadata are consistent, or whether there are abnormalities in the parsing process, when the verification is inconsistent or the parsing is abnormal, the compressed file is determined to be a damaged file;

[0033] 2.1 The statement "Verify consistency between the resolution, frame rate, duration, and video content in the metadata, or check for anomalies during parsing" is further defined as follows: When the uncompressed file is a video file, verify consistency between the resolution, frame rate, duration, and video content in the metadata, or check for anomalies during parsing. When the uncompressed file is an image file, only verify consistency between the image resolution and the metadata, or check for anomalies during parsing. The anomalies primarily involve format errors that prevent parsing the video.

[0034] For example, if the file b.mp4 has a resolution of 1920x1080, a frame rate of 24 frames per second, and a duration of 1 minute, but decoding the video reveals that its duration is only 30 seconds, it will be marked as corrupted.

[0035] 3. Save the paths of files identified as corrupted to the corruption record list.

[0036] The present invention also provides a system for optimizing and judging corrupted files, the system including a compressed file corruption judgment module, an uncompressed file corruption judgment module, and a corrupted file saving module;

[0037] The compressed file corruption judgment module: determines whether the file type is a compressed file. When the file type is a compressed file, it sets a threshold for the proportion of empty bytes in the compressed file, verifies whether the hash check code in the compressed file is incorrect or whether the proportion of empty bytes in the compressed file is greater than the threshold. When the hash check code is incorrect or the proportion of empty bytes in the compressed file is greater than the threshold, the compressed file is identified as a corrupted file.

[0038] 1.1 "When the file type is a compressed file, verify whether the hash checksum in the compressed file is incorrect or whether the proportion of empty bytes in the compressed file is greater than the threshold" is further defined as follows: When the file type is a compressed file, identify whether the file has a built-in hash value. If the compressed file has a built-in hash value, verify whether the hash value in the compressed file is incorrect. If the hash value in the compressed file is incorrect, mark the file as corrupted. When the file does not have a built-in hash value, set a threshold for empty bytes in the compressed file and determine whether the proportion of empty bytes in the file is greater than the threshold.

[0039] 1.2 "Verifying whether the hash checksum in the compressed file is incorrect" is further defined as follows: by reading the CRC checksum stored during WinRAR compression, recalculating it using the CRC algorithm, and then comparing it with the originally recorded CRC checksum. If they are different, the hash checksum in the compressed file is incorrect; if they are the same, the hash checksum in the compressed file is correct.

[0040] For example, if the CRC checksum of a file (e.g., a.rar) is incorrect, the file path will be saved to the corrupted record list.

[0041] For example, if a file is a a.mp4 file and 50% of the file is found to be empty bytes, the path of the file is saved to the damaged record list.

[0042] The non-compressed file damage judgment module: when the file type is a non-compressed file, whether the resolution, frame rate, duration and video content in the metadata are consistent or whether there is an exception in the parsing process is verified, and when the verification is inconsistent or the parsing is abnormal, the compressed file is determined to be a damaged file; the exception is mainly that the format is found to be incorrect and cannot be parsed when the video is parsed.

[0043] 2.1 "According to the resolution, frame rate, duration and video content in the metadata, whether they are consistent or whether there is an exception in the parsing process" is further: when the non-compressed file is a video file, whether the resolution, frame rate, duration and video content in the metadata are consistent or whether there is an exception in the parsing process is verified, and when the non-compressed file is a picture file, only the consistency of the picture resolution with the metadata or whether there is an exception in the parsing process needs to be verified.

[0044] For example, the resolution of the b.mp4 file is 1920x1080, the frame rate is 24 frames, the duration is 1 minute, and the duration is found to be only 30 seconds by decoding the video, and it is marked as damaged.

[0045] The damaged file saving module: the path of the damaged file determined is saved to the damaged record list.

[0046] In summary, under the condition that the file system is running normally, the internal metadata of the file is identified to determine whether the file is damaged, and the damaged file in the file can be better found through the technical solution of the application, different identification and determination methods are matched according to different file types, the damaged file can be found early, and there may be a way to repair the damaged file found early, and the possibility of repair will decrease if the time is long.

[0047] The above only describes the preferred embodiments of the application, and any changes and modifications made within the scope of the application should be included in the scope of the application.

Claims

1. A method for optimizing and judging corrupted files, characterized in that, The method includes the following steps: Step S1: Determine if the file type is a compressed file. If the file type is a compressed file, set a threshold for the percentage of empty bytes in the compressed file, and verify whether the hash checksum in the compressed file is incorrect or whether the percentage of empty bytes in the compressed file is greater than the threshold. If the hash checksum is incorrect or the percentage of empty bytes in the compressed file is greater than the threshold, the compressed file is considered a corrupted file. Step S2: When the file type is an uncompressed file, verify whether the resolution, frame rate, duration and video content in the metadata are consistent, or whether there are any abnormalities in the parsing process. If the verification is inconsistent or the parsing is abnormal, the compressed file is identified as a corrupted file. Step S3: Save the paths of the files identified as corrupted to the corruption record list; The step S1, "when the file type is a compressed file, set a threshold for the proportion of empty bytes in the compressed file, and verify whether the hash check code in the compressed file is incorrect or whether the proportion of empty bytes in the compressed file is greater than the threshold", is further defined as follows: when the file type is a compressed file, identify whether the file has a built-in hash value; when the compressed file has a built-in hash value, verify whether the hash value in the compressed file is incorrect; if the hash value in the compressed file is incorrect, mark the file as corrupted. When there is no built-in hash value in the file, a threshold is set for the percentage of empty bytes in the compressed file to determine whether the percentage of empty bytes in the file is greater than the threshold.

2. The method for optimizing and judging corrupted files according to claim 1, characterized in that, The step S2, "verifying whether the resolution, frame rate, duration, and video content in the metadata are consistent, or whether there are any abnormalities during the parsing process," is further defined as follows: when the uncompressed file is a video file, verifying whether the resolution, frame rate, duration, and video content in the metadata are consistent, or whether there are any abnormalities during the parsing process; when the uncompressed file is an image file, only verifying whether the image resolution is consistent with the metadata or whether there are any abnormalities during the parsing process.

3. The method for optimizing and judging corrupted files according to claim 1, characterized in that, The step S1, "verifying whether the hash check code in the compressed file is incorrect," further involves: reading the CRC check code stored during WinRAR compression, recalculating it using the CRC algorithm, and then comparing it with the originally recorded CRC check code. If the comparison is different, the hash check code in the compressed file is verified to be incorrect; if the comparison is the same, the hash check code in the compressed file is verified to be correct.

4. A system for optimizing and judging corrupted files, characterized in that, The system includes a compressed file corruption detection module, an uncompressed file corruption detection module, and a corrupted file saving module; The compressed file corruption judgment module: determines whether the file type is a compressed file. When the file type is a compressed file, it sets a threshold for the proportion of empty bytes in the compressed file, verifies whether the hash check code in the compressed file is incorrect or whether the proportion of empty bytes in the compressed file is greater than the threshold. When the hash check code is incorrect or the proportion of empty bytes in the compressed file is greater than the threshold, the compressed file is determined to be a corrupted file. The uncompressed file corruption judgment module: When the file type is uncompressed, it verifies whether the resolution, frame rate, duration and video content in the metadata are consistent, or whether there are any abnormalities in the parsing process. If the verification is inconsistent or the parsing is abnormal, the compressed file is identified as a corrupted file. The damaged file saving module: saves the paths of files identified as damaged to a damage record list; The "when the file type is a compressed file, verify whether the hash check code in the compressed file is incorrect or whether the proportion of empty bytes in the compressed file is greater than the threshold" in the compressed file corruption judgment module is further defined as follows: when the file type is a compressed file, identify whether the file has a built-in hash value; when the compressed file has a built-in hash value, verify whether the hash value in the compressed file is incorrect; if the hash value in the compressed file is incorrect, mark the file as corrupted. When there is no built-in hash value in the file, a threshold is set for the percentage of empty bytes in the compressed file to determine whether the percentage of empty bytes in the file is greater than the threshold.

5. The system for optimizing and judging corrupted files according to claim 4, characterized in that, The "verify whether the resolution, frame rate, duration and video content in the metadata are consistent, or whether there are any abnormalities in the parsing process" in the uncompressed file corruption judgment module is further defined as follows: when the uncompressed file is a video file, verify whether the resolution, frame rate, duration and video content in the metadata are consistent, or whether there are any abnormalities in the parsing process; when the uncompressed file is an image file, it is only necessary to verify whether the image resolution is consistent with the metadata or whether there are any abnormalities in the parsing process.

6. The system for optimizing and judging corrupted files according to claim 4, characterized in that, The "verify whether the hash checksum in the compressed file is incorrect" function in the compressed file corruption judgment module is further defined as follows: by reading the CRC checksum stored during WinRAR compression, recalculating it using the CRC algorithm, and then comparing it with the originally recorded CRC checksum. If the comparison is different, the hash checksum in the compressed file is verified to be incorrect; if the comparison is the same, the hash checksum in the compressed file is verified to be correct.

Images