Method for fast computing hash value of large file

Abstract

The application discloses a method for quickly calculating hash value of a large file, and belongs to the technical field of computer and software development, which comprises the following steps: S1, system preparation and file path acquisition; starting and initializing the large file hash calculation system, and inputting a file to be calculated for hash value after initialization; S2, file basic information reading; acquiring creation time information, modification time information and file size of the file to be calculated for hash value; S3, file metadata analysis; extracting metadata of the file to be calculated for hash value, combining the extracted metadata according to a preset order and a preset format, obtaining a metadata string, and calculating MD5 hash value of the metadata string; S4, segment extraction of the content of the file to be calculated for hash value; and S5, calculation of the hash value of the file to be calculated for hash value, and output of the calculated hash value result. The application can significantly improve the calculation efficiency, can be adaptively fitted to specific file types, and has the characteristics of multi-dimensional information comprehensive consideration.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer and software development, and particularly relates to a method for quickly calculating hash value of large file. BACKGROUND

[0002] In the era of rapid development of digital information, the processing and management of large files occupy an extremely important position in many fields, and the calculation of file hash value is a key technology link for guaranteeing data integrity, security, and realizing efficient data retrieval and identification.

[0003] The traditional large file hash value calculation method is mainly based on complex hash operation on the entire content of the file. Although this method can ensure high accuracy when processing general files, it has obvious shortcomings for specific types of large files such as video and audio files. Video and audio files usually have the characteristics of relatively difficult to change content, complex internal data structure and large data volume. For example, a high-definition movie file may be as high as several GB or even tens of GB. If the traditional method is used to calculate the hash value, a large amount of computing time and system resources will be consumed, which seriously affects the computing efficiency. Moreover, in actual application scenarios such as large-scale multimedia database management, video and audio content copyright protection, and data integrity verification in network transmission, it is often necessary to quickly and accurately process a large number of video and audio files. There is an urgent need for a hash value calculation algorithm specially designed for video, audio and other large files, which can quickly and accurately calculate the hash value based on not completely calculating the entire content of the file, and comprehensively consider the file feature information, so as to meet the needs of the development of modern digital technology and promote the progress of related fields. SUMMARY

[0004] The purpose of the present application is to overcome the shortcomings of the prior art and provide a method for quickly calculating hash value of large file.

[0005] The purpose of the present application is achieved by the following technical solutions:

[0006] The present application discloses a method for quickly calculating hash value of large file, comprising the following steps:

[0007] S1, system preparation and file path acquisition, starting the large file hash calculation system and initializing the system environment, inputting the file to be calculated into the initialized large file hash calculation system;

[0008] S2, file basic information reading, acquiring the creation time information, modification time information and file size of the file to be calculated;

[0009] S3, file metadata analysis, the metadata of the file to be calculated is extracted, and the extracted metadata is combined according to a preset order and a preset format to obtain a metadata string, and the MD5 hash value of the metadata string is calculated;

[0010] S4, the content of the file to be calculated is segmented and extracted;

[0011] S5, the hash value of the file to be calculated is calculated, and the calculated hash value result is output.

[0012] Further, the initialization system environment in step S1 includes: loading the required library file and allocating memory space for data storage and processing.

[0013] Preferably, the step S1 further includes: checking the validity of the path of the file to be calculated, verifying whether the file really exists in the corresponding file path, if the file does not exist, prompting the user with error information and terminating the calculation.

[0014] Preferably, the step S2 specifically includes: using the file access interface provided by the operating system to obtain the creation time information and modification time information of the file to be calculated, and converting the obtained creation time information and modification time information into timestamp format and storing them in the corresponding variable; obtaining the file size of the file to be calculated in byte unit.

[0015] Preferably, the step S3 specifically includes: extracting the metadata of the file to be calculated by a metadata analysis engine or a library function, including the file author, the file creation software and the file description; combining the extracted metadata according to a preset order and a preset format to obtain a metadata string, and calculating the MD5 hash value of the metadata string by MD5 hash algorithm.

[0016] Preferably, the step S4 specifically includes the following steps:

[0017] S41, setting the reference size of the extracted content based on the file size of the file to be calculated;

[0018] S42, setting the reference size of the extracted content as 4M, opening the read stream of the file to be calculated, positioning the file pointer to the head of the file to be calculated, reading the first 4M file content, and storing it in the predefined head content buffer;

[0019] S43, starting from the 4M after the file head, setting a loop extraction mechanism, reading the first 1KB file content every 4M offset, appending the 1KB data read each time to the predefined extracted content buffer in turn, until the file pointer is less than 4M from the file tail.

[0020] S44, if the file pointer is less than 4M from the file tail, the file pointer is positioned to a position 4M from the file tail, the last 4M file content is read and stored in the predefined tail content buffer.

[0021] Preferably, the step S5 specifically comprises the following steps:

[0022] S51, the file creation time, file modification time, file size, MD5 hash value of the metadata string, file content in the head content buffer, file content in the extraction content buffer and file content in the tail content buffer are integrated in a preset order and input into a data structure;

[0023] S52, the hash value of the large file is calculated by the formula: large file hash value = MD5 (file creation time + file modification time + file size + MD5 hash value of the metadata string + file content in the head content buffer + file content in the extraction content buffer + file content in the tail content buffer);

[0024] S53, the calculated large file hash value is stored in the corresponding storage area;

[0025] S54, the calculation result of the large file hash value is output in the system interface or the log;

[0026] S55, the temporary data buffer and file reading stream generated in the calculation process are cleaned and released.

[0027] The beneficial effects of the present application are:

[0028] 1) Significantly improve the calculation efficiency: when processing large files, the present method discards the traditional way of calculating the hash value of the entire file content; by extracting only the file creation time, modification time, size, metadata and specific file content fragments (head 4M + every 4M extract 1KB + tail 4M) to calculate the hash value. The present application greatly reduces the calculation range, which can shorten the calculation time by several times or even dozens of times, greatly improves the speed of hash value calculation, and enables the system to complete a large number of hash processing tasks of large files in a shorter time, which is suitable for scenarios with high time efficiency requirements, such as fast indexing and verification of large-scale video file library.

[0029] 2) Targeted adaptation to specific file types: Designed specifically for file types such as video, audio, etc. that are relatively less likely to change. Since the header and tail of such files usually contain key format information, metadata information and some iconic content, while the middle part, although large in data volume, has relatively less content change, the sampling strategy of this algorithm can effectively capture the characteristic information of these files. For example, in a video file, the header may contain important information such as video encoding format, resolution, etc., and the tail may have file end markers, etc. By extracting these key parts and representative sampling content in the middle, the hash value obtained can still accurately reflect the overall characteristics of the file. On the premise of ensuring the accuracy of the hash, the calculation process is further optimized, improving the adaptability and effectiveness of specific file types.

[0030] 3) Multi-dimensional information comprehensive consideration: This method does not rely solely on file content to calculate hash values, but includes file creation time, modification time, size and metadata in the scope of hash calculation. This makes the hash value more fully reflect the characteristics of the file. This multi-dimensional comprehensive consideration increases the uniqueness and recognition of the hash value, which is of great significance in file management, copyright protection, data tracing, etc. BRIEF DESCRIPTION OF DRAWINGS

[0031] Figure 1 A step flow diagram of a method for quickly calculating hash values of large files according to an embodiment of the present application;

[0032] Figure 2 A system architecture diagram of a method for quickly calculating hash values of large files according to an embodiment of the present application. DETAILED DESCRIPTION

[0033] The technical solutions of the present application will be described in detail below with reference to the embodiments. Obviously, the described embodiments are only a part of the embodiments of the present application, not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0034] The application discloses a method for quickly calculating hash value of large files, solves many disadvantages of traditional hash value calculation methods of large files in processing specific large files such as videos and audios, and meets dual requirements of efficiency and accuracy of modern data processing. The application calculates hash value by comprehensively considering suffix, creation time and modification time of a file to be calculated and representative front, middle and rear contents in the file content, avoids lengthy operation on the whole file content, greatly shortens calculation time, enables hash value calculation to be quickly completed, and adapts to requirements of scenes such as real-time processing of multimedia databases. Meanwhile, ensuring accuracy and effectiveness of the hash value is also a key purpose. The file suffix helps to distinguish file types, the creation and modification time provides time dimension identification for the file, and the three content samples of the file content can accurately capture core features of the file, so that multi-factor comprehensive consideration ensures that the calculated hash value can reliably reflect uniqueness and integrity of the file, thereby playing a practical and effective role in copyright protection and data security verification, providing more advanced, efficient and reliable hash value calculation solutions for processing and management of large files such as videos and audios in various application fields, and promoting data processing technology of related industries to a new level. Figure 1 as shown in the accompanying drawings, and specifically comprises the following steps:

[0035] S1, system preparation and file path acquisition, starting a large file hash calculation system and initializing a system environment, inputting a file to be calculated into the initialized large file hash calculation system;

[0036] S2, file basic information reading, acquiring creation time information, modification time information and file size of the file to be calculated;

[0037] S3, file metadata analysis, extracting metadata of the file to be calculated, combining the extracted metadata according to a preset order and a preset format, obtaining a metadata string, and calculating an MD5 hash value of the metadata string;

[0038] S4, segment extraction of the file content to be calculated;

[0039] S5, calculating hash value of the file to be calculated, and outputting the calculated hash value.

[0040] Specifically, the initialized system environment in step S1 comprises: loading required library files and allocating memory space for data storage and processing, so as to ensure that the system is in a stable and executable state.

[0041] Specifically, the step S1 further comprises: checking the validity of the path of the file to be calculated the hash value, verifying whether the file actually exists in the corresponding file path, if the file does not exist, prompting the user with error information and terminating the calculation.

[0042] Specifically, the step S2 specifically comprises: using the file access interface provided by the operating system to obtain the creation time information and the modification time information of the file to be calculated the hash value, and converting the obtained creation time information and the modification time information into a timestamp format for subsequent calculation and storage in the corresponding variable; obtaining the file size of the file to be calculated the hash value in bytes. The new method of obtaining the modification time of the file is helpful to reflect the change history of the file, which serves as a dimension for hash calculation; the file size data can directly reflect the size characteristics of the file, providing basic data support for hash calculation.

[0043] Specifically, the step S3 specifically comprises: extracting the metadata of the file to be calculated the hash value through a metadata parsing engine or a library function, including file author, file creation software and file description, etc. For some special format files, special parsing algorithms or tools may be needed to ensure accurate extraction of metadata; combining the extracted metadata in a predetermined order and format to obtain a metadata string, for example, "Author: [specific author name]; Creation software: [software full name]; Description: [file related description content]"; calculating the MD5 hash value of the metadata string through the MD5 hash algorithm. This value will be an important part of the overall hash calculation and will be involved in subsequent calculations.

[0044] Specifically, the step S4 specifically comprises the following steps:

[0045] S41, setting the reference size of the extracted content based on the actual file size of the file to be calculated the hash value; it can be 4M, 8M, 16M, 32M, 64M, 128M, 256M, etc., and is not limited to a specific value, but is selected according to the actual working environment;

[0046] S42, assuming that the reference size of the extracted content is 4M, opening the read stream of the file to be calculated the hash value, positioning the file pointer to the head of the file to be calculated the hash value, reading the first 4M of the file content, and storing it in the predefined head content buffer; ensuring the accuracy and integrity of data reading;

[0047] S43, starting from the 4M after the file head, setting a loop extraction mechanism, reading the first 1KB of file content every 4M offset, and appending the 1KB data read each time to the predefined extracted content buffer in turn, this process continues until the file pointer is less than 4M from the file tail.

[0048] S44, if the file pointer is less than 4M from the file tail, the file pointer is positioned to a position 4M from the file tail, the last 4M file content is read and stored in the predefined tail content buffer. Thus, the three-section extraction of the file content is completed. Through this front, middle and rear section extraction method, the key part of the file content can be obtained and the overall calculation amount can be reduced.

[0049] Specifically, the step S5 specifically includes the following steps:

[0050] S51, the file creation time, the file modification time, the file size, the MD5 hash value of the metadata string, the file content in the header content buffer, the file content in the extracted content buffer and the file content in the tail content buffer are integrated in a predetermined order and input into a data structure;

[0051] S52, the hash value of the large file is calculated by the formula: large file hash value = MD5 (file creation time + file modification time + file size + MD5 hash value of metadata string + file content in header content buffer + file content in extracted content buffer + file content in tail content buffer); the calculated hash value of the large file integrates the information of the file in multiple aspects, and the calculation efficiency is improved through the specific content sampling method;

[0052] S53, the calculated large file hash value is stored in the corresponding storage area; for example, the local hash value database or the specific file index system, which is associated with the corresponding large file, so as to perform subsequent file identification, verification, comparison and other operations;

[0053] S54, the calculation result of the large file hash value is output in the system interface or the log, and the calculated large file hash value is displayed to the user or the related system module for recording, querying and subsequent processing; the integrity and usability of the entire hash calculation process are ensured, which is especially suitable for the large file hash value calculation scene of video, audio and other relatively stable content with high requirement on calculation efficiency;

[0054] S55, the temporary data buffer, file reading stream and other resources generated in the calculation process are cleaned and released, so as to ensure the effective recovery and utilization of system resources, avoid resource leakage and waste, and prepare for the next hash calculation task.

[0055] The application aims at efficiently and accurately calculating the hash value of specific large files such as videos and audios. Selective content sampling, in the embodiment, the file head 4M, tail 4M and 1KB content extracted every 4M are accurately selected to avoid full calculation, while reducing the amount of calculation, effectively capturing the key features of the file, sampling the content of large files such as videos and audios in a unique way, significantly improving the calculation efficiency and ensuring the representativeness of the hash; Multi-dimensional data fusion hash: integrating file creation time, modification time, size and metadata (author, creation software, description) with specific file content, calculating the metadata MD5 value first and then calculating the overall MD5 hash, so as to fuse multi-dimensional file information into the hash value, enhance the comprehensiveness and uniqueness of the hash result; Specific file type adaptation: based on the relatively stable characteristics of video and audio file content, the calculation method is designed, the structure characteristics of such files are analyzed in depth, and the sampling strategy is optimized to adapt to the format specification, so as to ensure efficient and accurate processing of such files during calculation, and to improve the applicability and reliability of hash calculation for specific file types; The application breaks through the limitation of relying solely on file content. By combining time, content and suffix to calculate the hash value, the calculation efficiency is significantly improved on the basis of ensuring the accuracy and effectiveness of the hash value, especially suitable for large file hash calculation scenarios with relatively fixed content such as videos and audios, and is expected to be widely applied in multimedia data processing, file management and security verification and other fields, and to promote the development and progress of related technologies.

[0056] The above only describes the preferred embodiments of the application, and it should be understood that the application is not limited to the forms disclosed herein, and should not be considered as excluding other embodiments, but can be used in various other combinations, modifications and environments, and can be modified within the scope of the concepts described herein by the above teachings or related technical or knowledge. The modifications and changes made by those skilled in the art without departing from the spirit and scope of the application shall be within the protection scope of the appended claims of the application.

Claims

1. A method for quickly calculating the hash value of a large file, characterized in that: The following steps are involved: S1. System preparation and file path acquisition: starting the large file hash calculation system and initializing the system environment; inputting the file to be calculated as hash value into the initialized large file hash calculation system; S2. Read the basic file information to obtain the creation time, modification time, and file size of the file to be hashed. S3. File metadata parsing: extracting metadata from the file for which the hash value is to be calculated, combining the extracted metadata in a preset order and format to obtain a metadata string, and calculating the MD5 hash value of the metadata string; S4. Extract the contents of the file to be hashed by segments; S5. Calculate the hash value of the file to be hashed, and output the calculated hash value result; Step S3 specifically includes: extracting metadata of the file to be hashed, including the file author, file creation software, and file description, using a metadata parsing engine or library function; combining the extracted metadata in a predetermined order and format to obtain a metadata string; and calculating the MD5 hash value of the metadata string using the MD5 hash algorithm; Step S4 specifically includes the following steps: S41, setting a base size for the extracted content based on the file size of the file to be calculated for the hash value; S42: Assume that the base size of the extracted content is 4MB, open the read stream of the file to be hashed, position the file pointer at the header of the file to be hashed, read the first 4MB of the file content, and store it in a predefined header content buffer; S43. Starting from 4MB after the file header, a loop extraction mechanism is set up to read the first 1KB of the file content at an offset of 4MB, and append the 1KB of data read each time to the predefined extraction content buffer in sequence until the file pointer is less than 4MB away from the end of the file; S44. If the file pointer is less than 4MB away from the end of the file, position the file pointer to a position 4MB away from the end of the file, read the last 4MB of the file content, and store it in a predefined end content buffer; Step S5 specifically includes the following steps: S51, integrating the file creation time, file modification time, file size, MD5 hash value of the metadata string, file content in the header content buffer, file content in the extracted content buffer, and file content in the tail content buffer in a preset order, and inputting the result into a data structure; S52. Calculate the hash value of the large file using the formula: Large file hash value = MD5 (file creation time + file modification time + file size + MD5 hash value of the metadata string + file content in the header content buffer + file content in the extracted content buffer + file content in the tail content buffer); S53, storing the calculated large file hash value in the corresponding storage area; S54. Outputting the calculation result of the hash value of the large file in the system interface or log; S55. Clean up and release the temporary data buffer and file reading stream generated during the calculation process.

2. A method for quickly calculating the hash value of a large file according to claim 1, characterized in that: Initializing the system environment in step S1 includes: loading required library files and allocating memory space for data storage and processing.

3. A method for quickly calculating the hash value of a large file according to claim 2, characterized in that: The step S1 further includes: performing a validity check on the path of the file to be calculated for the hash value, verifying whether the file actually exists in the corresponding file path; if the file does not exist, prompting the user with an error message and terminating the calculation.

4. A method for quickly calculating the hash value of a large file according to claim 3, characterized in that: The step S2 specifically includes: using the file access interface provided by the operating system to obtain the creation time information and modification time information of the file to be calculated for the hash value, and converting the obtained creation time information and modification time information into a timestamp format and storing them in the corresponding variables; obtaining the file size of the file to be calculated for the hash value in bytes.

Images