File verification method and device, electronic equipment and medium

Abstract

The embodiment of the invention provides a file verification method, which is applied to the field of file verification, and comprises the following steps of: obtaining a to-be-verified target file, performing data grouping on the target file to obtain at least one data packet, performing compression calculation on the at least one data packet to obtain a data packet compression value, and according to the data packet compression values of all the data packets, verifying the file according to the data packet compression value. The target file compression value is obtained, the historical file compression value of the target file is obtained, the historical file compression value is adopted to verify the target file compression value, and the verification result of the target file is obtained. According to the method, the files are grouped, the multiple grouped data packets are subjected to compression calculation, the compression values of the files are obtained according to the compression values of the data packets, and a historical compression value verification mode is adopted, so that the file verification speed is increased.

Description

Technical Field

[0001] The present invention relates to the field of file verification, and in particular to a method, a device, an electronic device and a medium for file verification. Background Art

[0002] With the rapid development of big data, the size and complexity of modern files are growing exponentially. From massive log data, multimedia files to complex scientific computing results, the size and diversity of data pose unprecedented challenges to data integrity verification. Data integrity verification is not only an important part of ensuring data security, but also a key means to ensure that data is not tampered with or damaged during storage, transmission and processing.

[0003] In the big data environment, the importance of data integrity verification is reflected in many aspects. First, during the storage and transmission of large-scale data sets, any minor errors or tampering may lead to serious consequences, such as deviations in data analysis results, decision-making errors, and even security vulnerabilities. Second, with the popularization of data sharing and collaboration, the frequency of data transmission between different systems, platforms, and networks has increased significantly, making data integrity verification a necessary measure to ensure data consistency and reliability.

[0004] Related technologies implement file integrity verification by receiving complete data of large files, but the efficiency of uploading complete data is low, and it occupies a large network bandwidth at the same time, which may affect other businesses, and the calculation speed is relatively slow. Summary of the invention

[0005] In view of the above problems, a method, device, electronic device and medium for file verification are proposed to overcome the above problems or at least partially solve the above problems, including:

[0006] A method for file verification, characterized in that the method comprises:

[0007] Get the target file to be verified;

[0008] Grouping the target file into data packets to obtain at least one data packet;

[0009] Performing compression calculation on the at least one data packet respectively to obtain a data packet compression value;

[0010] According to the data packet compression values ​​of all data packets, the target file compression value is obtained;

[0011] A historical file compression value of the target file is obtained, and the historical file compression value is used to verify the target file compression value to obtain a verification result of the target file.

[0012] In an optional embodiment of the present application, before grouping the target file to obtain at least one data packet, the method includes:

[0013] Obtaining the data volume and file identifier of the target file;

[0014] Determining the number of groups of the target file according to a first preset length and the data volume of the target file;

[0015] The step of grouping the target file to obtain at least one data packet includes:

[0016] The target file is read according to the file identifier and the first preset length to obtain the at least one data packet of the grouped quantity.

[0017] In an optional embodiment of the present application, the at least one data packet includes a tail data packet, and the data grouping of the target file to obtain at least one data packet includes:

[0018] Determining the data length of the last data packet, and determining the relationship between the data length of the last data packet and a first preset length;

[0019] When it is determined that the data length of the tail data packet is less than the first preset length, the tail data packet is expanded according to a preset rule to obtain an expanded tail data packet of the first preset length.

[0020] In an optional embodiment of the present application, before respectively performing compression calculation on the at least one data packet to obtain a data packet compression value, the method includes:

[0021] Evenly dividing the at least one data packet according to a second preset length to obtain a first target number of target words;

[0022] Performing a translation calculation on a first designated target word among the first target number of target words to obtain a translation target word;

[0023] Performing an XOR operation on the translation target word and a second designated target word among the first target number of target words to obtain an extended word of a second target number;

[0024] The second target number of extension words is determined as the at least one data packet.

[0025] In an optional embodiment of the present application, performing compression calculation on the at least one data packet respectively to obtain a data packet compression value includes:

[0026] When it is determined that the at least one data packet is not empty, compression calculation is performed on the at least one data packet for a preset number of times to obtain the data packet compression value.

[0027] In an optional embodiment of the present application, the step of obtaining the target file to be verified includes:

[0028] Creating a plurality of file reading threads for reading the target file;

[0029] The target file is read by a plurality of the file reading threads.

[0030] In an optional embodiment of the present application, before respectively performing compression calculation on the at least one data packet to obtain a data packet compression value, the method includes:

[0031] Creating a data buffer, and storing the at least one data packet in the data buffer;

[0032] Create multiple cache reading threads for reading data from the data buffer;

[0033] The data buffer is read by a plurality of cache reading threads to obtain at least one data packet.

[0034] A device for file verification, comprising:

[0035] A file acquisition module, used to acquire the target file to be verified;

[0036] A data grouping module, used for grouping the target file to obtain at least one data packet;

[0037] A compression calculation module, used to perform compression calculation on the at least one data packet to obtain a data packet compression value;

[0038] A compression determination module, used for obtaining a target file compression value according to the data packet compression values ​​of all data packets;

[0039] The result verification module is used to obtain the historical file compression value of the target file, and use the historical file compression value to verify the target file compression value to obtain the verification result of the target file.

[0040] An electronic device comprises a processor, a memory and a computer program stored in the memory and capable of running on the processor, wherein the computer program implements the above-mentioned file verification method when executed by the processor.

[0041] A computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the file verification method described above is implemented.

[0042] The embodiments of the present application have the following advantages:

[0043] By obtaining the target file to be verified, grouping the target file data to obtain at least one data packet, performing compression calculation on at least one data packet respectively to obtain the data packet compression value, obtaining the target file compression value according to the data packet compression values ​​of all data packets, obtaining the historical file compression value of the target file, and using the historical file compression value to verify the target file compression value to obtain the verification result of the target file. By grouping the files, and performing compression calculation on the grouped multiple data packets, obtaining the file compression value according to the data packet compression value, and using the historical compression value verification method, the speed of file verification is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the technical solution of the present invention, the accompanying drawings required for use in the description of the present invention will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.

[0045] Figure 1 It is a flowchart of the steps of a file verification method provided by some embodiments of the present application;

[0046] Figure 2 is a system architecture diagram of an encryption system and a proxy system provided in an embodiment of the present application;

[0047] Figure 3 is a functional module relationship diagram between an encryption system and an agent system provided in some embodiments of the present application;

[0048] Figure 4 This is a flowchart of another method for file verification provided by an embodiment of the present application;

[0049] Figure 5 It is an autonomous process of the proxy system in some embodiments of the present application;

[0050] Figure 6 is the encryption process of the encryption system in some embodiments of the present application;

[0051] Figure 7 This is a structural block diagram of a file verification device provided in some embodiments of the present application. DETAILED DESCRIPTION

[0052] In order to make the above-mentioned purposes, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0053] Reference Figure 1 , shows a flowchart of a method for file verification provided by an embodiment of the present invention, which may specifically include the following steps:

[0054] Step 101: Obtain the target file to be verified;

[0055] The target file to be verified refers to the file that needs to be verified, checked or confirmed in a certain process. This file may contain data, code, configuration information, etc. Its purpose is to ensure the integrity, accuracy or security of the file.

[0056] In a specific implementation, the target file to be verified may be a large file but is not limited to a large file. A large file generally refers to a file whose size exceeds the size of a regular file in a computer system, so that special attention or special measures are required when processing, storing or transmitting. There is no fixed standard for the size of a large file, but generally speaking, a large file may reach hundreds of megabytes (MB), several gigabytes (GB) or even several terabytes (TB). In practical applications, the processing of large files generally involves video editing, scientific computing, big data analysis, database backup, virtual machine mirroring and other fields. Large files require a large amount of storage space, and may require the use of high-capacity hard disks, network storage (NAS) or cloud storage services; processing large files may require more powerful computing resources, such as more memory, faster processors or specialized processing algorithms; transmitting large files may take longer, and may require the use of high-speed network connections or specialized file transfer protocols. When transmitting or storing large files, special attention should be paid to the integrity of the data to prevent file damage or data loss.

[0057] In some implementations of this embodiment, all steps of file verification can be performed through cooperation between the encryption system and the proxy system. Figure 2 The system architecture diagram of the encryption system and the proxy system provided in some embodiments of the present application is given, which is composed of the encryption system and multiple proxy systems. Each proxy system is deployed in a different file system, and regularly reads the target file to be verified and sends data.

[0058] The file system is a mechanism used in the operating system to manage files and directories on computer storage devices. The file system defines the organization, storage, access rights, naming rules, etc. of files, allowing users and applications to easily store, retrieve, and manage data. The file system is responsible for storing files on storage devices (such as hard disks, SSDs, flash memory, etc.) and ensuring the integrity and reliability of file data. In addition, the file system provides a file access interface that allows users and applications to read, write, modify, and delete files. The file system also supports a directory structure, and users can organize and manage files through directories to achieve hierarchical storage and access of files.

[0059] Step 102: grouping the target file into data packets to obtain at least one data packet;

[0060] Grouping the target file to obtain at least one data packet means dividing a larger file into multiple smaller data units, which are also called "data packets". The embodiment of the present application improves the transmission efficiency and processing efficiency of large files by grouping the target files, and also makes the management of the target files more convenient.

[0061] In some implementations of this embodiment, the target files may be grouped by the proxy system in the above system architecture diagram to obtain grouped data packets. Figure 3 As shown, it is a functional module relationship diagram between the encryption system and the proxy system provided in some embodiments of the present application. The proxy system mainly includes a file reading module and a data sending module. Among them, the file reading module records all files that need integrity verification and the corresponding time, reads files regularly and actively calls the data sending module to send files to the encryption system to realize the verification of large files. Specifically, the out-of-order synchronous sending of multiple data packets after grouping can also be realized by creating and initializing multiple threads.

[0062] In the specific implementation, after the proxy system is started, it will automatically read the configured files that need to be verified for integrity, and actively call the data sending module to implement file integrity verification. During the sending process, the data will be divided to obtain at least one data packet. The IP address and port of the encryption system are configured in the data sending module. The IP address is used to uniquely identify devices in the network (such as computers, servers, routers, etc.), which is similar to the "house number" of the device, ensuring that the data can be accurately sent to the target device. The port is used to identify a specific application or service on the device, which is similar to the "room number" of the device, ensuring that the data can be accurately sent to the target application. Through the combination of IP addresses and ports, network communication can achieve accurate data transmission between devices and applications. The data sending module will label the data and mark the actual location of the data packet in the file.

[0063] Specifically, in the first step, the target file is divided into multiple smaller parts according to certain rules (such as fixed size, specific format, etc.), each part can be processed independently, which helps to improve parallel processing capabilities; in the second step, each divided part is encapsulated into a data packet. Data packets usually contain the data itself and some additional information, such as the sequence number, checksum, source address, destination address, etc. of the data packet, which helps to correctly reassemble the file at the receiving end; in the third step, the generated data packets can be transmitted or stored independently. During the transmission process, the data packets can be distributed to different receiving ends through the network, and the receiving ends can reassemble the original file according to the sequence number and checksum information of the data packets. In the fourth step, at the receiving end, the data packets are reassembled according to the sequence number or other identification information to restore the original file. This process requires ensuring that all data packets are received correctly and that the order of multiple data packets is correct when reassembled.

[0064] Step 103: performing compression calculation on at least one data packet respectively to obtain a data packet compression value;

[0065] Among them, compression computing is a process of compressing data, reducing the demand for data storage space and transmission bandwidth by removing redundant information, optimizing coding and other technical means.

[0066] After the target file is divided into multiple data packets, each data packet can be compressed and calculated to obtain the compression value corresponding to each data packet. Among them, the compression calculation can be specifically SM3 compression calculation, which is the core step of the SM3 algorithm. The SM3 algorithm is mainly used in the fields of digital signature and verification, message authentication code generation and verification, random number generation, etc. The SM3 algorithm can ensure the integrity and authenticity of the data. The SM3 algorithm has the characteristics of one-way, collision resistance and anti-image attack. Through the steps of message padding, message grouping, initialization, message expansion, compression function and output, the input message of any length is compressed into a 256-bit hash value. Through a series of complex logical operations and permutation functions, the input data is gradually compressed into a message digest of fixed length or fixed number of bits. During the compression calculation process, the state variables are continuously updated to ensure the randomness and unpredictability of the message digest, and finally a fixed-length message digest is generated for data integrity verification and secure communication.

[0067] In some implementations of this embodiment, this step can be implemented by the above encryption system, such as Figure 3As shown, the encryption system consists of a message receiving module, a message expansion module, an iterative compression module, a data receiving buffer, a filling result array and an integrity check module. In the data receiving module, a fixed port can be monitored through multi-threading, so that data packets from the proxy system can be received at any time. The message expansion module and the iterative compression module are the main calculation modules. The data receiving buffer and the filling result array are the main memory modules of the system. In the specific implementation, the SM3 compression algorithm can be preset in the iterative compression model, and then each data packet can be compressed and calculated through the iterative compression module in the encryption system to obtain the compression value corresponding to each data packet.

[0068] Step 104: Obtain a target file compression value according to the data packet compression values ​​of all data packets.

[0069] After compressing each data packet to obtain the data packet compression value, the compression value of the entire target file can be calculated by summarizing the compression values ​​of all data packets, thereby evaluating the integrity of the large file.

[0070] In a specific implementation, the compression value of the target file can be obtained by summarizing the compression values ​​of all data packets, and the summarization method can be simple concatenation or more complex calculation.

[0071] Step 105: Obtain a historical file compression value of the target file, and use the historical file compression value to verify the target file compression value to obtain a verification result of the target file.

[0072] The historical file compression value refers to the compression value of the target file at different time points or different versions. The compression value is also called a hash value. The hash value is a fixed-length string calculated by a certain algorithm and is used to represent the content of the file.

[0073] The historical file compression value of the target file can be obtained by querying the historical hash value or historical compression value of the file from the database or storage system according to the unique identifier of the large file. The historical hash value or historical compression value may be stored in a log, database or other storage medium. The current compression value of the target file is compared with the historical compression value. If the two are consistent, it means that the file has not been tampered with; if they are inconsistent, it means that the file may have been tampered with; by comparing the consistency of the compression value, the integrity check of the file is completed.

[0074] In some implementations of this embodiment, if the historical compression value is consistent with the current compression value, the verification result is "passed"; otherwise, the verification result is "failed"; the verification result can be a Boolean value (such as True or False, true or false) or a status code (such as 0 for passed, 1 for failed), and the verification result is returned to the caller or system for further processing or display.

[0075] The embodiment of the present application obtains the target file to be verified, performs data grouping on the target file to obtain at least one data packet, performs compression calculation on at least one data packet respectively to obtain a data packet compression value, obtains the target file compression value based on the data packet compression values ​​of all data packets, obtains the historical file compression value of the target file, and uses the historical file compression value to verify the target file compression value to obtain the verification result of the target file. By grouping the files, and performing compression calculation on the grouped multiple data packets, the compression value of the file is obtained according to the data packet compression value, and the historical compression value verification method is used to improve the speed of file verification.

[0076] Reference Figure 4 , shows a flowchart of another method for file verification provided by an embodiment of the present invention, which may specifically include the following steps:

[0077] Step 401: Create multiple file reading threads for reading target files;

[0078] Step 402: reading a target file through multiple file reading threads;

[0079] Among them, a thread is the smallest unit that the operating system can use to schedule operations. It is contained in a process and is the actual operating unit in the process. Multiple threads can execute different tasks at the same time, thereby improving the concurrency and response speed of the program.

[0080] In a specific implementation, before starting to transfer a large file, multiple file reading threads can be created and initialized in the proxy system, and data can be read from the same file stream in the file system based on the unique identifier of the file and the unique identifier of the file reading thread. Creating multiple file reading threads to read the target file can improve the reading efficiency, especially when processing large files. After the proxy reads the data from the collection, it determines whether the encryption system buffer meets the data sending conditions and sends the data. At the same time, the encryption system needs to initialize the receiving data buffer, set the buffer queue S, and the receiving data buffer occupies a memory space of M. The encryption system can also create and initialize multiple data receiving threads T1, T2 to listen to the buffer data.

[0081] Step 403: Group the target file into data packets to obtain at least one data packet.

[0082] Step 404: performing compression calculation on the at least one data packet respectively to obtain a data packet compression value;

[0083] Step 405: Obtain a target file compression value according to the data packet compression values ​​of all data packets;

[0084] Step 406: Obtain a historical file compression value of the target file, and use the historical file compression value to verify the target file compression value to obtain a verification result of the target file.

[0085] The execution process of step 403 to step 406 is similar to that of step 102 to step 105, and will not be repeated here.

[0086] In some embodiments of the present application, the following sub-steps are included before step 403:

[0087] Step 11: Obtain the data volume and file identifier of the target file;

[0088] Step 12: Determine the number of groups of the target file according to the first preset length and the data volume of the target file;

[0089] Among them, the data volume refers to the total size of the data contained in the target file. It is usually measured in bytes. The size of the data volume determines the time and resource consumption of file processing. In multi-threaded programming, the size of the data volume also affects the division of threads and task allocation. The first preset length refers to the size of the data packet processed by each thread or group in multi-threaded file processing, which is usually measured in bytes. The first preset length determines the amount of data processed by each thread or group. A reasonable first preset length can balance the load of the thread and improve processing efficiency. The number of groups refers to the number of data blocks into which the target file is divided, and each data block is processed by a thread or a group of threads.

[0090] In the specific implementation of this embodiment, the first preset length can be adjusted according to system resources, file size and number of threads. For example, if the file is very large, a larger first preset length can be set; if the system resources are limited, a smaller first preset length can be set.

[0091] Before starting the file transfer, assuming that the data volume of the target file is n bits, the file identifier of the target file is "ABCD", and the first preset length is 512 bits, then according to the first preset length and the data volume of the target file, the number of packets corresponding to the target file to be processed can be determined to be

[0092] In some embodiments of the present application, step 403 includes the following sub-steps:

[0093] Sub-step 21: Read the target file according to the file identifier and the first preset length to obtain the at least one data packet of the grouped quantity.

[0094] In some implementations of this embodiment, data can be read from the same file stream by different file reading threads based on the file identifier "ABCD" and its corresponding thread identifier, and 512 bits of data are read each time to form a data packet. The set of all data packets is in the form of [Did(data1), Did(data2), ... Did(dataj), ..., Did(datax-1), Did(datax)], where x represents the total number of data packets, that is, the amount of packet data corresponding to the file to be processed. After the proxy system reads the data from the set, it determines whether the data receiving buffer of the encryption system meets the data sending conditions. If so, the data is sent.

[0095] In some embodiments of the present application, the at least one data packet includes a tail data packet, and step 403 may further include the following sub-steps:

[0096] Sub-step 22: determining the data length of the last data packet, and determining the relationship between the data length of the last data packet and a first preset length;

[0097] Sub-step 23: When it is determined that the data length of the last data packet is less than the first preset length, the last data packet is expanded according to a preset rule to obtain an expanded last data packet of the first preset length.

[0098] The final packet refers to the last packet used to mark the end of data transmission during the data transmission process. In network communications, data is usually divided into multiple packets for transmission. The final packet is used to inform the receiver that the data transmission has been completed to ensure that the receiver can correctly reassemble and process the data. The final packet usually contains special identification information, such as an end flag, checksum, etc., to ensure the integrity and correctness of data transmission.

[0099] In some implementations of this embodiment, when the file has been read to the end, assuming that the length of the remaining data packet is nl, that is, the data length of the last data packet is nl, it is necessary to determine the size relationship between the data length of the last data packet and the first preset length. When it is determined that the data length of the last data packet is less than the first preset length, the last data packet is expanded according to the preset rules to obtain the expanded last data packet of the first preset length.

[0100] If nl=512, it means that the length of the last data packet after the grouping is exactly the first preset length. At this time, the data packet can be sent to form a new 512-bit tail data packet. The format of the tail data packet can be the first bit is 1, followed by 447 bits of 0, plus the binary representation of 64 bits of n, and the newly generated 512-bit data packet is sent.

[0101] If nl < 512, it is determined that the data length of the last data packet is less than the first preset length, and the data with the current data packet structure of length nl is filled. The filling method can be to add a binary 1 after the nl-bit data, a binary number of all 0s with a length of 512 - nl - 65, and then add the 64-bit binary representation of n to form a 512-bit data packet for sending.

[0102] In the above manner, special identification information is set in the last data packet to ensure the integrity and correctness of data transmission.

[0103] In some embodiments of the present application, before step 404, it may include:

[0104] Sub-step 31: Uniformly divide the at least one data packet according to the second preset length to obtain a first target number of target words;

[0105] Sub-step 32: Perform a translation calculation on the first specified target word among the first target number of target words to obtain a translated target word;

[0106] Sub-step 33: Perform an exclusive OR operation on the translated target word and the second specified target word among the first target number of target words to obtain a second target number of extended words;

[0107] Sub-step 34: Determine the second target number of extended words as the at least one data packet.

[0108] In some implementation manners of this embodiment, the encryption system includes a message expansion module. The message expansion module processes the received data packet Did(dataj) and implements data filling. The second preset length can be but is not limited to 32 bits, and the second preset length can be adjusted according to actual situations.

[0109] Specifically, the message expansion module can uniformly divide the received 512-bit data packet according to the second preset length of 32, and uniformly divide the 512-bit data packet Did(dataj) into 16 words {W0, W1,..., W 15}, and then expand these 16 words into data with a length of 132 words, and fill these 16 words in the first 16 bits of {W i}. During the process of data expansion by the message expansion module, multiple threads can also be added for parallel processing to implement a fast expansion method and prevent affecting the processing speed of subsequent iterative compression algorithms.

[0110] Specifically, when 15 < i < 68, W i satisfies:

[0111]

[0112] Among them, P1 is the permutation function in the SM3 algorithm. The P1 function is an important permutation function in the SM3 algorithm, which is used to process data during the message expansion process. The P1 function performs complex transformations on the input data through a series of logical operations, such as exclusive OR, circular left shift, bitwise AND and bitwise OR, constant addition, etc., so that the output data has higher randomness and unpredictability. W i represents the i-th word, is the exclusive OR operator, <<< is the symbol for circular left shift, and W i-3 <<<15 means circularly shifting the data in W i-3 to the left by 15 bits, and W i-13 <<<7 means circularly shifting the data in W i-13 to the left by 7 bits. Among them, circular left shift is a bit operation used to shift binary data to the left by a specified number of bits and reinsert the shifted bits to the right side of the data to form a circular effect. In the embodiments of the present application, by changing the bit order of the data, the randomness and unpredictability of the data are increased, and the security and efficiency of data processing are improved.

[0113] When i satisfies 67 < i < 132, W i satisfies:

[0114]

[0115] Through the above formula, all data of W 16 -W 131 can be obtained, and the data is stored in the array Did[j] to be compressed, where id refers to the file identifier and j represents the number of the data packet. id-j {W0, W1,..., W 130 , W 131}, and the data is stored in the array Did[j] to be compressed, where id refers to the file identifier and j represents the number of the data packet.

[0116] In some embodiments of the present application, step 404 may include the following sub-steps:

[0117] In the case of determining that at least one data packet is non-empty, perform compression calculation on the at least one data packet for a preset number of times to obtain the data packet compression value.

[0118] In the specific implementation manner of this embodiment, the expanded data packet can be iteratively compressed through the iterative compression module of the encryption system. Specifically, the iterative compression algorithm in the SM3 algorithm can be preset in the iterative compression module. The compression calculation of the data packet is realized through the iterative compression algorithm in the SM3 algorithm. First, set the initial value V iThe SM3 iterative compression algorithm CF is used to perform 64 rounds of compression on the 132-word padding data expanded in the above step to generate a new V i+1 , the formula is:

[0119] V i+1 =CF(V i ,W i )

[0120] Set the initial value to IV. The first calculation needs to determine whether the data packet is empty, that is, check whether there is data in Did[0]. If not, wait. If the data packet is not empty, use the above formula to calculate the value of V0 and perform iterative calculation. Generate V through 64 rounds of calculation 63 , and store the result in IV, and query D id [1], and so on. id [datax] After data processing, the final generated IV is returned to the integrity verification module. The specific implementation of the embodiment of the present application divides the data expansion module and the iterative compression module in the encryption system into different modules, thereby realizing the decoupling of the compression iteration algorithm and the data expansion operation, thereby realizing the algorithm parallel calculation method, realizing multi-module collaborative calculation, and increasing the data processing speed. In the data expansion module, multiple threads are processed to achieve rapid expansion to prevent affecting the processing speed of the iterative compression algorithm of the subsequent iterative compression model. Although the iterative compression algorithm needs to rely on the calculation results of the preceding position during the calculation process and is difficult to process in parallel, when the data expansion module speed is sufficient, the iterative compression algorithm speed is faster, thereby improving the overall calculation speed.

[0121] In some embodiments of the present application, before step 404, the following may also be included:

[0122] Step 41: Create a data buffer, and store the at least one data packet in the data buffer.

[0123] Step 42: Create multiple buffer reading threads for reading data from the data buffer.

[0124] Step 43: Read the data buffer through a plurality of the buffer reading threads to obtain at least one data packet.

[0125] Among them, the data buffer is an area in a computer system for temporarily storing data. It plays a buffering role in the data transmission and processing process to ensure that data can be efficiently and stably transmitted and processed. The main functions of the data buffer include smoothing the data transmission rate, reducing delays, processing burst data and improving system stability. According to different application scenarios, the data buffer can be divided into types such as input buffer, output buffer, memory buffer and disk buffer. The data buffer can be implemented by means of arrays, linked lists, queues and ring buffers, and is widely used in the fields of network transmission, file reading and writing, audio and video processing, database operations, etc. The embodiment of the present application can solve the problem of data transmission rate mismatch by setting the data buffer, ensuring that data can be smoothly transmitted and processed. In some implementation methods of the present embodiment, in order to prevent the memory overflow problem caused by the file being too large, a dynamic expansion method can be performed to prevent the overflow problem that may exist caused by multi-threaded data transmission, and an expansion upper limit is set, which can improve the data transmission efficiency and the stability of system operation in this way.

[0126] In some implementations of this embodiment, data can be received through the data receiving module of the encryption system. The encryption system can create a data buffer in advance. In addition, multiple file processing threads can be created and initialized to process the data sent by the proxy system. Multiple file processing threads put the received data packets into the data buffer first. If the data buffer is full, the receiving of data is suspended until there is a certain amount of free space in the data buffer. For example, when there is one-third or one-quarter of the free area in the data buffer, the proxy system is notified again to continue sending data. The message expansion module in the encryption system can read data blocks from the data buffer in parallel by creating and initializing multiple buffer reading threads.

[0127] like Figure 5As shown, it is the autonomous process of the proxy system in some embodiments of the present application. After the proxy system is started, it will automatically read the configured files that need to implement integrity verification, and actively call the data sending module to implement file integrity verification. Before sending data, multiple threads are created in advance, and the multiple threads created in advance are initialized at the beginning of sending, and the file information is read through multiple threads. In the process of sending data, the data of the large file will be divided, for example, 512 bits are sent as a data packet. The total number of data blocks n is calculated according to the data length L, and the number and the corresponding file id are marked in the data block, and then parallel data transmission is realized. In the process of sending, it is also necessary to monitor the data receiving buffer state of the encryption system in real time. If the buffer is full, it can be determined that the other party cannot accept it. At this time, it is necessary to suspend data generation, and send data to the data receiving buffer of the encryption system when the buffer meets the conditions for receiving data. The sending process can be specifically to obtain the number of the data packet first, and then read the data packet corresponding to the number.

[0128] like Figure 6 As shown, the encryption process of the encryption system in some embodiments of the present application, the encryption system also needs to create and initialize multiple threads, which are used to receive data and process data respectively. All received data will be stored in the data receiving buffer first, and then taken out by the message expansion module, and the message expansion processing will be performed in parallel. Finally, the iterative compression module returns the target file compression value, and completes the file integrity check by comparing the consistency relationship between the target file compression value and the historical data of the target file.

[0129] Taking Java language as an example, the data receiving buffer of the encryption system is implemented using the BlockingQueue blocking queue. When the queue is full, it can block the thread of the data receiving module by itself, thereby stopping data reception. After the queue is blocked, the thread that sends data through the proxy will automatically pause the thread through the sleep() method of the Thread class if the sending fails.

[0130] Taking the integrity check of a 1GB large file as an example, a 1GB large file is specifically 8589934592 bits, with a total of 16777216 512-bit formal data blocks. In some cases, it may be necessary to add additional 64 bits of information to the large file, such as for checksum, identification information, etc.

[0131] Step 1: The proxy system calculates the number of data blocks to be sent, which is (8589934592+64) / 512, and modulo the result upwards, which is 16777217 512-bit data blocks.

[0132] Step 2: Determine that the encryption system buffer is available and start sending data

[0133] Step 3: The encryption system receives the data and stores it in the buffer. The encryption system sets the buffer size in the memory to 20M, so the cache queue can accept a total of 327,680 data blocks.

[0134] Step 4: The expansion module generates a total of 132 words, i.e., 4224 bits of data, by performing an expansion algorithm on a 512-bit data block, and stores it in an array. The array length is set to 4224*1000 bits, i.e., actual expansion data of 1000 data packets is provided. After completing 397180 data processing, the entire array is cleared and reassigned. It can be seen that the space occupied by the array in the encryption system memory is 24M. Therefore, for the processing of a large file of 1G, the memory required in the embodiment of the present application is only 44M, which reduces the memory usage and improves the performance of large file verification compared to receiving the complete data of the large file and then performing the integrity check of the file.

[0135] Step 5: The iterative compression algorithm implements iterative data compression and sends the final value to the next rotation process. Every 64 rotations, the next set of 132 words of data is taken to finally generate a file compression value.

[0136] Step 6: Query the file history hash value based on the file unique identifier id, compare it to complete the integrity check, and return the check result to the value proxy module.

[0137] The embodiment of the present application obtains the target file to be verified, performs data grouping on the target file to obtain at least one data packet, performs compression calculation on at least one data packet respectively to obtain a data packet compression value, obtains the target file compression value based on the data packet compression values ​​of all data packets, obtains the historical file compression value of the target file, and verifies the target file compression value using the historical file compression value to obtain the verification result of the target file. By grouping the files, and performing compression calculation on the multiple data packets after grouping, the compression value of the file is obtained according to the data packet compression value, and the historical compression value verification method is adopted to improve the speed of file verification; by concurrently processing data through multiple file exclusive threads and multiple cache reading threads, efficient data sending and receiving capabilities are achieved, the system's processing capabilities for large files are increased, and the performance loss in file transmission is reduced. The embodiment of the present application is suitable for the integrity verification of elastic files, disks and other data in the cloud resource pool, and is particularly suitable for file systems with large data volumes and the need for regular integrity verification.

[0138] It should be noted that, for the sake of simplicity, the method embodiments are described as a series of action combinations, but those skilled in the art should be aware that the embodiments of the present invention are not limited by the order of the actions described, because according to the embodiments of the present invention, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present invention.

[0139] Reference Figure 7 , shows a schematic diagram of the structure of a file verification device provided by an embodiment of the present invention, which may specifically include the following modules:

[0140] The file acquisition module 701 is used to acquire the target file to be verified;

[0141] A data grouping module 702 is used to group the target file into data packets to obtain at least one data packet;

[0142] A compression value calculation module 703, configured to perform compression calculation on the at least one data packet to obtain a data packet compression value;

[0143] A compression value determination module 704, used to obtain a target file compression value according to the data packet compression values ​​of all data packets;

[0144] The result verification module 705 is used to obtain the historical file compression value of the target file, and use the historical file compression value to verify the target file compression value to obtain the verification result of the target file.

[0145] In an optional embodiment of the present application, the device further includes:

[0146] A first acquisition submodule is used to acquire the data volume and file identifier of the target file;

[0147] A first grouping submodule, used to determine the number of groups of the target file according to a first preset length and the data volume of the target file;

[0148] The data grouping module comprises:

[0149] The second grouping submodule is used to read the target file according to the file identifier and the first preset length to obtain the at least one data packet of the grouping quantity.

[0150] In an optional embodiment of the present application, the at least one data packet includes a tail data packet, and the data grouping module 702 includes:

[0151] A first determining submodule, used to determine the data length of the last data packet, and determine the relationship between the data length of the last data packet and a first preset length;

[0152] The first expansion submodule is used to expand the tail data packet according to a preset rule to obtain an expanded tail data packet of the first preset length when it is determined that the data length of the tail data packet is less than the first preset length.

[0153] The device also includes:

[0154] A first segmentation module, configured to evenly segment the at least one data packet according to a second preset length to obtain a first target number of target words;

[0155] A first translation module, configured to perform a translation calculation on a first designated target word among the first target number of target words to obtain a translation target word;

[0156] A second expansion module is used for performing an XOR operation on the translation target word and a second designated target word in the first target number of target words to obtain an expansion word of a second target number;

[0157] The first determining module is used to determine the second target number of extended words as the at least one data packet.

[0158] In an optional embodiment of the present application, the compression value calculation module 703 includes:

[0159] The first compression submodule is used to perform a preset number of compression calculations on the at least one data packet to obtain the data packet compression value when it is determined that the at least one data packet is not empty.

[0160] In an optional embodiment of the present application, the file acquisition module 701 includes:

[0161] A first creation submodule is used to create a plurality of file reading threads for reading the target file;

[0162] The first reading submodule is used to read the target file through the multiple file reading threads.

[0163] In an optional embodiment of the present application, the device comprises:

[0164] A cache creation module, used for creating a data buffer and storing the at least one data packet in the data buffer;

[0165] A thread creation module, used for creating a plurality of cache reading threads for reading data from a data buffer;

[0166] The thread reading module is used to read the data buffer through a plurality of cache reading threads to obtain at least one data packet.

[0167] An embodiment of the present invention further provides an electronic device, which may include a processor, a memory, and a computer program stored in the memory and capable of running on the processor, and the computer program implements the above file verification method when executed by the processor.

[0168] An embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the above file verification method is implemented.

[0169] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0170] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0171] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0172] Those skilled in the art will appreciate that the embodiments of the present invention may be provided as methods, devices, or computer program products. Therefore, the embodiments of the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the embodiments of the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0173] The embodiments of the present invention are described with reference to the flowcharts and / or block diagrams of the methods, terminal devices (systems), and computer program products according to the embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0174] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0175] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable terminal device to implement the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0176] Although the preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present invention.

[0177] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "including one..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the above elements.

[0178] The method and device for file verification, electronic device, and storage medium provided above are introduced in detail. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea; at the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A method for file verification, characterized in that: The method comprises: Get the target file to be verified; Grouping the target file into data packets to obtain at least one data packet; Performing compression calculation on the at least one data packet respectively to obtain a data packet compression value; According to the data packet compression values ​​of all data packets, the target file compression value is obtained; A historical file compression value of the target file is obtained, and the historical file compression value is used to verify the target file compression value to obtain a verification result of the target file.

2. The method according to claim 1, characterized in that: Before the target file is grouped to obtain at least one data packet, the method includes: Obtaining the data volume and file identifier of the target file; Determining the number of groups of the target file according to a first preset length and the data volume of the target file; The step of grouping the target file to obtain at least one data packet includes: The target file is read according to the file identifier and the first preset length to obtain the at least one data packet of the grouped quantity.

3. The method according to claim 1, characterized in that The at least one data packet includes a tail data packet, and the target file is grouped to obtain at least one data packet, including: Determining the data length of the last data packet, and determining the relationship between the data length of the last data packet and a first preset length; When it is determined that the data length of the tail data packet is less than the first preset length, the tail data packet is expanded according to a preset rule to obtain an expanded tail data packet of the first preset length.

4. The method according to claim 1, characterized in that: Before respectively performing compression calculation on the at least one data packet to obtain the data packet compression value, the method includes: Evenly dividing the at least one data packet according to a second preset length to obtain a first target number of target words; Performing a translation calculation on a first designated target word among the first target number of target words to obtain a translation target word; Performing an XOR operation on the translation target word and a second designated target word among the first target number of target words to obtain an extended word of a second target number; The second target number of extension words is determined as the at least one data packet.

5. The method according to claim 1, characterized in that The performing compression calculation on the at least one data packet respectively to obtain a data packet compression value comprises: When it is determined that the at least one data packet is not empty, compression calculation is performed on the at least one data packet for a preset number of times to obtain the data packet compression value.

6. The method according to any one of claims 1 to 5, characterized in that: The step of obtaining the target file to be verified includes: Creating a plurality of file reading threads for reading the target file; The target file is read by a plurality of the file reading threads.

7. The method according to claim 5, characterized in that Before respectively performing compression calculation on the at least one data packet to obtain the data packet compression value, the method includes: Creating a data buffer, and storing the at least one data packet in the data buffer; Create multiple cache reading threads for reading data from the data buffer; The data buffer is read by a plurality of cache reading threads to obtain at least one data packet.

8. A file verification device, characterized in that: The device comprises: A file acquisition module, used to acquire the target file to be verified; A data grouping module, used for grouping the target file to obtain at least one data packet; A compression calculation module, used to perform compression calculation on the at least one data packet to obtain a data packet compression value; A compression determination module, used for obtaining a target file compression value according to the data packet compression values ​​of all data packets; The result verification module is used to obtain the historical file compression value of the target file, and use the historical file compression value to verify the target file compression value to obtain the verification result of the target file.

9. An electronic device, characterized in that: The invention comprises a processor, a memory and a computer program stored in the memory and capable of running on the processor, wherein the computer program implements the file verification method according to any one of claims 1 to 7 when executed by the processor.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the file verification method according to any one of claims 1 to 7 is implemented.

Images