File content identification method and device based on text partitioning, equipment and medium

Abstract

The invention discloses a file content recognition method and device based on text partitioning, equipment and a medium, and relates to the technical field of computers. Comprising the following steps: respectively performing blocking operation on a first target file and a second target file to obtain a plurality of file blocks respectively corresponding to the first target file and the second target file, and respectively calculating block fingerprint information corresponding to each file block; generating first file fingerprint information and second file fingerprint information respectively corresponding to the first target file and the second target file based on the plurality of pieces of block fingerprint information, and determining a file similarity index of the first target file and the second target file based on the first file fingerprint information and the second file fingerprint information; and determining target file contents based on the file similarity index and the file contents of the first target file and the second target file. Therefore, the whole file content or the partial file content can be identified based on the similarity of the calculated text.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and particularly to a method, device, equipment and medium for identifying file content based on text chunking. Background Art

[0002] When enterprises and institutions process a large number of files and data, they face increasingly high requirements for security and confidentiality, and need to identify confidential files efficiently and accurately. Traditional identification methods mainly rely on manual review, which is not only inefficient but also prone to errors. With the continuous development of text similarity calculation technology, people have begun to use this technology to automatically identify confidential files. Text similarity calculation technology is a natural language processing technology that can calculate the similarity between two pieces of text, thereby helping us process information more effectively. Currently, most common text similarity calculation technologies are based on pre-trained language models using deep learning methods, mapping words or sentences to a high-dimensional semantic space, and evaluating text similarity by calculating the cosine similarity between vectors. This method requires a large amount of pre-training data, has high hardware resource requirements, and is time-consuming for calculation and training. Language models in different fields are not universal. There are also some methods that first segment and weight the text, and then use the locality-sensitive hashing algorithm to convert the segmented words into binary fingerprints, merge and reduce the dimensions to generate fingerprints of the text, and evaluate text similarity by comparing the Hamming distance between the two text fingerprints. This method has a low similarity recognition rate for short texts and is inaccurate for calculating the similarity of texts with large length differences.

[0003] As can be seen from the above, how to identify the overall file content or part of the file content based on the calculation of text similarity is an urgent problem to be solved at present. Summary of the Invention

[0004] In view of this, the purpose of the present invention is to provide a method, device, equipment and medium for identifying file content based on text chunking, which can identify the overall file content or part of the file content based on the calculation of text similarity. The specific scheme is as follows:

[0005] In a first aspect, the present application provides a method for identifying file content based on text chunking, including:

[0006] Performing chunking operations on a first target file and a second target file respectively to obtain a number of file chunks corresponding to the first target file and the second target file respectively, and calculating the chunk fingerprint information corresponding to each file chunk;

[0007] Generate first file fingerprint information and second file fingerprint information corresponding to the first target file and the second target file respectively based on the several piece fingerprint information, and determine a file similarity index of the first target file and the second target file based on the first file fingerprint information and the second file fingerprint information; the file similarity index includes file similarity, file similar byte count, piece identical rate, and piece similar rate;

[0008] Determine target file content based on the file similarity index and the file content of the first target file and the second target file; wherein, the target file content includes identical file content and similar file content.

[0009] Optionally, the calculating piece fingerprint information corresponding to each file piece respectively includes:

[0010] Determine piece length data of each file piece, and generate a first piece hash value by using a preset digest algorithm;

[0011] Perform word segmentation on the file content corresponding to the file piece, and generate a second piece hash value by using a preset locality-sensitive hashing algorithm based on the word segmentation result;

[0012] Generate the piece fingerprint information based on the piece length data, the first piece hash value, and the second piece hash value.

[0013] Optionally, the determining the file similarity index of the first target file and the second target file based on the first file fingerprint information and the second file fingerprint information includes:

[0014] Perform similarity calculation based on the first file fingerprint information and the second file fingerprint information to obtain the file similarity and the file similar byte count;

[0015] Determine identical file pieces in the first target file and the second target file based on the first piece hash value, and calculate the piece identical rate based on the identical file pieces;

[0016] Calculate the Hamming distance of other file pieces based on the second piece hash value, so as to determine similar file pieces based on the Hamming distance and a preset Hamming distance similarity mapping table, and calculate the piece similar rate based on the similar file pieces;

[0017] Generate the file similarity index of the first target file and the second target file based on the file similarity, the file similar byte count, the piece identical rate, and the piece similar rate.

[0018] Optionally, after generating the first file fingerprint information and the second file fingerprint information corresponding to the first target file and the second target file respectively based on the plurality of block fingerprint information, the method further includes:

[0019] Retaining the block index and the block file content of each file block;

[0020] Correspondingly, determining the target file content based on the file similarity metric and the file contents of the first target file and the second target file includes:

[0021] Determining the block indexes corresponding to the identical file blocks and the similar file blocks based on the file similarity metric;

[0022] Restoring the block file contents corresponding to the identical file blocks and the similar file blocks based on the block indexes to obtain the target file content.

[0023] Optionally, the performing block operations on the first target file and the second target file respectively includes:

[0024] Performing block operations on the first target file and the second target file respectively based on a preset block strategy;

[0025] Wherein, the preset block strategy includes a dynamic block strategy based on file content and a block strategy based on natural paragraphs of the file.

[0026] Optionally, for the dynamic block strategy based on file content, the performing block operations on the first target file and the second target file respectively based on the preset block strategy includes:

[0027] Configuring file block strategies with different block sizes, and selecting one or two file block strategies from the file block strategies as the target file block strategy according to the preset computing resource conditions;

[0028] Determining block boundaries based on the target block strategy, and performing block operations on the first target file and the second target file based on the block boundaries.

[0029] Optionally, for the block strategy based on natural paragraphs of the file, the performing block operations on the first target file and the second target file respectively based on the preset block strategy includes:

[0030] Configuring a minimum segment length, and adding segment markers to the file contents of the first target file and the second target file based on the minimum segment length;

[0031] Performing block operations on the first target file and the second target file based on the segment markers.

[0032] In a second aspect, the present application provides a file content recognition device based on text chunking, including:

[0033] A file chunking module, configured to perform chunking operations on a first target file and a second target file respectively, so as to obtain a number of file chunks corresponding to the first target file and the second target file respectively, and calculate chunk fingerprint information corresponding to each file chunk;

[0034] A file similarity index generation module, configured to generate first file fingerprint information and second file fingerprint information corresponding to the first target file and the second target file respectively based on the number of chunk fingerprint information, and determine a file similarity index of the first target file and the second target file based on the first file fingerprint information and the second file fingerprint information; the file similarity index includes file similarity, file similar byte count, chunk same rate, and chunk similarity rate;

[0035] A file content recognition module, configured to determine target file content based on the file similarity index and the file content of the first target file and the second target file; wherein, the target file content includes identical file content and similar file content.

[0036] In a third aspect, the present application provides an electronic device, including:

[0037] A memory, configured to store a computer program;

[0038] A processor, configured to execute the computer program to implement the foregoing file content recognition method based on text chunking.

[0039] In a fourth aspect, the present application provides a computer-readable storage medium, configured to store a computer program, wherein the computer program, when executed by a processor, implements the foregoing file content recognition method based on text chunking.

[0040] The present application provides a method for identifying file content based on text chunking. First, perform chunking operations on the first target file and the second target file respectively to obtain a number of file chunks corresponding to the first target file and the second target file respectively, and calculate the chunk fingerprint information corresponding to each file chunk; then generate the first file fingerprint information and the second file fingerprint information corresponding to the first target file and the second target file respectively based on the number of chunk fingerprint information, and determine the file similarity index of the first target file and the second target file based on the first file fingerprint information and the second file fingerprint information; the file similarity index includes file similarity, file similar byte count, chunk same rate, and chunk similarity rate; finally, determine the target file content based on the file similarity index and the file content of the first target file and the second target file; wherein, the target file content includes identical file content and similar file content.

[0041] As can be seen from the above, the present application chunks the file and generates the fingerprint information of the file chunks. By calculating the similarity between each chunk of the two files through the fingerprint information of the file chunks, and finally merging the chunk similarity through a specified algorithm to generate the similarity of the file, the accuracy of calculating the similarity of texts with large length differences can be improved. Through the file and chunk similarity, the identical content (or paragraphs) and similar content (or paragraphs) of the file can be found, and the relevance of the file can be expressed by the similarity, and the scenarios where the overall file content or partial file content is similar can be identified. Thus, it is possible to identify the overall file content or partial file content based on the calculated similarity of the text. Brief Description of the Drawings

[0042] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the drawings in the following description are only the embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on the provided drawings.

[0043] Figure 1 It is a flowchart of a method for identifying file content based on text chunking disclosed in the present application;

[0044] Figure 2 It is a flowchart of a specific method for identifying file content based on text chunking disclosed in the present application;

[0045] Figure 3 It is a flowchart of generating file fingerprints disclosed in the present application;

[0046] Figure 4Schematic diagram of file fingerprint generation based on different chunking strategies disclosed in this application;

[0047] Figure 5 Schematic diagram of hash value comparison disclosed in this application;

[0048] Figure 6 Schematic diagram of simhash value comparison disclosed in this application;

[0049] Figure 7 Schematic diagram of a file content recognition device based on text chunking disclosed in this application;

[0050] Figure 8 Structural diagram of an electronic device disclosed in this application. Detailed implementation manners

[0051] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0052] Enterprises and institutions face increasingly high requirements for security and confidentiality when processing a large number of files and data, and need to identify classified files efficiently and accurately. Traditional identification methods mainly rely on manual review, which is not only inefficient but also error-prone. With the continuous development of text similarity calculation technology, people have begun to use this technology to automatically identify classified files. Text similarity calculation technology is a natural language processing technology that can calculate the similarity between two pieces of text, thus helping us process information more effectively. Currently, most common text similarity calculation technologies are based on pre-trained language models using deep learning methods, mapping words or sentences to a high-dimensional semantic space, and evaluating text similarity by calculating the cosine similarity between vectors. This method requires a large amount of pre-training data, high hardware resource requirements, and long calculation and training time, and the language models in different fields cannot be used interchangeably. There are also some methods that first tokenize and weight the text, and then use the locality-sensitive hashing algorithm to convert the tokens into binary fingerprints, merge and reduce the dimensions to generate the fingerprint of the text, and evaluate text similarity by comparing the Hamming distance between the two text fingerprints. This method has a low recognition rate for the similarity of short texts and is inaccurate for calculating the similarity of texts with large length differences. Therefore, this application provides a file content recognition solution based on text chunking, which can identify the overall file content or part of the file content based on calculating the similarity of the text.

[0053] See Figure 1As shown in the figure, an embodiment of the present application discloses a method for identifying file content based on text chunking, including:

[0054] Step S11: Perform chunking operations on the first target file and the second target file respectively to obtain a number of file chunks corresponding to the first target file and the second target file respectively, and calculate the chunk fingerprint information corresponding to each file chunk.

[0055] In this embodiment, chunking operations are performed on the first target file and the second target file respectively based on a preset chunking strategy; wherein, the preset chunking strategy includes a dynamic chunking strategy based on file content and a chunking strategy based on natural paragraphs of the file. Specifically, for the dynamic chunking strategy based on file content, the performing of chunking operations on the first target file and the second target file respectively based on the preset chunking strategy may include: configuring content-based dynamic chunking algorithms (CDC, content-defined chunking) with different chunk sizes, and selecting one or two dynamic chunking algorithms from the dynamic chunking algorithms as target dynamic chunking algorithms according to the pre-set computing resource conditions. That is, selecting two dynamic chunking algorithms as target dynamic chunking algorithms can generate two types of file chunks, which can further improve the accuracy of file similarity calculation, but also has a higher demand for computing resources. Therefore, the specific target dynamic chunking algorithm can be determined according to the actual application situation. Further, since the CDC chunking processes according to the byte stream and does not consider the boundaries of strings, when the chunks are restored to text, garbled characters may appear at both the beginning and the end. To solve the above problem, boundary adjustment should be performed during chunking to ensure that the boundaries of the chunks contain complete characters. For example, in a specific embodiment, strings encoded in UTF-8, Chinese characters, and full-width characters occupy 3 bytes, while English, numbers, and half-width characters occupy 1 byte. After determining the boundary for each CDC chunk, a 3-bit window is used to attempt UTF-8 encoding of the last 3 bytes of the chunk. If the attempt is successful, it means that the window contains a complete character and the boundary can be returned; if the attempt fails, it means that the window contains an incomplete character, and the window position needs to be adjusted and the attempt is repeated until the attempt is successful to determine the boundary containing the complete character.

[0056] Further, for the chunking strategy based on the natural paragraphs of the file, the chunking operation performed on the first target file and the second target file respectively based on the preset chunking strategy may include: configuring the minimum segment length, and adding segment markers to the file contents of the first target file and the second target file based on the minimum segment length; performing a chunking operation on the first target file and the second target file based on the segment markers. That is, by using the natural paragraphs of the file for chunking, it can be ensured that the chunking of the same paragraph in different files is consistent, and the accuracy of file similarity calculation is higher in the scenario of whole-paragraph copying. It should be noted that this chunking method has relatively high requirements for the processing of the original file content. Specifically, it is required that the extracted original file content has a unified segment identifier, such as "\r\n", ".\r\n", "\n", etc. A unified segment identifier should be added when extracting the file content for unified processing during subsequent file chunking. Secondly, too small segments should be avoided. Usually, titles, chart descriptions, etc. will be processed as segments when extracting the file content, which will lead to redundancy in the file fingerprint library and low efficiency of file similarity calculation. At the same time, because the chunks are too small, the accuracy of simhash (Locality-Sensitive Hashing algorithm) comparison is also relatively low. Therefore, it is necessary to define a minimum length for chunking and merge too small segments into a large chunk. Finally, the situation where chunking cannot be performed should be prevented. For example, the original file may not be recognizable or there are no natural paragraphs in the original file, and the accuracy of file similarity calculation will be even lower in this case. Double-chunk fingerprints can be made together with the CDC chunking. When calculating the file similarity, two types of fingerprints are calculated simultaneously, and the result with a higher file similarity is output.

[0057] In this embodiment, the chunk fingerprint information corresponding to each file chunk is generated; the chunk fingerprint information includes the chunk size, the traditional hash value, and the simhash value. Specifically, the generation of the chunk fingerprint information corresponding to each file chunk respectively may include: determining the chunk length data of each file chunk, and generating the traditional hash value by using a preset digest algorithm; performing a word segmentation operation on the file content corresponding to the file chunk, and generating the simhash value based on the word segmentation result by using a preset locality-sensitive hashing algorithm; generating the chunk fingerprint information based on the chunk length data, the traditional hash value, and the simhash value. Among them, the specific type of the preset digest algorithm can be determined according to the actual application situation. For example, in a specific implementation, the traditional hash value can be generated by the md5 digest algorithm. The traditional hash value is used to determine whether two chunks are the same during similarity calculation, and the simhash value is generated by the locality-sensitive hashing algorithm. The Hamming distance between the simhash values of two chunks is compared during similarity calculation to determine the similarity of the chunks. The Hamming distance represents the number of different bits at the corresponding positions of the binary strings of the two simhash values, and the smaller the number, the more similar.

[0058] Step S12: Generate first file fingerprint information and second file fingerprint information corresponding to the first target file and the second target file respectively based on the plurality of block fingerprint information, and determine the file similarity index of the first target file and the second target file based on the first file fingerprint information and the second file fingerprint information.

[0059] In this embodiment, the file fingerprint information is formed by merging all corresponding block fingerprints, and the file similarity index between two files is determined by comparing the file fingerprint information of the two files; wherein, the file fingerprint includes the block size, traditional hash value, and simhash value of each file block; the file similarity index includes file similarity, file similar byte count, block same rate, and block similarity rate. Specifically, determining the file similarity index of the first target file and the second target file based on the first file fingerprint information and the second file fingerprint information may include: performing similarity calculation based on the first file fingerprint information and the second file fingerprint information to obtain the file similarity and the file similar byte count; determining the same file blocks in the first target file and the second target file based on the traditional hash value, and calculating the block same rate based on the same file blocks; calculating the Hamming distance of other file blocks based on the simhash value, so as to determine similar file blocks based on the Hamming distance and a preset Hamming distance similarity mapping table, and calculating the block similarity rate based on the similar file blocks; generating the file similarity index of the first target file and the second target file based on the file similarity, file similar byte count, block same rate, and block similarity rate.

[0060] It is worth mentioning that after generating the first file fingerprint information and the second file fingerprint information corresponding to the first target file and the second target file respectively based on the plurality of block fingerprint information, it may further include: retaining the block index and block file content of each file block for subsequent steps to generate the corresponding target file content.

[0061] Step S13: Determine the target file content based on the file similarity index and the file content of the first target file and the second target file.

[0062] In this embodiment, determine the target file content based on the file similarity index and the file content of the first target file and the second target file.

[0063] In this embodiment, the target file content includes identical file content and similar file content. Specifically, determining the target file content based on the file similarity index and the file content of the first target file and the second target file includes: determining the identical file chunks and the chunk indexes corresponding to the similar file chunks based on the file similarity index; restoring the chunk file content corresponding to the identical file chunks and the similar file chunks based on the chunk indexes to obtain the target file content. That is, by inputting the content of two comparison files, the indexes of the identical and similar chunks can be recorded during similarity calculation, and according to the chunk indexes, the content of the identical and similar chunks of the two files can be restored and output.

[0064] As can be seen from the above, in the embodiment of the present application, the file is chunked by using the content dynamic chunking algorithm, and the fingerprint information of the file chunks is generated. The similarity between each chunk of the two files is calculated through the fingerprint information of the file chunks, and finally the chunk similarity is merged through a specified algorithm to generate the similarity of the file, which can improve the accuracy of text similarity calculation with a large length difference. Based on the locality-sensitive hashing algorithm technology, the similarity calculation is realized by extracting the binary fingerprint of the file chunk instead of directly analyzing its content. This method is general and has high calculation efficiency. Through the file and chunk similarity, the identical content (or paragraphs) and similar content (or paragraphs) of the file can be found, and the relevance of the file can be expressed by the similarity, and the scenarios where the overall file content or part of the file content is similar can be identified. Thus, the overall file content or part of the file content can be identified based on the calculated text similarity.

[0065] See Figure 2 As shown, the embodiment of the present application discloses a specific method for identifying file content based on text chunking, including:

[0066] The embodiment of the present application includes two stages. The first stage generates file fingerprints, and the second stage calculates the similarity index of the files according to the file fingerprints. Among them, generating file fingerprints is divided into two steps. The first step is to chunk the file, and the second step is to generate the fingerprint information of the file chunks. It can be understood that after calculating the similarity of the two file fingerprints, a similarity index will be generated. The similarity index includes two main indicators, namely file similarity (0-1) and file similar byte count, and some auxiliary indicators such as chunk identical rate and chunk similarity rate. Finally, by inputting the content of two comparison files, the content of the identical and similar chunks of the two files is output.

[0067] In this embodiment, the content-based dynamic chunking algorithm is used. Its core idea is to determine the chunk boundaries based on content rather than location or size, ensuring that even the same file processed at different time points can produce consistent chunking results. It supports the configuration of chunking strategies with two different chunk sizes, generating two types of file chunks, which further improves the accuracy of file similarity calculation. In actual applications, the single-chunk strategy or the double-chunk strategy can be selected according to the computing resource situation.

[0068] In this embodiment, refer to Figure 3 As shown, the file fingerprint is composed of the merged fingerprints of all file chunks. The file chunk fingerprint includes the chunk size, traditional hash value, and simhash value. The traditional hash value is generated by the md5 digest algorithm and is used to determine whether two chunks are the same during similarity calculation. The simhash value is generated by the locality-sensitive hashing algorithm. During similarity calculation, the Hamming distance between the simhash values of two chunks is compared to determine the similarity of the chunks. The Hamming distance represents the number of different bits at the corresponding positions of the binary strings of two simhash values. The smaller the number, the more similar. It should be noted that refer to Figure 4 As shown, if two file chunk strategies are used simultaneously, two file fingerprints will be generated respectively. The two fingerprints are connected by a preset symbol to generate the final file fingerprint.

[0069] In this embodiment, for the comparison of the chunks of two files, first compare whether the hash values are the same, and directly divide the total number of chunks with the same hash value by the total number of chunks in the comparison file to obtain the chunk similarity rate. Refer to Figure 5 As shown, through comparison, the hash values of the three pairs of chunks b1-b`2, b4-b`5, and b7-b`6 are the same. Further, for the chunks with different hash values, the Hamming distance is calculated using the simhash value. The Hamming distance is an integer from 0 to 64. The smaller the value, the more similar the files. By defining a mapping table from Hamming distance to similarity, the Hamming distance of the chunks is converted into the similarity in the table during calculation. Finally, the similarities of all chunks are added up and divided by the total number of chunks to obtain the chunk similarity rate. Refer to Figure 6 As shown, the Hamming distances of the simhash values of b2, b3, b5, and b6 are calculated with b`1, b`3, and b`4 in sequence. After calculation, the combination with the smallest Hamming distance among b`1, b`3, and b`4 is selected, such as b`1-b3, b`3-b2, b`4-b6 (as Figure 6 shown by the red line in). Finally, the contents of the two compared files are input. During similarity calculation, the indexes of the same and similar chunks can be recorded. Finally, according to the chunk indexes, the contents of the same and similar chunks of the two files can be restored and output.

[0070] As can be seen from the above, the embodiments of the present application combine two technologies, namely the content-based dynamic chunking algorithm (CDC) and the locality-sensitive hashing algorithm (simhash), to calculate the similarity of files. The two technologies complement each other, featuring faster recognition speed, higher accuracy, and more recognition scenarios. Generating file fingerprints does not require feature analysis of the file. File similarity calculation is based on file fingerprints without the need to know the file content. The Hamming distance is converted into a similarity between 0 and 1 through the Hamming distance similarity mapping table. On this basis, various similarity metrics such as file similarity, file similar byte count, chunk identical rate, and chunk similarity rate are generated, supporting the recognition of various file content scenarios.

[0071] See Figure 7 As shown, the embodiments of the present application disclose a file content recognition device based on text chunking, including:

[0072] A file chunking module 11, configured to perform chunking operations on a first target file and a second target file respectively to obtain a number of file chunks corresponding to the first target file and the second target file respectively, and calculate the chunk fingerprint information corresponding to each file chunk;

[0073] A file similarity metric generation module 12, configured to generate a first file fingerprint information and a second file fingerprint information corresponding to the first target file and the second target file respectively based on the number of chunk fingerprint information, and determine the file similarity metrics of the first target file and the second target file based on the first file fingerprint information and the second file fingerprint information; the file similarity metrics include file similarity, file similar byte count, chunk identical rate, and chunk similarity rate;

[0074] A file content recognition module 13, configured to determine the target file content based on the file similarity metrics and the file content of the first target file and the second target file; wherein, the target file content includes identical file content and similar file content.

[0075] As can be seen from the above, the embodiments of the present application chunk files and generate fingerprint information of the file chunks. The similarity between each chunk of two files is calculated through the fingerprint information of the file chunks. Finally, the chunk similarity is merged through a specified algorithm to generate the file similarity, which can improve the accuracy of text similarity calculation with a large length difference. Through file and chunk similarity, the identical content (or paragraphs) and similar content (or paragraphs) of the file can be found, and the relevance of the file can be expressed by similarity, enabling the recognition of scenarios where the overall file content or part of the file content is similar. Thus, the overall file content or part of the file content can be recognized based on the calculated text similarity.

[0076] In some specific embodiments, the file chunking module 11 may specifically include:

[0077] A file chunking sub-module, configured to perform chunking operations on the first target file and the second target file respectively based on a preset chunking strategy; wherein, the preset chunking strategy includes a dynamic chunking strategy based on file content and a chunking strategy based on natural paragraphs of the file;

[0078] A first chunk hash value generation unit, configured to determine the chunk length data of each file chunk and generate a first chunk hash value using a preset digest algorithm;

[0079] A second chunk hash value generation unit, configured to perform word segmentation on the file content corresponding to the file chunk and generate a second chunk hash value based on the word segmentation result using a preset locality-sensitive hashing algorithm;

[0080] A chunk fingerprint information generation unit, configured to generate the chunk fingerprint information based on the chunk length data, the first chunk hash value, and the second chunk hash value;

[0081] Further, the file chunking sub-module may specifically include:

[0082] A file chunking strategy configuration unit, configured to configure file chunking strategies with different chunk sizes and select one or two file chunking strategies from the file chunking strategies as the target file chunking strategy according to pre-set computing resource conditions;

[0083] A first file chunking execution unit, configured to determine chunk boundaries based on the target chunking strategy and perform chunking operations on the first target file and the second target file based on the chunk boundaries;

[0084] A file segmentation unit, configured to configure a minimum segmentation length and add segmentation marks to the file content of the first target file and the second target file based on the minimum segmentation length;

[0085] A second file chunking execution unit, configured to perform chunking operations on the first target file and the second target file based on the segmentation marks.

[0086] In some specific embodiments, the file similarity metric generation module 12 may specifically include:

[0087] A similarity calculation unit, configured to perform similarity calculation based on the first file fingerprint information and the second file fingerprint information to obtain the file similarity and the file similar byte count;

[0088] A block similarity rate calculation unit, configured to determine the same file blocks in the first target file and the second target file based on the first block hash value, and calculate the block similarity rate based on the same file blocks;

[0089] A block similarity ratio calculation unit, configured to calculate the Hamming distance of other file blocks based on the second block hash value, so as to determine similar file blocks based on the Hamming distance and a preset Hamming distance similarity mapping table, and calculate the block similarity ratio based on the similar file blocks;

[0090] A file similarity index generation unit, configured to generate a file similarity index for the first target file and the second target file based on the file similarity, the file similar byte count, the block similarity rate, and the block similarity ratio.

[0091] In some specific embodiments, the file content recognition device based on text blocks may further include:

[0092] A block index generation unit, configured to retain the block index and the block file content of each file block;

[0093] Correspondingly, the file content recognition module 13 may specifically include:

[0094] A block index determination unit, configured to determine the block indexes corresponding to the same file blocks and the similar file blocks based on the file similarity index;

[0095] A target file content determination unit, configured to restore the block file content corresponding to the same file blocks and the similar file blocks based on the block indexes to obtain the target file content.

[0096] Furthermore, an embodiment of the present application also discloses an electronic device, Figure 8 is a structural diagram of an electronic device 20 shown according to an exemplary embodiment, and the content in the figure cannot be considered as any limitation to the scope of use of the present application. The electronic device 20 may specifically include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. Among them, the memory 22 is used to store a computer program, and the computer program is loaded and executed by the processor 21 to implement the relevant steps in the file content recognition method based on text blocks disclosed in any of the foregoing embodiments. In addition, the electronic device 20 in this embodiment may specifically be an electronic computer.

[0097] In this embodiment, the power supply 23 is used to provide operating voltages for the various hardware devices on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and external devices, and the communication protocol it follows can be any communication protocol applicable to the technical solution of this application, and no specific limitation is imposed thereon here; the input / output interface 25 is used to obtain external input data or output data to the outside, and its specific interface type can be selected according to specific application requirements, and no specific limitation is made here.

[0098] In addition, the memory 22, as a carrier for resource storage, can be a read-only memory, a random access memory, a magnetic disk, an optical disc, etc. The resources stored thereon can include an operating system 221, a computer program 222, etc., and the storage method can be transient storage or permanent storage.

[0099] Among them, the operating system 221 is used to manage and control the various hardware devices and the computer program 222 on the electronic device 20, and it can be Windows Server, Netware, Unix, Linux, etc. In addition to the computer program that can be used to complete the file content recognition method based on text chunking executed by the electronic device 20 disclosed in any of the foregoing embodiments, the computer program 222 can further include computer programs that can be used to complete other specific tasks.

[0100] Furthermore, this application also discloses a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, it implements the file content recognition method based on text chunking disclosed above. For the specific steps of this method, reference can be made to the corresponding content disclosed in the foregoing embodiments, and details are not described herein again.

[0101] In this specification, the various embodiments are described in a progressive manner. Each embodiment focuses on the differences from other embodiments. The same or similar parts among the various embodiments can be referred to each other. For the device disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple, and the relevant parts can be referred to the description of the method part.

[0102] Those skilled in the art can further realize that the units and algorithm steps of the examples described in combination with the embodiments disclosed herein can be implemented by electronic hardware, computer software, or a combination of the two. To clearly illustrate the interchangeability of hardware and software, the components and steps of the examples have been generally described according to their functions in the above description. Whether these functions are executed in a hardware or software manner depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of this application.

[0103] The steps of the methods or algorithms described in connection with the embodiments disclosed herein may be implemented directly in hardware, in software modules executed by a processor, or in a combination thereof. The software modules may be placed in a random access memory (RAM), internal memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0104] Finally, it should also be noted that in this document, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprising", "including" or any other variant thereof are intended to cover non-exclusive inclusion, such that a process, method, article or device comprising a series of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article or device. Without further limitation, an element defined by the phrase "comprising a..." does not exclude the presence of additional identical elements in the process, method, article or device comprising the element.

[0105] The technical solutions provided in this application have been introduced in detail above. Specific examples are used herein to illustrate the principles and implementation manners of this application. The description of the above embodiments is only used to help understand the method and its core idea of this application; at the same time, for those of ordinary skill in the art, according to the idea of this application, there will be changes in the specific implementation manners and application scopes. In summary, the content of this specification should not be construed as a limitation to this application.

Claims

1. A method for identifying file content based on text chunking, characterized in that, Including: Performing chunking operations on the first target file and the second target file respectively to obtain a number of file chunks corresponding to the first target file and the second target file respectively, and calculating the chunk fingerprint information corresponding to each file chunk respectively; Generating first file fingerprint information and second file fingerprint information corresponding to the first target file and the second target file respectively based on the number of chunk fingerprint information, and determining the file similarity index of the first target file and the second target file based on the first file fingerprint information and the second file fingerprint information; the file similarity index includes file similarity, file similar byte count, chunk same rate, and chunk similarity rate; Determining the target file content based on the file similarity index and the file content of the first target file and the second target file; wherein, the target file content includes identical file content and similar file content.

2. The method for identifying file content based on text chunking according to claim 1, wherein The calculating the chunk fingerprint information corresponding to each file chunk respectively includes: Determining the chunk length data of each file chunk, and generating a first chunk hash value using a preset digest algorithm; Performing word segmentation on the file content corresponding to the file chunk, and generating a second chunk hash value based on the word segmentation result using a preset locality-sensitive hashing algorithm; Generating the chunk fingerprint information based on the chunk length data, the first chunk hash value, and the second chunk hash value.

3. The method for identifying file content based on text chunking according to claim 2, wherein The determining the file similarity index of the first target file and the second target file based on the first file fingerprint information and the second file fingerprint information includes: Performing similarity calculation based on the first file fingerprint information and the second file fingerprint information to obtain the file similarity and the file similar byte count; Determining the identical file chunks in the first target file and the second target file based on the first chunk hash value, and calculating the chunk same rate based on the identical file chunks; Calculating the Hamming distance of other file chunks based on the second chunk hash value, so as to determine the similar file chunks based on the Hamming distance and a preset Hamming distance similarity mapping table, and calculating the chunk similarity rate based on the similar file chunks; Generating the file similarity index of the first target file and the second target file based on the file similarity, the file similar byte count, the chunk same rate, and the chunk similarity rate.

4. The method for identifying file content based on text chunking according to claim 3, wherein After generating the first file fingerprint information and the second file fingerprint information corresponding to the first target file and the second target file respectively based on the number of chunk fingerprint information, further including: Retaining the chunk index and the chunk file content of each file chunk; Correspondingly, the determining the target file content based on the file similarity index and the file content of the first target file and the second target file includes: Determining the chunk indexes corresponding to the identical file chunks and the similar file chunks based on the file similarity index; Restoring the chunk file content corresponding to the identical file chunks and the similar file chunks based on the chunk indexes to obtain the target file content.

5. The method for identifying file content based on text chunking according to any one of claims 1 to 4, characterized in that The separately performing chunking operations on the first target file and the second target file includes: Performing chunking operations on the first target file and the second target file respectively based on a preset chunking strategy; Wherein, the preset chunking strategy includes a dynamic chunking strategy based on file content and a chunking strategy based on natural paragraphs of the file.

6. The method for identifying file content based on text chunking according to claim 5, wherein For the dynamic chunking strategy based on file content, the separately performing chunking operations on the first target file and the second target file based on the preset chunking strategy includes: Configuring file chunking strategies with different chunk sizes, and selecting one or two file chunking strategies from the file chunking strategies as target file chunking strategies according to preset computing resource conditions; Determining chunk boundaries based on the target chunking strategy, and performing chunking operations on the first target file and the second target file based on the chunk boundaries.

7. The method for identifying file content based on text chunking according to claim 5, characterized in that, For the chunking strategy based on natural paragraphs of the file, the separately performing chunking operations on the first target file and the second target file based on the preset chunking strategy includes: Configuring a minimum segmentation length, and adding segmentation marks to the file content of the first target file and the second target file based on the minimum segmentation length; Performing chunking operations on the first target file and the second target file based on the segmentation marks.

8. A file content recognition device based on text chunking, characterized in that, Includes: A file chunking module, configured to perform chunking operations on the first target file and the second target file respectively to obtain a number of file chunks corresponding to the first target file and the second target file respectively, and calculate chunk fingerprint information corresponding to each file chunk; A file similarity index generation module, configured to generate first file fingerprint information and second file fingerprint information corresponding to the first target file and the second target file respectively based on the number of chunk fingerprint information, and determine file similarity indexes of the first target file and the second target file based on the first file fingerprint information and the second file fingerprint information; the file similarity indexes include file similarity, file similar byte count, chunk same rate, and chunk similarity rate; A file content recognition module, configured to determine target file content based on the file similarity indexes and the file content of the first target file and the second target file; wherein, the target file content includes identical file content and similar file content.

9. An electronic device, characterized in that, Includes: A memory, configured to store a computer program; A processor, configured to execute the computer program to implement the file content recognition method based on text chunking according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, For storing a computer program, wherein the computer program, when executed by a processor, implements the file content recognition method based on text chunking according to any one of claims 1 to 7.

Images