Duplicate document detection method, device and related equipment

By calculating the jaccard distance between the document fingerprint and the candidate document fingerprint, the efficiency and accuracy of repeated document detection in massive text data are improved, and the problem of insufficient efficiency and accuracy in the prior art is solved.

CN118363931BActive Publication Date: 2024-11-19ATHENAEYES CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410781539.9
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-06-18
Publication Date
2024-11-19
Estimated Expiration
2044-06-18

AI Technical Summary

Technical Problem

When the prior art detects duplicate or approximate duplicate documents in massive text data, the efficiency and accuracy are insufficient, resulting in an increase in storage space usage and a decrease in information retrieval efficiency.

Method used

By calculating the simhash document fingerprint of the document to be detected, and generating a collection of candidate document fingerprints, calculating the jaccard distance between the candidate document fingerprint and the document to be detected, outputting the detection results based on the distance, improving the efficiency and accuracy of repeated document detection.

Benefits of technology

It effectively improves the efficiency and accuracy of repeated document detection and reduces the difficulty of users to obtain effective information.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118363931B_ABST
    Figure CN118363931B_ABST
Patent Text Reader

Abstract

The present invention discloses a duplicate document detection method, device and related equipment. The method comprises: obtaining a document X to be detected i ; Calculate the document X to be detected i The simhash document fingerprint F i ; According to the simhash document fingerprint F i , generate candidate document fingerprint set C i , the candidate document fingerprint set C i Includes at least one document fingerprint F i The corresponding bit flip fingerprint; calculate the document to be detected X i and the candidate document fingerprint set C i The jaccard distance of each candidate document fingerprint in the output; output the document to be detected X according to the jaccard distance i The present invention can improve the detection efficiency and accuracy of duplicate documents.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of information processing technology, and in particular to a duplicate document detection method, device and related equipment. Background Art

[0002] With the advent of the Internet era, information has exploded, and a huge amount of text data is generated every day. In the process of spreading these data on the Internet, it is inevitable that there will be a lot of duplicate or nearly duplicate content. These duplicate contents not only occupy valuable storage space, but also reduce the efficiency of information retrieval and processing, and also increase the difficulty for users to obtain effective information.

[0003] In order to solve the above problems, the near-duplicate document detection technology came into being. It aims to quickly and accurately detect these duplicate or near-duplicate documents from massive documents and improve the quality and availability of text data. Simhash is an algorithm proposed by Google to remove redundant data. The algorithm is a local sensitive hashing algorithm that can probabilistically reduce the dimension of high-dimensional data and map it to a fingerprint with a small number of bits and a fixed number. Then, the fingerprint is compared to reflect the similarity between the data. It is widely used in the fields of near-duplicate text detection, redundant data deduplication, anomaly detection, etc. However, due to the complexity and diversity of language, the Simhash algorithm may not be able to quickly and accurately determine whether two texts are duplicates or near-duplicates.

[0004] Therefore, how to further improve the detection efficiency and accuracy of duplicate documents is an urgent problem to be solved. Summary of the invention

[0005] The embodiments of the present invention provide a duplicate document detection method, apparatus and related equipment, which can improve the detection efficiency and accuracy of duplicate documents and reduce the difficulty for users to obtain valid information.

[0006] In order to solve the above technical problems, the present application provides a duplicate document detection method, comprising the following steps: obtaining a document X to be detected i ; Calculate the document X to be detected i The simhash document fingerprint F i ; According to the simhash document fingerprint F i , generate candidate document fingerprint set C i , the candidate document fingerprint set C i Includes at least one document fingerprint F i The corresponding bit flip fingerprint; calculate the document to be detected X i and the candidate document fingerprint set C iThe jaccard distance of each candidate document fingerprint in the output; output the document to be detected X according to the jaccard distance i The test results.

[0007] In a possible implementation, the document X to be detected is i Perform feature segmentation to obtain the text block feature set F i text ; Calculate the text block feature set F i text The weight W of each text block in i j ; Calculate the text block feature set F using the hash function i text The hash feature F of each text block in i hash ; The hash value feature F i hash and the W i j Merge, weight, and reduce the dimension to obtain the document X to be detected i The simhash document fingerprint F i .

[0008] In another possible implementation, the weight W below the first threshold is i j and the corresponding text blocks are deleted, and the weight W above the second threshold is i j and the corresponding text block to delete.

[0009] In another possible implementation, according to the document X to be detected i The simhash document fingerprint F i , the hash feature F i hash And the hash feature F i hash The corresponding weight W i j , calculate the document X to be detected i The simhash document fingerprint F i The bit flip probability P of each bit in i j ; According to the bit flip probability P i j Generate the candidate document fingerprint set C i .

[0010] In another possible implementation, a Hamming distance threshold is set; based on the Hamming distance threshold and the bit flip probability Pi j , determine the document X to be detected i The simhash document fingerprint F i The corresponding bit flipping number is less than or equal to the Hamming distance threshold bit flipping fingerprint set C total ; Calculate the bit flip fingerprint set C total Each bit flip fingerprint in the document to be detected is i The simhash document fingerprint F i Approximately repeat the probability value, and sort the bit flip fingerprint set C in the order of the probability value total The bit flip fingerprints in the document are arranged in descending order; the top k bit flip fingerprints are used as the candidate document fingerprint set C i .

[0011] In another possible implementation, the candidate document fingerprint set C is calculated in sequence i Each document T j The text block feature set F j text With the document to be detected X i The text block feature set F i text The jaccard distance between them is calculated as follows:

[0012] Dis jaccard =|F j text ∩F i text | / |F j text ∪F i text |

[0013] Among them, ∩ means taking the intersection, and ∪ means taking the union.

[0014] In another possible implementation, the language detection model is used to detect the document X to be detected. i Perform language detection to determine the document X to be detected i The language category to which it belongs; according to the document to be detected X i The language category to which the text belongs determines the detection library corresponding to the language category.

[0015] In order to solve the above technical problems, the present application also provides a duplicate document detection device, including: an acquisition module, for acquiring a document X to be detected. i The language category to which it belongs; a calculation module, used to calculate the document X to be detected i The simhash document fingerprint F i; Generate a module for generating fingerprint F according to the simhash document i , generate candidate document fingerprint set C i , the candidate document fingerprint set C i Includes at least one document fingerprint F i The corresponding bit flip fingerprint; the calculation module is also used to calculate the document to be detected X i and the candidate document fingerprint set C i The jaccard distance of each candidate document fingerprint in the output module is used to output the document to be detected X according to the jaccard distance. i The test results.

[0016] In order to solve the above technical problems, an embodiment of the present application also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above method when executing the computer program.

[0017] In order to solve the above technical problems, an embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the above method are implemented.

[0018] The duplicate document detection method, device and related equipment provided by the embodiment of the present invention calculate the document fingerprint of the document to be detected, calculate the bit flip probability of each bit in the document fingerprint, then generate a candidate document fingerprint set based on the bit flip probability, and then calculate the jaccard distance between each candidate document fingerprint in the candidate document fingerprint set and the document fingerprint of the document to be detected, so that the duplicate document detection result can be obtained according to the jaccard distance, that is, whether there are duplicate or nearly duplicate documents in the detection library. In this way, the detection efficiency and accuracy of duplicate documents can be improved, and the difficulty for users to obtain effective information can be reduced. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.

[0020] Figure 1 is an exemplary system architecture diagram to which the present application can be applied.

[0021] Figure 2 It is a flow chart of an embodiment of the duplicate document detection method of the present application.

[0022] Figure 3 It is a schematic diagram of the document fingerprint calculation process of this application.

[0023] Figure 4 It is a schematic diagram of the bit flipping of the present application.

[0024] Figure 5 It is a structural schematic diagram of an embodiment of a duplicate document detection device according to the present application.

[0025] Figure 6 It is a structural diagram of an embodiment of a computer device according to the present application. DETAILED DESCRIPTION

[0026] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by technicians in the technical field of the present application; the terms used in the specification of the application herein are only for the purpose of describing specific embodiments and are not intended to limit the present application; the terms "including" and "having" and any variations thereof in the specification and claims of the present application and the above-mentioned drawings are intended to cover non-exclusive inclusions. The terms "first", "second", etc. in the specification and claims of the present application or the above-mentioned drawings are used to distinguish different objects, not to describe a specific order.

[0027] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0028] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0029] See also Figure 1 ,like Figure 1 As shown, the system architecture 100 may include a document detection system 110, a network 120, and a text database 130. The network 120 is used to provide a medium for a communication link between the document detection system 110 and the text database 130. The network 120 may include various connection types, such as wired, wireless communication links, or fiber optic cables, etc.

[0030] Users can use document detection system 110 to interact with text database 130 via network 120 to receive or send messages, etc.

[0031] The text database 130 may be deployed in an edge computing device or a central computing device, for example, in a database server.

[0032] The document detection system 110 may be deployed on a server that provides document detection services, such as a backend server that provides similarity detection for documents stored in a server that deploys the text database 130 .

[0033] It should be noted that the duplicate document detection method provided in the embodiment of the present application is executed by a server, and accordingly, the duplicate document detection device is arranged in the server.

[0034] It should be understood that Figure 1 The number of text detection systems, networks and text databases in the embodiment is only for illustration. According to the implementation requirements, any number of text detection systems, networks and text databases can be provided.

[0035] See also Figure 2 , Figure 2 A duplicate document detection method provided by an embodiment of the present invention is shown and described in detail as follows.

[0036] S201, obtain the document X to be detected i .

[0037] Optionally, the document to be detected X can be obtained in a variety of ways i , for example, obtain the document X to be detected from the above-mentioned text database 130 i , or directly obtain the document to be detected X through user input i , or use a crawler tool to crawl the document to be tested from the Internet i , this application is for the document to be detected X i The specific method of obtaining is not limited.

[0038] In a possible implementation, after obtaining the document X to be detected i Afterwards, the language detection model is used to detect the document X i Perform language detection to determine the document to be detected X i The language category to which it belongs; according to the document to be tested X i The language category to which it belongs determines the detection library corresponding to the language category.

[0039] Specifically, for the documents to be tested X in different languages i , different types of detection libraries are constructed. Therefore, after obtaining the document to be detected X iAfter that, the language detection model needs to be used to determine the acquired document X to be detected i The language category to which it belongs is determined, and then the detection library corresponding to the language category is selected to complete the subsequent detection process.

[0040] S202, calculate the document X to be detected i The simhash document fingerprint F i .

[0041] Specifically, the document X to be detected can be calculated according to the simhash algorithm i The document fingerprint F i .

[0042] In a possible implementation, the document X to be detected i Perform feature segmentation to obtain the text block feature set F i text , calculate the feature set F of the text block i text The weight W of each text block in i j , using hash function to calculate the text block feature set F i text The hash feature F of each text block in i hash , the hash feature F i hash and weight W i j Merge, weight, and reduce the dimension to obtain the document X to be detected i The simhash document fingerprint F i .

[0043] Optionally, in the document to be detected X i When performing feature segmentation on the text in the text block, different text segmentation methods can be used, such as text segmentation, n-gram-t-skip, etc., to segment it, thereby obtaining the text block feature set F i text This application does not specifically limit the specific segmentation method to be used.

[0044] In addition, this application uses the term frequency-inverse document frequency (TF-IDF) statistical method to calculate the text block feature set F i text The weight W of each text block in i jTF refers to the frequency of a word appearing in a text. In a long text, the number of times a word appears will be higher, so the frequency is used to calculate the frequency in this algorithm. IDF refers to the inverse document frequency. The fewer documents containing a word, the greater the IDF. IDF=log(total number of documents / number of documents containing the word+1). For example, assuming there are 1,000 articles, the word "snacks" appears 10 times in article A, and there are 100 words in A, then TF=10 / 100=0.1, and "snacks" has appeared in 50 articles, then IDF=log(1000 / 51)=1.29, so, .

[0045] In a possible implementation, the weight W below the first threshold is i j and the corresponding text blocks are deleted, and the weight W above the second threshold is i j and the corresponding text block to delete.

[0046] Specifically, after calculating the weight of each text block, each weight value needs to be compared with the first threshold and the second threshold. If it is lower than the first threshold or higher than the second threshold, then the weight value and the feature block corresponding to the weight value need to be deleted, where the first threshold and the second threshold can be set according to actual needs, and this application does not limit the specific settings.

[0047] It can be understood that feature blocks that are lower than the first threshold or higher than the second threshold are dispensable features in the document and will not have much impact on the document semantics, but will affect the effect of document near-duplicate detection. For example, the frequency of occurrence of certain modal particles is too high, which affects the detection effect. Therefore, it is necessary to delete weight values ​​and feature blocks that are too high or too low, so as to improve the detection effect and accuracy of duplicate documents.

[0048] Furthermore, a hash function (such as UniforHash) is used to calculate the text block feature set F i text The hash value of each text block in the text block is the n-bit hash feature F composed of the binary number 01. i hash In this application, n is preferably 64, wherein f i j Represents the hash value feature corresponding to the j-th text block feature. Finally, the hash feature F i hash and the corresponding weight W i j Merge, weight, and reduce the dimension to obtain the document X to be detected i The simhash document fingerprint F i .

[0049] For example, Figure 3 As shown in the figure, for the document to be detected, it is processed by word segmentation, and weights can be set for the word segmentation, so that a document can be turned into a set of (word segmentation (feature): weight (weight)) with a length of n. Then, each word segmentation is hashed to obtain a 64-bit binary hash value, and then a set of (hash value (hash): weight (weight)) with a length of n is obtained. For each position in the hash value, whether it is 1 or 0, a positive weight or a negative weight is selected accordingly. If a position is 1, the weight of the corresponding position is a positive weight. If a position is 0, the weight of the corresponding position is a negative weight. Then weighting is performed, and the weight vector is equal to the hash value of each word segmentation multiplied by the weight corresponding to the word. Then, the weight vectors of all word segmentations are accumulated column-wise to obtain a new weight vector, such as (13, 108, -22, -5, -32, 55) in the figure. Finally, dimensionality reduction is performed, that is, positions greater than 0 are set to 1, and positions less than or equal to 0 are set to 0, so as to obtain the document X to be detected. i The simhash document fingerprint F i , such as (110001) in the figure.

[0050] S203, according to the simhash document fingerprint F i , generate candidate document fingerprint set C i .

[0051] Specifically, after obtaining the document fingerprint F i Afterwards, the candidate document fingerprint set C can be generated i , in the candidate document fingerprint set C i contains at least one corresponding bit flip fingerprint.

[0052] In a possible implementation, according to the document X to be detected i The simhash document fingerprint F i , hash feature F i hash and hash feature F i hash The corresponding weight W i j , calculate the document to be detected X i The simhash document fingerprint F i The bit flip probability P of each bit in i j , according to the bit flip probability P i j Generate candidate document fingerprint set C i .

[0053] Specifically, after calculating the fingerprint F of the document to be detected i Afterwards, based on the document fingerprint F i And the hash feature F i hash and the corresponding weight W i j , calculate the document fingerprint F i The flip probability of each bit in , that is, the document fingerprint F i The probability value of each of the 64 bits flipping from 1 to 0, or from 0 to 1.

[0054] See also Figure 4 , Figure 4 is a bit flip diagram, such as Figure 4 As shown, the second bit flips from 1 to 0, and its flip probability is p i 2 , the flip probability of each other bit can be calculated by the following formula:

[0055] P i j =p(Y j >|W j (u)

[0056] Where u represents document X i The corresponding n-bit feature F i hash The collection of W j (u) represents the weight of the jth bit, which is defined as shown in the following formula:

[0057] ;

[0058] Among them, f j t Represents the j-th n-bit feature F i hash The value of the tth bit, w j t (u) represents the j-th n-bit feature F i hash Corresponding weight W i j The value of .

[0059] Furthermore, using the calculated bit flip probability P i j Calculate document fingerprint F i Similar candidate fingerprint set C under Hamming distance i .

[0060] Specifically, a Hamming distance threshold is set. The threshold can be set according to actual needs. In this application, it is preferably set to 3. When the Hamming distance between the fingerprints of two documents is less than or equal to 3, the two documents are judged to be nearly duplicated; when the Hamming distance between the fingerprints of two documents is greater than 3, the two documents are judged to be non-duplicates. Set document X i The document fingerprint F i The corresponding flip fingerprint set is C total , C total Each candidate fingerprint in the document to be detected X i The probability of near-duplicates is calculated using the following formula:

[0061] ;

[0062] Among them, S represents the flip fingerprint set C total The i-th candidate document fingerprint F c-j The index value with the middle bit set to 1, for example, F c-j is 10001, then S={1,5}.

[0063] It should be noted that the right side of the above equation is divided into two parts. The first half represents the flip probability product of the bit value of 0 in the document fingerprint to be detected flipped to 1, and the second half represents the flip probability product of the bit value of 1 in the document fingerprint to be detected flipped to 0. The product of the two parts represents the approximate duplicate conditional probability of the candidate document fingerprint corresponding to the document fingerprint to be detected under the set threshold.

[0064] After calculating each candidate fingerprint for the document to be detected X i After the approximate duplication probability is calculated, the fingerprints are sorted in descending order according to the probability value, and the top k flipped fingerprints are taken as the candidate document fingerprint set C. i .

[0065] For example, the Hamming distance threshold is defined as 3. Assuming that the document fingerprint of the document to be detected is (100001), then for this fingerprint, we can obtain a set of only 1-bit flipped fingerprints: (000001, 110001, 101001, 100101, 100011, 100000), a set of 2-bit flipped fingerprints: (010001, 111001, ..., 100010), and a set of 3-bit flipped fingerprints: (011001, 111101, ..., 100110). Then, the probability of each candidate fingerprint in each set being approximately repeated for the document to be detected is calculated, and finally the probability sizes are sorted in descending order, and the top k flipped fingerprints are selected as the candidate document fingerprint set.

[0066] It should be understood that the greater the flip probability of a certain bit, the greater the possibility of the bit flipping, and vice versa. The greater the flip probability of a bit, the greater the possibility of the document fingerprint of a similar document flipping at that bit, that is, the document fingerprint with a flipped bit has a greater probability of being a candidate near-duplicate document of the document to be detected. For example, assuming that the fingerprint of the document to be detected is (1001), then the size of the candidate near-duplicate document fingerprint set is 2 to the fourth power, that is, 16, and the flip probability of each bit is: (0.9, 0.7, 0.5, 0.6). It can be seen that the flip probability of the 0th bit is the largest, so (0001) has a greater probability of being a candidate near-duplicate document fingerprint of the document to be detected. Therefore, the purpose of calculating bit flips in this application is to select a subset of candidate documents that are most likely to be near-duplicate documents from a large set of fingerprint candidates.

[0067] S204, calculate the document X to be detected i and the candidate document fingerprint set C i The jaccard distance of each candidate document fingerprint in .

[0068] Specifically, the candidate document fingerprint set C obtained in step S3 is calculated in sequence: i Each document T j The text block feature set F j text With Document X i The text block feature set F i text The jaccard distance between them can be calculated by the following formula:

[0069] Dis jaccard =|F j text ∩F i text | / |F j text ∪F i text |

[0070] Among them, ∩ means taking the intersection, and ∪ means taking the union.

[0071] S205: Output the document to be detected X according to the jaccard distance i The test results.

[0072] Specifically, after obtaining the jaccard distance between the text block feature set of each candidate document in the candidate document fingerprint set and the text block feature set of the document to be detected in step S204, the candidate documents corresponding to the text block feature sets with a distance greater than a threshold are put into the candidate document set.

[0073] Furthermore, if the number of documents in the candidate document set is greater than 0, it means that the document to be detected X i There are nearly duplicate documents in the library, and the output detection result is: exists, document X i Not stored in the database; if the number of documents in the candidate document set is 0, it means that the document to be detected is X i There is no near-duplicate document in the database, and the output detection result is: does not exist, document X i Storage in warehouse.

[0074] In this embodiment, the document fingerprint of the document to be detected is calculated, the bit flip probability of each bit is calculated based on the document fingerprint, and a candidate document fingerprint set is generated according to the bit flip probability, so as to select a candidate document subset that is most likely to be a near-duplicate document from a huge fingerprint candidate set. Finally, the jaccard distance between the document to be detected and all candidates in the candidate document fingerprint set is calculated to output the detection result. Compared with the traditional detection algorithm, the detection efficiency and accuracy of duplicate documents can be effectively improved, thereby reducing the difficulty for users to obtain effective information.

[0075] It should be understood that the order of execution of the steps in the above embodiment does not necessarily mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present invention.

[0076] Figure 5 FIG. 2 is a block diagram showing a principle of a duplicate document detection device corresponding to the duplicate document detection method in the above embodiment. Figure 5 As shown, the duplicate document detection device 500 includes an acquisition module 510, a calculation module 520, a generation module 530 and an output module 540. Each functional module is described in detail as follows.

[0077] The acquisition module 510 is used to acquire the document X to be detected. i The language category.

[0078] The calculation module 520 is used to calculate the document X to be detected. i The simhash document fingerprint F i .

[0079] Generating module 530, for generating a fingerprint F according to the simhash document i , generate candidate document fingerprint set C i , candidate document fingerprint set C i Includes at least one document fingerprint F i The corresponding bit-flip fingerprint.

[0080] The calculation module 520 is also used to calculate the document X to be detected.i and the candidate document fingerprint set C i The jaccard distance of each candidate document fingerprint in .

[0081] Output module 540, used to output the document to be detected X according to the jaccard distance i The test results.

[0082] The specific definition of the duplicate document detection device can be found in the definition of the duplicate document detection method above, which will not be repeated here. Each module in the above-mentioned duplicate document detection device can be implemented in whole or in part by software, hardware and a combination thereof. Each of the above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory in the computer device in the form of software, so that the processor can call and execute the operations corresponding to each of the above modules.

[0083] To solve the above technical problems, the present application also provides a computer device. Figure 6 , Figure 6 This is a basic structural block diagram of the computer device in this embodiment.

[0084] The computer device 600 includes a memory 610, a processor 620, and a network interface 630 that are interconnected through a system bus. It should be noted that the figure only shows a computer device 600 having components connected to the memory 610, the processor 620, and the network interface 630, but it should be understood that it is not required to implement all the components shown, and more or fewer components can be implemented instead. Among them, those skilled in the art can understand that the computer device here is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to microprocessors, application specific integrated circuits (ASIC), programmable gate arrays (FPGA), digital signal processors (DSP), embedded devices, etc.

[0085] The computer device may be a computing device such as a desktop computer, a notebook, a PDA, a cloud server, etc. The computer device may interact with a user through a keyboard, a mouse, a remote controller, a touch pad, or a voice control device.

[0086] The memory 610 includes at least one type of readable storage medium, and the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or D interface display memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 610 can be an internal storage unit of the computer device 600, such as a hard disk or memory of the computer device 600. In other embodiments, the memory 610 can also be an external storage device of the computer device 600, such as a plug-in hard disk equipped on the computer device 600, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. Of course, the memory 610 can also include both the internal storage unit of the computer device 600 and its external storage device. In this embodiment, the memory 610 is generally used to store an operating system and various application software installed on the computer device 600, such as program codes for controlling electronic files, etc. In addition, the memory 610 can also be used to temporarily store various data that have been output or are to be output.

[0087] The processor 620 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chips in some embodiments. The processor 620 is generally used to control the overall operation of the computer device 600. In this embodiment, the processor 620 is used to run the program code stored in the memory 610 or process data, such as running the program code for controlling the electronic file.

[0088] The network interface 630 may include a wireless network interface or a wired network interface. The network interface 630 is generally used to establish a communication connection between the computer device 600 and other electronic devices.

[0089] The present application also provides another implementation, namely, providing a computer-readable storage medium, wherein the computer-readable storage medium stores an interface display program, and the interface display program can be executed by at least one processor to enable the at least one processor to perform the steps of the image library construction method as described above.

[0090] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for enabling a terminal device (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0091] Obviously, the embodiments described above are only some embodiments of the present application, rather than all embodiments. The preferred embodiments of the present application are given in the accompanying drawings, but they do not limit the patent scope of the present application. The present application can be implemented in many different forms. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present application more thorough and comprehensive. Although the present application is described in detail with reference to the aforementioned embodiments, for those skilled in the art, it is still possible to modify the technical solutions recorded in the aforementioned specific implementation methods, or to perform equivalent replacement of some of the technical features therein. Any equivalent structure made using the contents of the specification and drawings of this application, directly or indirectly used in other related technical fields, is similarly within the scope of patent protection of this application.

Claims

1. A duplicate document detection method, characterized in that: The method comprises: Get the document Xi to be detected; Perform feature segmentation on the document Xi to be detected to obtain a text block feature set Fitext, and calculate the weight Wij of each text block in the text block feature set Fitext; Calculate the hash feature Fihash of each text block in the text block feature set Fitext by using a hash function, and merge, weight, and reduce the dimension of the hash feature Fihash and Wij to obtain the simhash document fingerprint Fi of the document Xi to be detected; According to the simhash document fingerprint Fi of the document Xi to be detected, the hash feature Fihash and the weight Wij corresponding to the hash feature Fihash, calculate the bit flip probability Pij of each bit in the simhash document fingerprint Fi of the document Xi to be detected; A Hamming distance threshold is set, and based on the Hamming distance threshold and the bit flip probability Pij, a bit flip fingerprint set Ctotal whose bit flip number corresponding to the simhash document fingerprint Fi of the document to be detected Xi is less than or equal to the Hamming distance threshold is determined; Calculate the probability value of approximate duplication between each bit-flipped fingerprint in the bit-flipped fingerprint set Ctotal and the simhash document fingerprint Fi of the document to be detected Xi, and arrange the bit-flipped fingerprints in the bit-flipped fingerprint set Ctotal in descending order according to the probability value, and take the top k bit-flipped fingerprints as the candidate document fingerprint set Ci; Calculate the jaccard distance between the document to be detected Xi and each candidate document fingerprint in the candidate document fingerprint set Ci; The detection result of the document Xi to be detected is output according to the jaccard distance.

2. The method according to claim 1, characterized in that Before calculating the hash value and the hash value feature Fihash of each text block in the text block feature set Fitext by using a hash function, the method further includes: The weight Wij and the corresponding text blocks that are lower than the first threshold are deleted, and the weight Wij and the corresponding text blocks that are higher than the second threshold are deleted.

3. The method according to claim 1 or 2, characterized in that The calculation of the jaccard distance between the document to be detected Xi and each candidate document fingerprint in the candidate document fingerprint set Ci includes: The jaccard distance between the text block feature set Fjtext of each document Tj in the candidate document fingerprint set Ci and the text block feature set Fitext of the document Xi to be detected is calculated in sequence. The calculation formula is as follows: Disjaccard=|Fjtext∩Fitext| / |Fjtext∪Fitext| Among them, ∩ means taking the intersection, and ∪ means taking the union.

4. The method according to claim 1, characterized in that After obtaining the document Xi to be detected, the method further includes: Performing language detection on the document Xi to be detected by using a language detection model to determine the language category to which the document Xi to be detected belongs; According to the language category to which the document Xi to be detected belongs, a detection library corresponding to the language category is determined.

5. A duplicate document detection device, characterized in that: The device comprises: An acquisition module is used to acquire the language category to which the document Xi to be detected belongs; A calculation module is used to perform feature segmentation on the document Xi to be detected, obtain a text block feature set Fitext, calculate a weight Wij of each text block in the text block feature set Fitext, calculate a hash feature Fihash of each text block in the text block feature set Fitext using a hash function, and merge, weight, and reduce the dimension of the hash feature Fihash and Wij to obtain a simhash document fingerprint Fi of the document Xi to be detected; A generating module is used to calculate the bit flip probability Pij of each bit in the simhash document fingerprint Fi of the document Xi to be detected according to the simhash document fingerprint Fi of the document Xi to be detected, the hash feature Fihash and the weight Wij corresponding to the hash feature Fihash; set a Hamming distance threshold, and based on the Hamming distance threshold and the bit flip probability Pij, determine a bit flip fingerprint set Ctotal whose bit flip number corresponding to the simhash document fingerprint Fi of the document Xi to be detected is less than or equal to the Hamming distance threshold, calculate the probability value of each bit flip fingerprint in the bit flip fingerprint set Ctotal being approximately repeated with the simhash document fingerprint Fi of the document Xi to be detected, and arrange the bit flip fingerprints in the bit flip fingerprint set Ctotal in descending order according to the order of the probability values, and use the top k bit flip fingerprints as the candidate document fingerprint set Ci; The calculation module is further used to calculate the jaccard distance between the document to be detected Xi and each candidate document fingerprint in the candidate document fingerprint set Ci; An output module is used to output the detection result of the document Xi to be detected according to the jaccard distance.

6. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 4 is implemented.

7. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 4 is implemented.