Document duplicate checking method and device, computer equipment and storage medium

Abstract

The invention relates to a document duplicate checking method and device, computer equipment and a storage medium. The method comprises the following steps: acquiring at least two documents to be subjected to duplicate checking; for each document to be subjected to duplicate checking, determining a text vector corresponding to a document text in the document to be subjected to duplicate checking, and determining an image set corresponding to a document image in the document to be subjected to duplicate checking; any two documents to be subjected to duplicate checking are selected as a first document and a second document respectively; according to the content similarity of the document content of the first document and the document content of the second document under different content categories; and according to the content similarity under the different content categories, determining the repetition degree between the first document and the second document. By adopting the method, the document comparison efficiency can be improved, and the document auditing efficiency and accuracy can be improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a document duplication checking method, apparatus, computer equipment and storage medium. Background Art

[0002] With the popularization of electronic bidding, the transparency and efficiency in the bidding field have been improved, but it also faces the challenge of unfair competition such as bid rigging and collusion. Especially in the expert review process, it is necessary to identify possible unfair competition behaviors, reduce potential risks in the subsequent audit process, and ensure the fairness and transparency of bidding activities. Therefore, how to accurately review bidding-related documents is crucial.

[0003] In traditional technology, the commonly used means of identifying unfair competition behaviors such as bid rigging and collusion are, firstly, to determine whether the bids are made by the same machine or the same IP segment based on the network information and hardware information such as the IP address, MAC address, network card serial number, etc. submitted by the bidders; secondly, to determine whether there is a direct or indirect relationship between the bidders based on the bidder information (legal person, equity, contact information, etc.).

[0004] Although this method can complete the review of bidding-related documents, there is a problem of inaccurate review. Summary of the invention

[0005] Based on this, it is necessary to provide a document duplication checking method, device, computer equipment and storage medium that can accurately review the above-mentioned technical problems.

[0006] In a first aspect, the present application provides a document duplication checking method, comprising:

[0007] Obtain at least two documents to be checked for duplicates;

[0008] For each document to be checked for duplicates, determine a text vector corresponding to the document text in the document to be checked for duplicates, and determine an image set corresponding to the document image in the document to be checked for duplicates;

[0009] Select any two documents to be checked for duplicates as the first document and the second document respectively;

[0010] content similarity between the first document and the second document in different content categories;

[0011] The degree of repetition between the first document and the second document is determined according to the content similarities under different content categories.

[0012] In one embodiment, the content similarity under different content categories includes text similarity under text category and image similarity under image category; accordingly, according to the content similarity between the document content of the first document and the document content of the second document under different content categories, the content similarity includes:

[0013] Determining text similarity between the text vector of the first document and the text vector of the second document; and,

[0014] An image similarity is determined between the image set of the first document and the image set of the second document.

[0015] In one embodiment, determining image similarity between an image set of a first document and an image set of a second document includes:

[0016] extracting image vocabulary in the image collection of the first document and image vocabulary in the image collection of the second document;

[0017] The image similarity between the image set of the first document and the image set of the second document is determined according to the distribution of each image word in the image set of the first document and the image set of the first document.

[0018] In one embodiment, determining the image similarity between the image set of the first document and the image set of the second document according to the distribution of each image word in the image set of the first document and in the image set of the first document includes:

[0019] determining a first number of identical image words in the image collection of the first document and in the image collection of the second document;

[0020] determining a second number of all image words in the image collection of the first document and the image collection of the second document;

[0021] The image similarity is determined according to the proportion of the first quantity in the second quantity.

[0022] In one embodiment, determining an image set corresponding to a document image in a document to be checked for duplicates includes:

[0023] Obtaining a document image from the document to be checked for duplicates;

[0024] For each document image, text detection is performed on the document image to obtain an image text of the document image; the image text includes at least one image word;

[0025] Based on the image vocabulary of each document image, an image set of the document image is generated.

[0026] In one embodiment, determining a text vector corresponding to a document text in a document to be checked for duplicates includes:

[0027] Get the document text in the document to be checked for duplicates;

[0028] Perform word segmentation on the document text to obtain a vocabulary set of the document text;

[0029] The text vector of the document text is determined based on the word vector of each text word in the vocabulary set.

[0030] In a second aspect, the present application also provides a document duplication checking device, comprising:

[0031] A document acquisition module is used to acquire at least two documents to be checked for duplicates;

[0032] A document separation module is used to determine, for each document to be checked for duplicates, a text vector corresponding to the document text in the document to be checked for duplicates, and to determine an image set corresponding to the document image in the document to be checked for duplicates;

[0033] A document selection module, used to select any two documents to be checked for duplicates as the first document and the second document respectively;

[0034] A similarity determination module, configured to determine content similarities between the document content of the first document and the document content of the second document under different content categories;

[0035] The duplication determination module is used to determine the duplication between the first document and the second document according to the content similarity under different content categories.

[0036] In a third aspect, the present application further provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0037] Obtain at least two documents to be checked for duplicates;

[0038] For each document to be checked for duplicates, determine a text vector corresponding to the document text in the document to be checked for duplicates, and determine an image set corresponding to the document image in the document to be checked for duplicates;

[0039] Select any two documents to be checked for duplicates as the first document and the second document respectively;

[0040] content similarity between the first document and the second document in different content categories;

[0041] The degree of repetition between the first document and the second document is determined according to the content similarities under different content categories.

[0042] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the following steps are implemented:

[0043] Obtain at least two documents to be checked for duplicates;

[0044] For each document to be checked for duplicates, determine a text vector corresponding to the document text in the document to be checked for duplicates, and determine an image set corresponding to the document image in the document to be checked for duplicates;

[0045] Select any two documents to be checked for duplicates as the first document and the second document respectively;

[0046] content similarity between the first document and the second document in different content categories;

[0047] The degree of repetition between the first document and the second document is determined according to the content similarities under different content categories.

[0048] In a fifth aspect, the present application further provides a computer program product, including a computer program, which implements the following steps when executed by a processor:

[0049] Obtain at least two documents to be checked for duplicates;

[0050] For each document to be checked for duplicates, determine a text vector corresponding to the document text in the document to be checked for duplicates, and determine an image set corresponding to the document image in the document to be checked for duplicates;

[0051] Select any two documents to be checked for duplicates as the first document and the second document respectively;

[0052] content similarity between the first document and the second document in different content categories;

[0053] The degree of repetition between the first document and the second document is determined according to the content similarities under different content categories.

[0054] The above-mentioned document duplicate checking method, device, computer equipment and storage medium obtain at least two documents to be checked for duplicates; for each document to be checked for duplicates, determine the text vector corresponding to the document text in the document to be checked for duplicates, and determine the image set corresponding to the document image in the document to be checked for duplicates; select any two documents to be checked for duplicates as the first document and the second document respectively; determine the duplication between the first document and the second document based on the content similarity between the document content of the first document and the document content of the second document under different content categories; and determine the duplication between the first document and the second document based on the content similarity under different content categories. This embodiment can improve the efficiency of document comparison and improve the efficiency and accuracy of document review. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the related technologies, the drawings required for use in the embodiments or the related technical descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0056] FIG1 is an application environment diagram of a document duplication checking method provided by this embodiment;

[0057] FIG2 is a schematic diagram of a flow chart of a first document duplication checking method provided in this embodiment;

[0058] FIG3 is a schematic diagram of a flow chart of a second document duplication checking method provided in this embodiment;

[0059] FIG4 is a structural block diagram of a document duplication checking device provided in this embodiment;

[0060] FIG5 is an internal structure diagram of a first computer device provided in this embodiment;

[0061] FIG6 is a diagram showing the internal structure of a second type of computer device provided in this embodiment. DETAILED DESCRIPTION

[0062] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0063] The document duplicate checking method provided in the embodiment of the present application can be applied in the application environment shown in Figure 1. Among them, the terminal 102 communicates with the server 104 through the network. The data storage system can store the data that the server 104 needs to process. The data storage system can be integrated on the server 104, or it can be placed on the cloud or other network servers. The computer device obtains at least two documents to be checked for duplicates; for each document to be checked for duplicates, the text vector corresponding to the document text in the document to be checked for duplicates is determined, and the image set corresponding to the document image in the document to be checked for duplicates is determined; any two documents to be checked for duplicates are selected as the first document and the second document respectively; according to the content similarity of the document content of the first document and the document content of the second document under different content categories; according to the content similarity under different content categories, the duplication between the first document and the second document is determined. Among them, the computer device can be either a terminal or a server. The terminal 102 can be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, Internet of Things devices and portable wearable devices. The Internet of Things devices can be smart speakers, smart TVs, smart air conditioners, smart car-mounted devices, etc. Portable wearable devices can be smart watches, smart bracelets, head-mounted devices, etc. The server 104 may be implemented as an independent server or a server cluster consisting of multiple servers.

[0064] In an exemplary embodiment, as shown in FIG2 , a document duplication checking method is provided, which is described by taking the method applied to the computer device in FIG1 as an example, and includes the following steps 201 to 205. Among them:

[0065] Step 201, obtaining at least two documents to be checked for duplicates.

[0066] Exemplarily, the bidding documents uploaded by the bidding user are obtained from the file system, and the PDF and DOC files are converted into DOCX files through the Spire.office plug-in to facilitate the program to perform the duplicate checking algorithm.

[0067] Step 202: for each document to be checked for duplicates, determine the text vector corresponding to the document text in the document to be checked for duplicates, and determine the image set corresponding to the document image in the document to be checked for duplicates.

[0068] Exemplarily, determining the text vector corresponding to the document text in the document to be checked for duplicates includes: obtaining the document text in the document to be checked for duplicates; performing word segmentation on the document text to obtain a vocabulary set of the document text; and determining the text vector of the document text based on the vocabulary vectors of each text vocabulary in the vocabulary set.

[0069] Exemplarily, determining an image set corresponding to a document image in a document to be checked for duplicates includes: obtaining a document image in the document to be checked for duplicates; for each document image, performing text detection on the document image to obtain an image text of the document image; the image text includes at least one image vocabulary; and generating an image set of the document image based on the image vocabulary of each document image.

[0070] Step 203: select any two documents to be checked for duplicates as the first document and the second document respectively.

[0071] Step 204: Determine the content similarity between the first document and the second document in different content categories.

[0072] Exemplarily, the text similarity between the text vector of the first document and the text vector of the second document is determined, and the image similarity between the image set of the first document and the image set of the second document is determined, and the text similarity and the image similarity are used as the content similarity between the document content of the first document and the document content of the second document under different content categories.

[0073] Specifically, determining the text similarity between the text vector of the first document and the text vector of the second document includes: determining the cosine similarity between the text vector of the first document and the text vector of the second document; and using the cosine similarity as the text similarity.

[0074] Specifically, determining the image similarity between the image set of the first document and the image set of the second document includes: determining a first number of identical image words in the image set of the first document and the image set of the second document; determining a second number of all image words in the image set of the first document and the image set of the second document; and determining the image similarity based on the first number and the second number.

[0075] Step 205: Determine the degree of repetition between the first document and the second document according to the content similarities under different content categories.

[0076] Specifically, when the text similarity is not less than the text threshold, and the image similarity is not less than the image threshold, it is determined that there is duplication between the first document and the second document. When the text similarity is less than the text threshold, and / or the image similarity is less than the image threshold, it is determined that there is no duplication between the first document and the second document.

[0077] For example, all bidding documents (documents to be checked for duplicates) are compared in pairs, and the text similarity and image similarity of different bidding documents are recorded and stored in the database. The threshold of duplication is defined based on the results of historical bidding document training, and those with similarity exceeding the threshold are judged as duplicates. The duplication results and coefficients are displayed in the electronic bidding system in a list and graphic format according to the similarity level, providing judgment criteria for expert review.

[0078] The above document duplicate checking method obtains at least two documents to be checked for duplicates; for each document to be checked for duplicates, determines the text vector corresponding to the document text in the document to be checked for duplicates, and determines the image set corresponding to the document image in the document to be checked for duplicates; selects any two documents to be checked for duplicates as the first document and the second document respectively; determines the degree of duplication between the first document and the second document according to the content similarity between the document content of the first document and the document content of the second document under different content categories; and determines the degree of duplication between the first document and the second document according to the content similarity under different content categories. This embodiment can improve the efficiency of document comparison, and improve the efficiency and accuracy of document review.

[0079] It should be noted that before determining the text vector of the document text and the image set of the document image of the document to be checked for duplicates, the document text and the document image of the document need to be extracted from the document to be checked for duplicates. For example, the python-docx library of python can read the title, paragraph and format in the bidding document from top to bottom, separate the text content and image content of each paragraph in the bidding document, and form the text part W(T) and the image part W(P) of the current bidding document.

[0080] In one embodiment, determining the text vector corresponding to the document text in the document to be checked for duplicates includes: obtaining the document text in the document to be checked for duplicates; performing word segmentation processing on the document text to obtain a vocabulary set of the document text; and determining the text vector of the document text based on the vocabulary vectors of each text vocabulary in the vocabulary set.

[0081] Specifically, the titles of the document texts in the documents to be checked for duplicates are removed; the document texts are segmented to obtain a vocabulary set of the document texts; stop words in the vocabulary set are removed to obtain a vocabulary set without stop words; word frequency statistics are performed on the vocabulary set without stop words to obtain a word frequency set; the word frequency set is vectorized to determine the text vector of the document text.

[0082] Exemplarily, the title of the document text in the document to be checked for duplicates is removed, including: since the title content format of the bidding document is relatively fixed, it needs to be removed from the text content to reduce the impact on the accuracy of the duplicate checking algorithm, and obtain the untitled text W (T1).

[0083] Exemplarily, perform word segmentation on the document text to obtain the vocabulary set of the document text, including: using the pre-trained word segmentation model of the jieba library, splitting the text W(T1) into individual words or word units to form the vocabulary set W(L1). Here, to improve the accuracy, some common words in bidding are imported into the dictionary of the jieba library when pre-training the model.

[0084] Exemplarily, remove the stop words from the vocabulary set to obtain the vocabulary set after removing stop words, including: removing meaningless words in the vocabulary set W(L1), such as "de", "shi", punctuation marks, etc., to generate the cleaned vocabulary set W(L2).

[0085] Exemplarily, perform word frequency statistics on the vocabulary set after removing stop words to obtain the word frequency set, including: counting the frequency of each word in the vocabulary set W(L2) to generate the word frequency table W(L3).

[0086] Exemplarily, perform vectorization processing on the word frequency set to determine the text vector of the document text, including: using the TF-IDF (Term Frequency-Inverse Document Frequency) algorithm to calculate the weight value of a word in the document as the word frequency vector of the word in the current document. Calculate the vector values of all the words in the word frequency table W(L3) to form the text vector A. Among them, TF represents the number of times a word appears in a document, which is determined by the following formula (1-1); IDF represents the rarity of a word in the document set, which is determined by the following formula (1-2); multiply TF and IDF to get the TF-IDF value, that is, ; calculate the TF-IDF value of each word t in the word frequency table W(L3) as the vector to form the vector of the current text document .

[0087] (1-1)

[0088] where t is a specific word, d is the document text, n is the number of times t appears in the document text d, and N is the total number of words in the document text d.

[0089] (1-2)

[0090] where D is the total number of documents to be checked in the document set to be checked, is the number of documents to be checked containing the word t.

[0091] In this embodiment, obtaining the document text in the document to be checked; performing word segmentation on the document text to obtain the vocabulary set of the document text; determining the text vector of the document text according to the word vectors of the text words in the vocabulary set can more accurately determine the text vector of the document text.

[0092] In one embodiment, determining an image set corresponding to a document image in a document to be checked for duplicates includes: obtaining a document image in the document to be checked for duplicates; for each document image, performing text detection on the document image to obtain an image text of the document image; the image text includes at least one image vocabulary; and generating an image set of the document image based on the image vocabulary of each document image.

[0093] Specifically, the document images in the document to be checked for duplicates are obtained; for each document image, the document image is grayed to obtain a gray image; the gray image is binarized to obtain a binarized image; the binarized image is denoised and contrast enhanced to obtain an enhanced image; text detection is performed on the enhanced image to obtain the image text of the document image; stop words in the image vocabulary corresponding to the image text are removed to obtain the image vocabulary with the stop words removed; and an image set of the document image is generated according to the image vocabulary of each document image.

[0094] Exemplarily, grayscale processing is performed on the document image to obtain a grayscale image, including: grayscale processing is performed on each picture in the document image W(P) in the document to be checked for duplicates, and the color image is converted into a grayscale image W(P1), thereby reducing computational complexity and storage space.

[0095] Exemplarily, binarization is performed on the grayscale image to obtain a binary image, including: binarizing the processed grayscale image W(P1), ie, converting it into a black and white image W(P2), so that the characters and the background are more distinct.

[0096] Exemplarily, the binary image is subjected to denoising and contrast enhancement processing to obtain an enhanced image, including: further denoising the black and white image W(P2) to remove spots, lines, etc. in the image to maintain the clarity of the text, and contrast enhancing the image to obtain a clearer image W(P3).

[0097] For example, text detection is performed on the enhanced image to obtain the image text of the document image, including: introducing the processed image W(P3) into the pre-trained OCR model to detect and recognize various types of text and to segment and correct the text, and finally obtaining the recognized content W(PT) of the image. At the same time, the recognition accuracy can be continuously improved through continuous learning and optimization of the AI ​​model.

[0098] Exemplarily, the stop words in the image vocabulary corresponding to the image text are removed to obtain the image vocabulary without stop words, including: the pictures in the bidding documents are mostly qualifications and certificate types, such as name, certificate number and other labels are meaningless text. Through the preset stop word library, the label text in the W(PT) text content is removed, and after deduplication, the vocabulary set PT of all image texts in the current document is obtained.

[0099] In this embodiment, document images in the document to be checked for duplicates are obtained; for each document image, text detection is performed on the document image to obtain image text of the document image; the image text includes at least one image vocabulary; based on the image vocabulary of each document image, an image set of the document image is generated, which can more accurately determine the image set of the document image.

[0100] In one embodiment, the content similarity under different content categories includes text similarity under text category and image similarity under image category; accordingly, based on the content similarity between the document content of the first document and the document content of the second document under different content categories, it includes: determining the text similarity between the text vector of the first document and the text vector of the second document; and determining the image similarity between the image set of the first document and the image set of the second document.

[0101] For example, the cosine similarity algorithm calculates the similarity of the text content between two different documents W1 and W2, which can be determined by the following formula (1-1):

[0102] (1-3)

[0103] Among them, a i , b i is each document vector , The element vector in is the TF-IDF value of each word t in the word frequency table W(L3). The cosine value cos(θ) obtained at the end is the text similarity between the first document and the second document, and the value range is [-1,1], where -1 means completely dissimilar and 1 means completely similar.

[0104] In this embodiment, the text similarity between the text vector of the first document and the text vector of the second document is determined; and the image similarity between the image set of the first document and the image set of the second document is determined, so that the text similarity and image similarity can be determined more accurately.

[0105] In one embodiment, determining the image similarity between an image set of a first document and an image set of a second document includes: extracting image vocabulary in the image set of the first document and image vocabulary in the image set of the second document; and determining the image similarity between the image set of the first document and the image set of the second document based on the vocabulary distribution of each image vocabulary in the image set of the first document and in the image set of the first document.

[0106] In one embodiment, the image similarity between the image set of the first document and the image set of the second document is determined based on the vocabulary distribution of each image vocabulary in the image set of the first document and in the image set of the first document, including: determining a first number of identical image vocabulary in the image set of the first document and in the image set of the second document; determining a second number of all image vocabulary in the image set of the first document and in the image set of the second document; and determining the image similarity based on the proportion of the first number in the second number.

[0107] Exemplarily, the image similarity between the image set of the first document and the image set of the second document is determined by the following formula (1-4).

[0108] (1-4)

[0109] Among them, A and B refer to the text vocabulary set PT of the first document W1 and the second document W2 w1 PT w2 , |A∩B| represents the number of elements in the intersection of two sets, and |A∪B| represents the number of elements in the union of two sets. The final value is the image similarity between the two documents to be checked for duplicates.

[0110] In this embodiment, image vocabulary in the image set of the first document and image vocabulary in the image set of the second document are extracted; based on the vocabulary distribution of each image vocabulary in the image set of the first document and in the image set of the first document, the image similarity between the image set of the first document and the image set of the second document is determined, so that the image similarity can be determined more accurately.

[0111] In one embodiment, this embodiment provides an optional method for checking for duplicate documents, and takes the method applied to a server as an example for explanation. As shown in FIG3 , the method includes the following steps:

[0112] Step 301, obtaining at least two documents to be checked for duplicates.

[0113] Step 302: for each document to be checked for duplicates, determine the text vector corresponding to the document text in the document to be checked for duplicates, and determine the image set corresponding to the document image in the document to be checked for duplicates.

[0114] Step 303, for each document to be checked for duplicates, obtain the document image in the document to be checked for duplicates; for each document image, perform text detection on the document image to obtain the image text of the document image; the image text includes at least one image vocabulary; and generate an image set of the document image based on the image vocabulary of each document image.

[0115] Step 304: select any two different documents to be checked for duplicates as the first document and the second document respectively.

[0116] Step 305 , determining the text similarity between the text vector of the first document and the text vector of the second document, and determining the image similarity between the image set of the first document and the image set of the second document.

[0117] Step 306: When the text similarity is not less than the text threshold and the image similarity is not less than the image threshold, it is determined that there is duplication between the first document and the second document.

[0118] This embodiment can check whether the bidding documents contain original content, avoiding violating copyright laws or bidding rules by plagiarizing other people's works. Through technical means, it ensures fair competition in the content of all bidding documents and prevents unfair behaviors such as bid rigging and collusion. The automated duplicate checking process saves time for manual comparison and improves the efficiency of the entire bidding document review. It reduces legal risks and reputation risks caused by content duplication or plagiarism, and protects the interests of bidders and tenderers. The application of duplicate checking tools makes the bidding process more transparent, and all bidding documents are tested according to unified standards.

[0119] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0120] Based on the same inventive concept, the embodiment of the present application also provides a document duplicate checking device for implementing the document duplicate checking method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more document duplicate checking device embodiments provided below can refer to the limitations of the document duplicate checking method above, and will not be repeated here.

[0121] In an exemplary embodiment, as shown in FIG4 , a document duplicate checking device is provided, comprising: a document acquisition module 10, a document separation module 11, a document selection module 12, a similarity determination module 13 and a duplication determination module 14, wherein:

[0122] A document acquisition module 10 is used to acquire at least two documents to be checked for duplicates;

[0123] The document separation module 11 is used to determine, for each document to be checked for duplicates, a text vector corresponding to the document text in the document to be checked for duplicates, and to determine an image set corresponding to the document image in the document to be checked for duplicates;

[0124] A document selection module 12, used for selecting any two documents to be checked for duplicates as the first document and the second document respectively;

[0125] A similarity determination module 13, configured to determine content similarities between the document content of the first document and the document content of the second document under different content categories;

[0126] The duplication determination module 14 is used to determine the duplication between the first document and the second document according to the content similarity under different content categories.

[0127] In one embodiment, the similarity determination module 13 in FIG. 4 is further used to determine the text similarity between the text vector of the first document and the text vector of the second document; and to determine the image similarity between the image set of the first document and the image set of the second document.

[0128] In one embodiment, the similarity determination module 13 in Figure 4 is also used to extract image vocabulary in the image set of the first document and image vocabulary in the image set of the second document; based on the vocabulary distribution of each image vocabulary in the image set of the first document and in the image set of the first document, determine the image similarity between the image set of the first document and the image set of the second document.

[0129] In one embodiment, the similarity determination module 13 in Figure 4 is also used to determine a first number of identical image words in the image set of the first document and the image set of the second document; determine a second number of all image words in the image set of the first document and the image set of the second document; and determine the image similarity based on the proportion of the first number in the second number.

[0130] In one embodiment, the document separation module 11 in Figure 4 is also used to obtain document images in the document to be checked for duplicates; for each document image, text detection is performed on the document image to obtain the image text of the document image; the image text includes at least one image vocabulary; and based on the image vocabulary of each document image, an image set of the document image is generated.

[0131] In one embodiment, the document separation module 11 in Figure 4 is also used to obtain the document text in the document to be checked for duplicates; perform word segmentation on the document text to obtain a vocabulary set of the document text; and determine the text vector of the document text based on the vocabulary vector of each text vocabulary in the vocabulary set.

[0132] Each module in the above-mentioned document duplicate checking device can be implemented in whole or in part by software, hardware and their combination. Each module can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the corresponding operations of each module above.

[0133] In an exemplary embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be shown in FIG5. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. The processor, the memory and the input / output interface are connected via a system bus, and the communication interface is connected to the system bus via the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, a document duplication detection method is implemented.

[0134] In an exemplary embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be shown in FIG6. The computer device includes a processor, a memory, an input / output interface, a communication interface, a display unit, and an input device. The processor, the memory, and the input / output interface are connected via a system bus, and the communication interface, the display unit, and the input device are connected to the system bus via the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner may be implemented through WIFI, a mobile cellular network, NFC (near field communication) or other technologies. When the computer program is executed by the processor, a document duplication checking method is implemented. The display unit of the computer device is used to form a visually visible picture, which may be a display screen, a projection device, or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the computer device shell, or an external keyboard, touchpad or mouse.

[0135] Those skilled in the art will understand that the structures shown in FIGS. 5 and 6 are merely block diagrams of partial structures related to the scheme of the present application, and do not constitute a limitation on the computer device to which the scheme of the present application is applied. The specific computer device may include more or fewer components than those shown in the figures, or combine certain components, or have a different arrangement of components.

[0136] In one embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above method embodiments when executing the computer program.

[0137] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0138] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0139] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0140] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0141] The technical features of the above embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0142] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A document duplication checking method, characterized in that: The method comprises: Obtain at least two documents to be checked for duplicates; For each document to be checked for duplicates, determining a text vector corresponding to the document text in the document to be checked for duplicates, and determining an image set corresponding to the document image in the document to be checked for duplicates; Select any two documents to be checked for duplicates as the first document and the second document respectively; content similarity between the first document and the second document in different content categories; The degree of repetition between the first document and the second document is determined according to the content similarities under different content categories.

2. The method according to claim 1, characterized in that: in, The content similarities under different content categories include text similarity under a text category and image similarity under an image category; correspondingly, the content similarities under different content categories based on the document content of the first document and the document content of the second document include: determining a text similarity between the text vector of the first document and the text vector of the second document; and, The image similarity between the image set of the first document and the image set of the second document is determined.

3. The method according to claim 2, characterized in that The determining of the image similarity between the image set of the first document and the image set of the second document comprises: extracting image vocabulary in the image set of the first document and image vocabulary in the image set of the second document; The image similarity between the image set of the first document and the image set of the second document is determined according to the distribution of each image word in the image set of the first document and the words in the image set of the first document.

4. The method according to claim 3, characterized in that: The determining, based on the image collection of the first document and the vocabulary distribution in the image collection of the first document, the image similarity between the image collection of the first document and the image collection of the second document comprises: determining a first number of identical image words in the image collection of the first document and in the image collection of the second document; determining a second number of all image words in the image collection of the first document and the image collection of the second document; The image similarity is determined according to a proportion of the first number in the second number.

5. The method according to any one of claims 1 to 4, characterized in that: The step of determining an image set corresponding to the document image in the document to be checked for duplicates includes: Acquire a document image from the document to be checked for duplicates; For each document image, performing text detection on the document image to obtain an image text of the document image; the image text includes at least one image word; An image set of the document image is generated based on the image vocabulary of each document image.

6. The method according to claim 1, characterized in that The determining of the text vector corresponding to the document text in the document to be checked for duplicates includes: Obtaining the document text in the document to be checked for duplicates; Performing word segmentation processing on the document text to obtain a vocabulary set of the document text; The text vector of the document text is determined according to the vocabulary vector of each text vocabulary in the vocabulary set.

7. A document duplication checking device, characterized in that: The device comprises: A document acquisition module is used to acquire at least two documents to be checked for duplicates; A document separation module, for determining, for each document to be checked for duplicates, a text vector corresponding to a document text in the document to be checked for duplicates, and an image set corresponding to a document image in the document to be checked for duplicates; A document selection module, used to select any two documents to be checked for duplicates as the first document and the second document respectively; A similarity determination module, configured to determine content similarities between the document content of the first document and the document content of the second document under different content categories; The duplication determination module is used to determine the duplication between the first document and the second document according to the content similarity under different content categories.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images