Method, apparatus, electronic device, and storage medium for repetitive identification of documents

Abstract

This application relates to the field of artificial intelligence technology, and discloses a method, device, electronic device, and storage medium for identifying document repetition. Extract at least two target words of the target document, and construct a target word sequence of the target document according to the at least two target words; wherein, the target words at least include nouns, verbs, and quantifiers in the target document; determine the one-hot encoding vector of each character in the target word sequence; determine the feature vector of the target document according to the one-hot encoding vector of each character in the target word sequence; determine the repetition of the target document according to the distance between the feature vector of other documents and the feature vector of the target document. The above technical solution improves the accuracy of document query on the premise of balancing the time complexity and space complexity of document duplicate checking, and provides a new idea for identifying document repetition.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of artificial intelligence technology, in particular to the field of natural language processing technology, and specifically relate to a method, device, electronic device and storage medium for identifying the repeatability of documents. Background Art

[0002] With the development of Internet technology, various documents in all walks of life can be obtained from the Internet. For example, financial institutions access a large number of financial documents from the Internet every day, including market express, financial information, research reports, policy interpretations, announcements, etc. Many documents from different data sources are the same or similar. If duplicate filtering is not carried out, a large number of duplicate documents or similar documents will flood in, greatly affecting the accurate transmission of information and work efficiency. Therefore, it is particularly important to accurately identify duplicate documents in a large number of documents while taking into account the execution efficiency and accuracy. Summary of the Invention

[0003] The present application provides a method, device, electronic device and storage medium for identifying the repeatability of documents to improve the accuracy of document duplicate checking.

[0004] In a first aspect, the embodiments of the present application provide a method for identifying the repeatability of documents, including:

[0005] Extracting at least two target words of a target document, and constructing a target word sequence of the target document according to the at least two target words; wherein the target words at least include nouns, verbs and numeral-classifiers in the target document;

[0006] Determining the one-hot encoding vector of each character in the target word sequence;

[0007] Determining the feature vector of the target document according to the one-hot encoding vector of each character in the target word sequence;

[0008] Determining the repeatability of the target document according to the distance between the feature vectors of other documents and the feature vector of the target document.

[0009] In a second aspect, the embodiments of the present application further provide a device for identifying the repeatability of documents, including:

[0010] A target sequence construction module, configured to extract at least two target words of a target document, and construct a target word sequence of the target document according to the at least two target words; wherein the target words at least include nouns, verbs and numeral-classifiers in the target document;

[0011] A one-hot vector determination module, configured to determine the one-hot encoding vector of each character in the target word sequence;

[0012] A feature vector determination module, configured to determine a feature vector of the target document according to the one-hot encoded vectors of each character in the target word sequence;

[0013] A repeatability determination module, configured to determine the repeatability of the target document according to the distance between the feature vectors of other documents and the target feature vector of the target document.

[0014] Thirdly, an embodiment of the present application further provides an electronic device, including:

[0015] One or more processors;

[0016] A memory, configured to store one or more programs;

[0017] When the one or more programs are executed by the one or more processors, the one or more processors implement the document repeatability recognition method provided in any embodiment of the present application.

[0018] Fourthly, an embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored, characterized in that when the program is executed by a processor, it implements the document repeatability recognition method provided in any embodiment of the present application.

[0019] The technical solution of the present application extracts at least two target words of the target document, and constructs a target word sequence of the target document according to the at least two target words; wherein, the target words at least include nouns, verbs and quantifiers in the target document, and determine the one-hot encoded vectors of each character in the target word sequence, and then determine the feature vector of the target document according to the one-hot encoded vectors of each character in the target word sequence, and further determine the repeatability of the target document according to the distance between the feature vectors of other documents and the feature vector of the target document. The above technical solution introduces one-hot encoding, solves the problem of inconsistent character set encoding, reduces the space distribution, and at the same time solves the problem of homophone substitution, thereby improving the accuracy of document query on the premise of balancing the time complexity and space complexity of document duplicate checking, and provides a new idea for document repeatability recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 is a flowchart of a document repeatability recognition method provided in Embodiment 1 of the present application;

[0021] Figure 2 is a flowchart of a document repeatability recognition method provided in Embodiment 2 of the present application;

[0022] Figure 3 is a flowchart of a document repeatability recognition method provided in Embodiment 3 of the present application;

[0023] Figure 4 It is a schematic structural diagram of a document repeatability recognition device provided in Embodiment 4 of the present application;

[0024] Figure 5 It is a schematic structural diagram of an electronic device provided in Embodiment 5 of the present application. Detailed implementation manners

[0025] The present application will be further described in detail below with reference to the drawings and embodiments. It can be understood that the specific embodiments described herein are only used to explain the present application, rather than limiting the present application. In addition, it should be noted that for the convenience of description, only parts related to the present application rather than all structures are shown in the drawings.

[0026] Embodiment 1

[0027] Figure 1 It is a flowchart of a document repeatability recognition method provided in Embodiment 1 of the present application; this embodiment is applicable to the situation of document repeatability recognition, especially applicable to the situation of financial document repeatability recognition. This method can be executed by a document repeatability recognition device, which is implemented by software / hardware and can be integrated into an electronic device with the function of document repeatability recognition, such as a server.

[0028] As Figure 1 shown, this method may specifically include:

[0029] S110. Extract at least two target words from the target document, and construct a target word sequence of the target document according to the at least two target words.

[0030] Among them, the target document refers to the document that needs to be recognized for repeatability; the target word refers to the word in the target document that can express the main meaning of the document, and the target words at least include nouns, verbs and numeral-classifiers in the target document; the target word sequence refers to the sequence composed of target words, which is used to represent the target document.

[0031] In this embodiment, based on the word segmentation technology, the target document can be segmented, and according to the segmentation result, the words without actual meaning (such as modal particles, adverbs and adjectives, etc.) and punctuation marks in the target document can be filtered out, so as to obtain at least two target words, and based on the order of the target words in the target document, the target words can be concatenated in sequence to construct the target word sequence of the target document.

[0032] S120. Determine the one-hot encoding vector of each character in the target word sequence.

[0033] In this embodiment, for each character in the target word sequence, based on the set character sequence, one-hot encoding is performed on the target character associated with the character to obtain the one-hot encoding vector of the character.

[0034] Among them, the one-hot encoding, that is, the One-Hot encoding, also known as one-hot effective encoding, mainly uses an N-bit status register to encode N states. Each state has its own independent register bit, and only one bit is valid at any given time. Suppose the character sequence includes initials and digits, and further includes single vowels; for example, the supposed character sequence can be composed of 6 single vowels, 23 initials, and 10 digits, a total of 39 characters, and can be sorted in the order of single vowels first, then initials, and then digits (such as: a, o, e,..., b, p, m,..., z, c, s, r, zh, ch, sh, y, w, 0, 1,..., 9); further, the one-hot encoding vector after encoding based on the supposed character sequence is also 39-dimensional. The target character associated with the character can be the initial of the pinyin of the character. Further, for a character with a single vowel, the single vowel of the character is taken as the target character; in addition, when the character is a digit, the target character can be a numerical value, such as 9.

[0035] Specifically, for each character in the target word sequence, obtain the target character associated with the character, and based on the supposed character sequence, perform one-hot encoding on the target character associated with the character to obtain the one-hot encoding vector of the character. For example, perform one-hot encoding on "9". The target character associated with "9" is "9" respectively. After performing one-hot encoding on "9", the one-hot encoding vector of "9" is "0000…000000001" (38 zeros and 1 one).

[0036] S130. Determine the feature vector of the target document according to the one-hot encoding vector of each character in the target word sequence.

[0037] Among them, the feature vector of the target document refers to the vector used to represent the features of the target document.

[0038] In this embodiment, the one-hot encoding vectors of each character in the target word sequence can be concatenated in sequence as the feature vector of the target document. Optionally, each character can also be concatenated by rows. For example, if there are m (m>1) characters in the target word sequence of the target document, then the finally concatenated feature vector of the target document is an m*39-dimensional matrix.

[0039] S140. Determine the repeatability of the target document according to the distance between the feature vector of another document and the feature vector of the target document.

[0040] Among them, another document refers to the document that needs to be judged for duplication with the target document, and the number of them can be one or multiple; the feature vectors of another document are also calculated and stored according to the steps such as S110-S130.

[0041] In this embodiment, based on deep learning technology, the feature vectors of other documents and the feature vector of the target document can be input into a deep learning model to determine the repeatability of the target document.

[0042] Optionally, the distance between the feature vector of other documents and the feature vector of the target document can also be calculated. If the distance is less than a set threshold, the target document is determined to be a duplicate document. The set threshold is set by those skilled in the art according to the actual situation.

[0043] Optionally, the feature vectors of other documents can be concatenated into numbers to obtain the feature signature of other documents; the feature vectors of the target document can be concatenated into numbers to obtain the feature signature of the target document; the repeatability of the target document is determined according to the distance between the feature signature of other documents and the feature signature of the target document. The feature signature is used to characterize the features of the document. Specifically, the feature vectors of other documents are sequentially concatenated into a string of numbers as the feature signature of other documents; similarly, the feature vectors of the target document are sequentially concatenated into a string of numbers as the feature signature of the target document; the Hamming distance between the feature signature of other documents and the feature signature of the target document is calculated. If the Hamming distance is less than the set distance threshold, the target document is determined to be a duplicate document. The set distance threshold is determined by those skilled in the art through a large number of data experiments.

[0044] The technical solution of this application extracts at least two target words from the target document, and constructs a target word sequence of the target document according to the at least two target words. The target words at least include nouns, verbs, and numeral-classifiers in the target document, and the one-hot encoding vector of each character in the target word sequence is determined. Then, according to the one-hot encoding vector of each character in the target word sequence, the feature vector of the target document is determined. Furthermore, according to the distance between the feature vector of other documents and the feature vector of the target document, the repeatability of the target document is determined. The above technical solution introduces one-hot encoding, solves the problem of inconsistent character set encoding, reduces the spatial distribution, and at the same time solves the problem of homophone substitution. Furthermore, on the premise of balancing the time complexity and space complexity of document duplicate checking, the accuracy of document query is improved, providing a new idea for the recognition of document repeatability.

[0045] Based on the above technical solutions, in order to more accurately determine the repeatability of the target document, as an alternative way in this embodiment, the target document can be subjected to format cleaning. Specifically, extra information such as HyperText Markup Language (html) tags, Cascading Style Sheets (CSS) formats, pictures, and copyright information of the target document is removed. It can be understood that by performing format cleaning on the target document, interference information is filtered out, and only text information is retained, providing a guarantee for subsequent repeatability recognition of the document.

[0046] Embodiment 2

[0047] Figure 2 It is a flowchart of a method for recognizing the repeatability of a document provided in Embodiment 2 of the present application; based on the above embodiment, "determining the feature vector of the target document according to the one-hot encoding vector of each character in the target word sequence" is further optimized to provide an alternative way.

[0048] As Figure 2 shown, the method may specifically include:

[0049] S210. Extract at least two target words from the target document, and construct a target word sequence of the target document according to the at least two target words.

[0050] S220. Determine the one-hot encoding vector of each character in the target word sequence.

[0051] S230. Traverse the target word sequence based on a set sliding window to obtain at least two word segments.

[0052] In this embodiment, the set sliding window is set by those skilled in the art according to the actual situation.

[0053] In this embodiment, by traversing the target word sequence based on the set sliding window, at least two word segments can be obtained. For example, if the target word sequence has m (m>1) characters and the set sliding window is 2, then m-1 word segments can be obtained, and each word segment contains 2 characters.

[0054] S240. Determine the word feature vectors of at least two word segments according to the one-hot encoding vectors of each character in the target word sequence.

[0055] In this embodiment, for each word segment, the occurrence frequency of each character in the word segment in the target document is determined. Then, based on the occurrence frequency of each character in the word segment in the target document and the one-hot encoding vector of each character in the word segment, the word feature vector of the word segment is determined. Specifically, for each character in the word segment, the occurrence frequency of each character in the word segment in the target document can be multiplied by the one-hot encoding vector of each character in the word segment bit by bit to obtain the word feature vector of the word segment.

[0056] Optionally, the one-hot encoding vector of each character in the word segment can also be mapped and transformed to obtain the character feature vector of each character in the word segment. For example, 0 in the one-hot encoding vector of each character in the word segment is mapped to -1 to obtain the character feature vector of each character in the word segment.

[0057] Then, based on the character feature vector of each character in the word segment and the occurrence frequency of each character in the word segment in the target document, the word feature vector of the word segment is obtained. Specifically, for each character in the word segment, the character feature vector of each character in the word segment and the occurrence frequency of each character in the word segment in the target document can be multiplied by bit to obtain the word feature vector of the word segment.

[0058] S250. Determine the feature vector of the target document according to the word feature vectors of at least two word segments.

[0059] In this embodiment, the word feature vectors of at least two word segments are concatenated in sequence as the feature vector of the target document.

[0060] Optionally, the word feature vectors of at least two word segments can also be concatenated by rows to obtain the intermediate vector of the target document. Then, the intermediate vector of the target document is summed by columns to obtain the row vector of the target document. The row vector is mapped and transformed bit by bit to obtain the feature vector of the target document.

[0061] For example, if the target word sequence has m (m>1) characters and the sliding window is set to 2, then m-1 word segments can be obtained, and each word segment has 2 characters. The character feature vector of each character is denoted as W, which is a 39-dimensional vector. Then, the feature vectors of the m-1 word segments are respectively (W1, W2), (W2, W3), (W3, W4)... (Wm-1, Wm). The vector feature of each word segment is 78-dimensional. The feature vectors of the m-1 word segments are concatenated by rows to obtain a (m-1)*78 matrix, that is, the intermediate vector of the target document. The matrix is summed by columns to obtain a 78-dimensional row vector. This row vector is mapped and transformed bit by bit, that is, positive numbers are mapped to 1 and negative numbers are mapped to 0, to obtain the feature vector of the target document.

[0062] S260. Determine the repeatability of the target document based on the distance between the feature vectors of other documents and the feature vector of the target document.

[0063] In the technical solution of this embodiment, based on a set sliding window, traverse the target word sequence to obtain at least two word segments. Then, according to the one-hot encoding vector of each character in the target word sequence, determine the word feature vectors of at least two word segments, and further determine the feature vector of the target document based on the word feature vectors of at least two word segments. The above technical solution introduces word feature vectors based on a set sliding window, so that the features of the extracted target document retain certain word order features, improving the accuracy of the features of the target document, and thus improving the accuracy of document duplicate checking.

[0064] Embodiment III

[0065] Figure 3 It is a flowchart of a method for identifying the repeatability of a document provided in Embodiment III of this application; on the basis of the above embodiment, a global signature and a local signature are added to further optimize the method for identifying the repeatability of a document.

[0066] As Figure 3 shown, the method may specifically include:

[0067] S310. Determine the global signature and local signature of the target document.

[0068] In this embodiment, the global signature is used to represent the overall features of the target document; the local signature is used to represent the significant local features of the target document.

[0069] Optionally, to determine the global signature of the target document, a hash operation may be performed on the text information in the target document to obtain the global signature of the target document. Specifically, the text information in the target document may be hashed using the Secure Hash Algorithm (SHA) to obtain the global signature of the target document. Among them, SHA is a data encryption algorithm. The idea of this algorithm is to receive a piece of plaintext and then convert it into a piece of ciphertext in an irreversible way; that is, the process of inputting a string of binary codes and converting them into an output sequence with a shorter length and a fixed number of bits, namely the hash value, also known as the message digest or message authentication code. SHA is irreversible, collision-resistant, and has a good avalanche effect.

[0070] Optionally, determining the local signature of the target document can be determining at least two keywords of the target document; wherein, the keywords are used to characterize the core content of the target document and at least include nouns and verbs. Specifically, based on the word segmentation technology, the target document can be segmented to obtain the core words (verbs or nouns) of the target document, and the obtained core words can be used as the keywords of the target document.

[0071] Furthermore, perform a hashing operation on at least two keywords to obtain the local signature of the target document. Specifically, the SHA algorithm can be used to perform a hashing operation on at least two keywords, and the result of the hashing operation can be used as the local signature of the target document.

[0072] Exemplarily, performing a hashing operation on at least two keywords to obtain the local signature of the target document can also be, according to the order of appearance of at least two keywords in the target document, concatenating at least two keywords to obtain the keyword sequence of the target document; performing a hashing operation on the keyword sequence to obtain the local signature of the target document.

[0073] Specifically, according to the order of appearance of at least two keywords in the target document, concatenate at least two keywords into sentences to obtain at least one keyword sequence of the target document; according to the length of the keyword sequence, select a set number of keyword sequences in descending order; furthermore, the SHA algorithm can be used to perform a hashing operation on the set number of keyword sequences, and the result of the hashing operation can be used as the local signature of the target document.

[0074] S320. Use the documents in the document set that have different global signatures from the target document but the same local signature as the other documents of the target document.

[0075] In this embodiment, use the documents in the document set that have different global signatures from the target document but the same local signature as the other documents of the target document.

[0076] It should be noted that the global signatures and local signatures of the documents in the document set are also determined according to the method in S310.

[0077] In addition, an inverted index can also be made for the global signatures and local signatures of each document in the document set to improve the retrieval efficiency. When the global signature of the target document is the same as the global signature of other documents, it is determined that the target document is a duplicate document; when the local signature of the target document is different from the local signature of other documents, it is determined that the target document is a non-duplicate document.

[0078] S330. Extract at least two target words of the target document, and construct a target word sequence of the target document according to the at least two target words.

[0079] S340. Determine the one-hot encoding vector for each character in the target word sequence.

[0080] S350. Determine the feature vector of the target document according to the one-hot encoding vector for each character in the target word sequence.

[0081] S360. Determine the repeatability of the target document according to the distance between the feature vectors of other documents and the feature vector of the target document.

[0082] The technical solution of this embodiment determines the global signature and local signature of the target document, and then uses the documents in the document set that have different global signatures but the same local signatures as the target document as other documents of the target document. By introducing the global signature and local signature to screen the documents, the efficiency of document duplicate checking is improved.

[0083] Embodiment 4

[0084] Figure 4 FIG. is a schematic structural diagram of a device for identifying the repeatability of a document provided in Embodiment 4 of the present application; this embodiment is applicable to the situation of document repeatability identification, especially applicable to the situation of financial document repeatability identification. The device is implemented by software / hardware and can be integrated into an electronic device with the function of document repeatability identification, such as a server.

[0085] As Figure 4 shown, the device includes a target sequence construction module 410, a one-hot vector determination module 420, a feature vector determination module 430, and a repeatability determination module 440, where

[0086] The target sequence construction module 410 is configured to extract at least two target words of the target document and construct a target word sequence of the target document according to the at least two target words; where the target words include at least nouns, verbs, and quantifiers in the target document.

[0087] The one-hot vector determination module 420 is configured to determine the one-hot encoding vector for each character in the target word sequence.

[0088] The feature vector determination module 430 is configured to determine the feature vector of the target document according to the one-hot encoding vector for each character in the target word sequence.

[0089] The repeatability determination module 440 is configured to determine the repeatability of the target document according to the distance between the feature vectors of other documents and the target feature vector of the target document.

[0090] The technical solution of this application extracts at least two target words from the target document, and constructs a target word sequence of the target document according to the at least two target words; wherein, the target words at least include nouns, verbs and quantifiers in the target document, and determines the one-hot encoding vector of each character in the target word sequence. Then, according to the one-hot encoding vector of each character in the target word sequence, the feature vector of the target document is determined. Furthermore, according to the distance between the feature vector of the other document and the feature vector of the target document, the repeatability of the target document is determined. The above technical solution introduces one-hot encoding, solves the problem of inconsistent character set encoding, reduces the spatial distribution, and at the same time solves the problem of homophone substitution. Furthermore, on the premise of balancing the time complexity and space complexity of document duplicate checking, the accuracy of document query is improved, providing a new idea for the recognition of document repeatability.

[0091] Further, the one-hot vector determination module 420 is specifically configured to:

[0092] For each character, based on the set character sequence, perform one-hot encoding on the target character associated with the character to obtain the one-hot encoding vector of the character; wherein, the set character sequence includes initials and numbers.

[0093] Further, the feature vector determination module 430 includes a word segment determination sub-module, a word feature vector determination sub-module and a feature vector determination sub-module, wherein,

[0094] The word segment determination sub-module is configured to traverse the target word sequence based on the set sliding window to obtain at least two word segments;

[0095] The word feature vector determination sub-module is configured to determine the word feature vectors of at least two word segments according to the one-hot encoding vectors of each character in the target word sequence;

[0096] The feature vector determination sub-module is configured to determine the feature vector of the target document according to the word feature vectors of at least two word segments.

[0097] Further, the word feature vector determination sub-module includes an occurrence frequency determination unit and a word feature vector determination unit, wherein,

[0098] The occurrence frequency determination unit is configured to, for each word segment, determine the occurrence frequency of each character in the word segment in the target document;

[0099] The word feature vector determination unit is configured to determine the word feature vector of the word segment according to the occurrence frequency of each character in the word segment in the target document and the one-hot encoding vector of each character in the word segment.

[0100] Furthermore, the word feature vector determination unit includes a character feature vector determination subunit and a word feature vector determination subunit, where

[0101] The character feature vector determination subunit is used to perform a mapping transformation on the one-hot encoding vector of each character in the word segment to obtain the character feature vector of each character in the word segment;

[0102] The word feature vector determination subunit is used to obtain the word feature vector of the word segment according to the character feature vector of each character in the word segment and the occurrence frequency of each character in the target document.

[0103] Furthermore, the feature vector determination sub-module includes an intermediate vector determination unit, a row vector determination unit, and a feature vector determination unit, where

[0104] The intermediate vector determination unit is used to splice the word feature vectors of at least two word segments row by row to obtain the intermediate vector of the target document;

[0105] The row vector determination unit is used to sum the intermediate vector of the target document column by column to obtain the row vector of the target document;

[0106] The feature vector determination unit is used to perform a mapping transformation on the row vector bit by bit to obtain the feature vector of the target document.

[0107] Furthermore, the repeatability determination module 440 includes a feature signature determination sub-module and a repeatability determination sub-module, where

[0108] The feature signature determination sub-module is used to splice the feature vectors of other documents into a number to obtain the feature signature of other documents;

[0109] The feature signature determination sub-module is also used to splice the feature vectors of the target document into a number to obtain the feature signature of the target document;

[0110] The repeatability determination sub-module is used to determine the repeatability of the target document according to the distance between the feature signature of other documents and the feature signature of the target document.

[0111] Furthermore, the device further includes a signature determination module and an other document determination module, where

[0112] The signature determination module is used to determine the global signature and the local signature of the target document;

[0113] The other document determination module is used to use the documents in the document set that have different global signatures from the target document but the same local signatures as the other documents of the target document.

[0114] Further, the signature determination module includes a keyword determination sub-module and a local signature determination sub-module, where

[0115] The keyword determination sub-module is used to determine at least two keywords of the target document; among them, the keywords include at least nouns and verbs;

[0116] The local signature determination sub-module is used to perform a hashing operation on at least two keywords to obtain the local signature of the target document.

[0117] Further, the local signature determination sub-module includes a keyword sequence determination unit and a local signature determination unit, where

[0118] The keyword sequence determination unit is used to splice at least two keywords according to the order of appearance of the at least two keywords in the target document to obtain the keyword sequence of the target document;

[0119] The local signature determination unit is used to perform a hashing operation on the keyword sequence to obtain the local signature of the target document.

[0120] Further, the signature determination module further includes a global signature determination sub-module, which is specifically used for:

[0121] Perform a hashing operation on the text information in the target document to obtain the global signature of the target document.

[0122] Further, the device further includes a cleaning module, which is specifically used for:

[0123] Perform format cleaning on the target document.

[0124] The above document duplication recognition device can execute the document duplication recognition method provided in any embodiment of the present application, and has the corresponding functional modules and beneficial effects for executing the method.

[0125] Embodiment Five

[0126] Figure 5 It is a schematic structural diagram of an electronic device provided in Embodiment Five of the present application, Figure 5 showing a block diagram of an exemplary device suitable for implementing the embodiments of the present application. Figure 5 The displayed device is only an example and should not impose any limitation on the functions and usage scope of the embodiments of the present application.

[0127] As Figure 5 shown, the electronic device 12 is presented in the form of a general-purpose computing device. The components of the electronic device 12 may include, but are not limited to: one or more processors or processing units 16, a system memory 28, and a bus 18 connecting different system components (including the system memory 28 and the processing unit 16).

[0128] Bus 18 represents one or more of several types of bus architectures, including a memory bus or memory controller, a peripheral bus, an Accelerated Graphics Port, a processor, or a local bus using any of a variety of bus architectures. By way of example, and not limitation, these architectures include Industry Standard Architecture (ISA) bus, Micro Channel Architecture (MAC) bus, Enhanced ISA bus, Video Electronics Standards Association (VESA) local bus, and Peripheral Component Interconnect (PCI) bus.

[0129] Electronic device 12 typically includes a variety of computer system readable media. These media can be any available media that can be accessed by electronic device 12, including both volatile and nonvolatile media, removable and non-removable media.

[0130] System memory 28 can include computer system readable media in the form of volatile memory, such as random access memory (RAM) 30 and / or cache memory 32. Electronic device 12 may further include other removable / non-removable, volatile / nonvolatile computer system storage media. By way of example only, storage system 34 can be used for reading and writing non-removable, nonvolatile magnetic media ( Figure 5 not shown and typically called a "hard disk drive"). Although Figure 5 not shown in the figures, a disk drive for reading and writing a removable nonvolatile disk (e.g., a "floppy disk"), and an optical disk drive for reading and writing a removable nonvolatile optical disk (e.g., a CD-ROM, DVD-ROM, or other optical media) can be provided. In these instances, each drive can be connected to bus 18 by one or more data media interfaces. System memory 28 can include at least one program product having a set (e.g., at least one) of program modules that are configured to carry out the functions of the embodiments of the present application.

[0131] A program / utility 40 having a set (at least one) of program modules 42 can be stored, for example, in system memory 28, such program modules 42 including, but not limited to, an operating system, one or more application programs, other program modules, and program data, each of which examples or some combination thereof may include an implementation of a network environment. Program modules 42 generally carry out the functions and / or methods of the embodiments described in the embodiments of the present application.

[0132] The electronic device 12 can also communicate with one or more external devices 14 (such as a keyboard, a pointing device, a display 24, etc.), and can also communicate with one or more devices that enable a user to interact with the electronic device 12, and / or communicate with any device that enables the electronic device 12 to communicate with one or more other computing devices (such as a network card, a modem, etc.). Such communication can be carried out through an input / output (I / O) interface 22. Moreover, the electronic device 12 can also communicate with one or more networks (such as a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) through a network adapter 20. As shown in the figure, the network adapter 20 communicates with other modules of the electronic device 12 through a bus 18. It should be understood that, although not shown in the figure, other hardware and / or software modules can be used in combination with the electronic device 12, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, etc.

[0133] The processing unit 16 executes various functional applications and data processing by running programs stored in the system memory 28, such as implementing the document repetitive recognition method provided by the embodiments of the present application.

[0134] Embodiment Six

[0135] Embodiment Six of the present application also provides a computer-readable storage medium, on which a computer program (or computer-executable instructions) is stored, and when the program is executed by a processor, it is used to execute the document repetitive recognition method provided by the embodiments of the present application.

[0136] The computer storage medium of the embodiments of the present application can adopt any combination of one or more computer-readable media. The computer-readable media can be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the above. More specific examples (non-exhaustive list) of the computer-readable storage medium include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this document, the computer-readable storage medium can be any tangible medium that contains or stores a program, and the program can be used by or in combination with an instruction execution system, apparatus, or device.

[0137] A computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, carrying computer-readable program code. Such a propagated data signal may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the foregoing. The computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device.

[0138] The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wire, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0139] The computer program code for performing the operations of the embodiments of the present application may be written in one or more programming languages or combinations thereof. The programming languages include object-oriented programming languages such as Java, Smalltalk, C++, and also include conventional procedural programming languages such as the "C" language or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, executed as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., by connecting through the Internet using an Internet service provider).

[0140] Note that the above is only the preferred embodiment of the present application and the applied technical principles. Those skilled in the art will understand that the present application is not limited to the specific embodiments described herein. Various obvious changes, re-adjustments, and substitutions can be made by those skilled in the art without departing from the protection scope of the present application. Therefore, although the embodiments of the present application have been described in more detail through the above embodiments, the embodiments of the present application are not limited to the above embodiments. Without departing from the concept of the present application, more other equivalent embodiments may be included, and the scope of the present application is determined by the scope of the appended claims.

Claims

1. A method for identifying the repeatability of a document, characterized in that, Including: Performing a hashing operation on the text information in the target document to obtain the global signature of the target document; Determining at least two keywords of the target document; wherein, the keywords at least include nouns and verbs; Performing a hashing operation on the at least two keywords to obtain the local signature of the target document; Regarding the documents in the document set that have a different global signature from the target document but the same local signature as the target document as other documents of the target document; Extracting at least two target words of the target document and constructing a target word sequence of the target document according to the at least two target words; wherein, the target words at least include nouns, verbs, and quantifiers in the target document; Determining the one-hot encoding vector of each character in the target word sequence; Determining the feature vector of the target document according to the one-hot encoding vector of each character in the target word sequence; Determining the repeatability of the target document according to the distance between the feature vector of other documents and the feature vector of the target document.

2. The method according to claim 1, wherein Determining the one-hot encoding vector of each character in the target word sequence includes: For each character, performing one-hot encoding on the target character associated with the character based on a set character sequence to obtain the one-hot encoding vector of the character; wherein, the set character sequence includes initials and numbers.

3. The method according to claim 1, wherein Determining the feature vector of the target document according to the one-hot encoding vector of each character in the target word sequence includes: Traversing the target word sequence based on a set sliding window to obtain at least two word segments; Determining the word feature vectors of the at least two word segments according to the one-hot encoding vector of each character in the target word sequence; Determining the feature vector of the target document according to the word feature vectors of the at least two word segments.

4. The method according to claim 3, characterized in that, Determining the word feature vectors of the at least two word segments according to the one-hot encoding vector of each character in the target word sequence includes: For each word segment, determining the occurrence frequency of each character in the target document in the word segment; Determining the word feature vector of the word segment according to the occurrence frequency of each character in the target document in the word segment and the one-hot encoding vector of each character in the word segment.

5. The method according to claim 4, characterized in that, Determining the word feature vector of the word segment according to the occurrence frequency of each character in the target document in the word segment and the one-hot encoding vector of each character in the word segment includes: Performing a mapping transformation on the one-hot encoding vector of each character in the word segment to obtain the character feature vector of each character in the word segment; Obtaining the word feature vector of the word segment according to the character feature vector of each character in the word segment and the occurrence frequency of each character in the target document in the word segment.

6. The method according to claim 3, wherein Determining the feature vector of the target document according to the word feature vectors of the at least two word segments includes: Concatenating the word feature vectors of the at least two word segments row by row to obtain the intermediate vector of the target document; Summing the intermediate vector of the target document column by column to obtain the row vector of the target document; Performing a mapping transformation on the row vector bit by bit to obtain the feature vector of the target document.

7. The method according to claim 1, characterized in that Determine the repeatability of the target document according to the distance between the feature vectors of other documents and the feature vector of the target document, including: Concatenate the feature vectors of the other documents into numbers to obtain the feature signature of the other documents; Concatenate the feature vectors of the target document into numbers to obtain the feature signature of the target document; Determine the repeatability of the target document according to the distance between the feature signature of the other document and the feature signature of the target document.

8. The method according to claim 1, characterized in that Perform a hashing operation on the at least two keywords to obtain the local signature of the target document, including: Concatenate the at least two keywords according to the order in which they appear in the target document to obtain the keyword sequence of the target document; Perform a hashing operation on the keyword sequence to obtain the local signature of the target document.

9. The method according to claim 1, wherein Before extracting at least two target words of the target document and constructing the target word sequence of the target document according to the at least two target words, it further includes: Perform format cleaning on the target document.

10. A device for recognizing the repeatability of a document, characterized in that, Include: A signature determination module, configured to perform a hashing operation on the text information in the target document to obtain the global signature of the target document; Determine at least two keywords of the target document; wherein, the keywords at least include nouns and verbs; perform a hashing operation on the at least two keywords to obtain the local signature of the target document; An other document determination module, configured to use the documents in the document set that are different from the global signature of the target document but have the same local signature as the other documents of the target document; A target sequence construction module, configured to extract at least two target words of the target document and construct a target word sequence of the target document according to the at least two target words; wherein, the target words at least include nouns, verbs and numeral-classifiers in the target document; A one-hot vector determination module, configured to determine the one-hot encoded vector of each character in the target word sequence; A feature vector determination module, configured to determine the feature vector of the target document according to the one-hot encoded vector of each character in the target word sequence; A repeatability determination module, configured to determine the repeatability of the target document according to the distance between the feature vectors of other documents and the feature vector of the target document.

11. An electronic device, characterized in that, Include: One or more processors; A memory, configured to store one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the document repeatability recognition method according to any one of claims 1-9.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the document repeatability recognition method according to any one of claims 1-9.

Images