A text deduplication method, device, electronic device and storage medium

Abstract

This invention discloses a text deduplication method, apparatus, electronic device, and storage medium. The method includes: determining representative words in the title of the text to be deduplicated; determining whether the representative words exist in the index space constructed from the titles of the deduplicated texts; if not, determining that the text to be deduplicated is not a duplicate text; if they exist, determining whether the text to be deduplicated is a duplicate text based on the title's part-of-speech tagging; when the text to be deduplicated is determined not to be a duplicate text, adding the representative words as the index and the title and part-of-speech tagging results as the keys to the index space. By obtaining the most influential representative words from the text's title for deduplication, the complexity of deduplication can be further reduced and the efficiency improved. Furthermore, based on the assumption that the most influential words are the same among similar semantically meaningful texts, when it is determined that this representative word exists in the index space constructed from the titles of the deduplicated texts, deduplication is then performed based on the title's part-of-speech tagging, thereby achieving semantically aware deduplication.

Description

Technical Field

[0001] The present invention relates to the technical field of text processing, and in particular to a text deduplication method, device, electronic device and storage medium. Background Art

[0002] At present, electronic bidding documents are becoming more and more popular. There are about millions of bidding documents published on the Internet every day. Enterprises can obtain a large amount of potential business information from these bidding documents. However, due to the existence of online reprinting and plagiarism, a large number of bidding documents crawled by enterprises have duplication problems. Storing these redundant bidding documents in the database is a waste of storage space and will also reduce the efficiency of downstream data processing tasks. Therefore, the problem of deduplication of bidding documents is a challenge for enterprises.

[0003] Traditional text deduplication methods include: 1. Jacobi similarity coefficient deduplication method, which is to determine the text repetition rate by calculating the ratio between the text intersection and the text union. This method cannot determine the semantics of the text, and each new text needs to be compared with all past texts. The required processing time increases linearly with the increase in the number of texts, so it cannot be applied to large-scale text deduplication tasks; 2. Simhash deduplication method, which was proposed by Google. It completes efficient indexing by hashing the text and completes text deduplication through Hamming distance. However, simhash is only applicable to English. In the case of Chinese, the text needs to be segmented to obtain the weight of the feature words. For long texts such as the bidding body, segmentation will incur a high computational cost. For short texts such as bidding titles, due to the small number of feature words, simhash cannot distinguish the semantics between short texts, resulting in a large number of misjudgments or omissions when facing short texts. Summary of the invention

[0004] The purpose of the present invention is to propose a text deduplication method, device, electronic device and storage medium to address the deficiencies of the above-mentioned prior art, and this purpose is achieved through the following technical solutions.

[0005] A first aspect of the present invention provides a method for deduplicating text, the method comprising:

[0006] Determine representative words in the title of the text to be deduplicated;

[0007] Determine whether the representative word exists in the index in the index space constructed by the title of the deduplicated text;

[0008] If it does not exist, it is determined that the text to be deduplicated is not a duplicate text;

[0009] If it exists, determine whether the text to be deduplicated is a duplicate text based on the title part of speech;

[0010] When it is determined that the text to be deduplicated is not a duplicate text, the representative word is used as an index, and the title and the part-of-speech tagging result of the title are used as key values ​​and added to the index space.

[0011] In some embodiments of the present application, determining representative words in the title of the text to be deduplicated includes:

[0012] The title is segmented to obtain a segmentation result; a term frequency-inverse document frequency TF-IDF value of each segmented word in the segmentation result is determined; and a segmented word corresponding to a maximum TF-IDF value is determined as a representative word.

[0013] In some embodiments of the present application, determining the term frequency-inverse text frequency TF-IDF value of each segmentation word in the segmentation result includes:

[0014] Determine the word frequency TF value of each word in the word segmentation result; obtain the inverse document frequency IDF value corresponding to each word segment from the preset vocabulary; use the TF value and IDF value of each word segment to determine the TF-IDF value of the word segment.

[0015] In some embodiments of the present application, determining whether the to-be-deduplicated text is a duplicate text based on the title part of speech includes:

[0016] Obtain an existing title represented by a key value corresponding to the representative word in the index space; compare the non-repeating words between the existing title and the title; if there are non-repeating words, determine whether the text to be deduplicated is a duplicate text based on the part of speech of the non-repeating words; if there are no non-repeating words, determine that the text to be deduplicated is a duplicate text.

[0017] In some embodiments of the present application, determining whether the to-be-deduplicated text is a duplicate text according to the part of speech of the non-duplicate words includes:

[0018] Determine whether the part of speech of the non-repeating word is a preset part of speech; if it is a preset part of speech, determine that the text to be deduplicated is not a duplicate text; if it is not a preset part of speech, determine whether the text to be deduplicated is a duplicate text based on the ratio of the number of non-repeating words to the total number of word segments in the title; if the ratio exceeds a preset value, determine that the text to be deduplicated is not a duplicate text; if the ratio does not exceed the preset value, determine that the text to be deduplicated is a duplicate text.

[0019] In some embodiments of the present application, the preset parts of speech include nouns, English, and quantifiers.

[0020] A second aspect of the present invention provides a text deduplication device, the device comprising:

[0021] A first determination module is used to determine representative words in the title of the text to be deduplicated;

[0022] A first judgment module is used to judge whether the representative word exists in the index in the index space constructed by the title of the deduplicated text;

[0023] A second determination module is used to determine that the to-be-deduplicated text is not a duplicate text when it is determined that the representative word does not exist;

[0024] A second judgment module is used to determine whether the to-be-deduplicated text is a duplicate text based on the title part of speech when it is determined that the representative word exists;

[0025] The space adding module is used to add the representative word as the index and the title and the part-of-speech tagging result of the title as the key value to the index space when it is determined that the text to be deduplicated is not a duplicate text.

[0026] The third aspect of the present invention provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the method described in the first aspect when executing the program.

[0027] A fourth aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein the program, when executed by a processor, implements the steps of the method described in the first aspect above.

[0028] Based on the text deduplication method and device described in the first and second aspects above, the present invention has at least the following beneficial effects or advantages:

[0029] Usually the title is a summary of the text, containing the core information in the text. The present invention can achieve the same effect as deduplication of the text by deduplicating the title. At the same time, the title is a short text, and the efficiency of processing the title is much higher than that of processing the text. For large-scale text data, the use of title deduplication can greatly reduce the processing time.

[0030] In addition, by obtaining the most influential representative words in the title of the text to be deduplicated, the complexity of deduplication can be further reduced and the efficiency of deduplication can be improved. Based on the assumption that the most influential words between similar semantic texts are the same, when it is determined that this representative word exists in the index space constructed by the title of the deduplicated text, deduplication is performed based on the title part of speech, thereby achieving semantic-aware deduplication.

[0031] It can be seen that this solution is not only able to perceive semantic deduplication, but also has a high deduplication efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0033] Figure 1 The present invention is a flowchart of a method for deduplication of text according to an exemplary embodiment of the present invention;

[0034] Figure 2 A schematic diagram of contents of an index space according to an exemplary embodiment of the present invention;

[0035] Figure 3 It is a structural schematic diagram of a text deduplication device according to an exemplary embodiment of the present invention;

[0036] Figure 4 It is a schematic diagram of the hardware structure of an electronic device according to an exemplary embodiment of the present invention;

[0037] Figure 5 The figure is a schematic diagram of the structure of a storage medium according to an exemplary embodiment of the present invention. DETAILED DESCRIPTION

[0038] Here, exemplary embodiments will be described in detail, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention.

[0039] The terms used in the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. The singular forms "a", "the" and "the" used in the present invention are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0040] It should be understood that although the terms first, second, third, etc. may be used in the present invention to describe various information, these information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of the present invention, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0041] In view of the problem that traditional deduplication methods cannot perceive the semantics between texts and have low deduplication efficiency, the present invention proposes an improved text deduplication method, namely, determining representative words in the title of the text to be deduplicated, and judging whether the index in the index space constructed by the title of the deduplicated text exists. If not, it is determined that the text to be deduplicated is not a duplicate text. If so, it is determined whether the text to be deduplicated is a duplicate text based on the title part of speech. When it is determined that the text to be deduplicated is not a duplicate text, the representative word is used as the index, and other words in the title except the representative word are added to the index space as key values.

[0042] The technical effects that can be achieved based on the above description are:

[0043] Usually the title is a summary of the text, containing the core information in the text. The present invention can achieve the same effect as deduplication of the text by deduplicating the title. At the same time, the title is a short text, and the efficiency of processing the title is much higher than that of processing the text. For large-scale text data, the use of title deduplication can greatly reduce the processing time.

[0044] In addition, by obtaining the most influential representative words in the title of the text to be deduplicated, the complexity of deduplication can be further reduced and the efficiency of deduplication can be improved. Based on the assumption that the most influential words between similar semantic texts are the same, when it is determined that this representative word exists in the index space constructed by the title of the deduplicated text, deduplication is performed based on the title part of speech, thereby achieving semantic-aware deduplication.

[0045] It can be seen that this solution can not only perceive semantic differences and remove duplicates, but also has a high deduplication efficiency. It performs well in large-scale text deduplication tasks and can meet actual production needs.

[0046] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application.

[0047] Embodiment 1:

[0048] Figure 1 The flowchart of an embodiment of a text deduplication method according to an exemplary embodiment of the present invention includes the following steps:

[0049] Step 101: Determine representative words in the title of the text to be deduplicated.

[0050] Among them, representative words refer to the most influential words in the title, which can represent the semantics of the title.

[0051] In an optional embodiment, the title is segmented to obtain a segmentation result, and the term frequency-inverse document frequency TF-IDF value of each segmentation in the segmentation result is determined, and then the segmentation corresponding to the maximum TF-IDF value is determined as a representative word.

[0052] Among them, the larger the TF-IDF value, the greater the influence of the word in the title, so the word corresponding to the maximum TF-IDF value is determined as the representative word.

[0053] The calculation process of the term frequency-inverse document frequency TF-IDF value is as follows:

[0054] First, an IDF vocabulary table of the target field is constructed. The present invention takes the bidding field as an example.

[0055] The inverse document frequency (IDF) is calculated by dividing the total number of documents by the number of documents containing the term, and represents a measure of the prevalence of a term. The size of the IDF value represents the commonness of a term. The higher the IDF value, the less common the term is. Usually, uncommon terms can better represent the semantics of a paragraph. In order to obtain an IDF vocabulary suitable for the bidding field, the present invention performs word segmentation and part-of-speech tagging on large-scale bidding field titles (e.g., 5000w titles), and counts and organizes the IDF vocabulary in the bidding field. The calculation method of IDF is as follows:

[0056]

[0057] Where D represents the total number of titles, |{1:t j ∈d j}| means that the word t is included j Total number of titles.

[0058] Then, the IDF value of each word in the word segmentation result is obtained from the IDF vocabulary table, and the TF value of each word is determined at the same time. The calculation formula of the TF value is as follows:

[0059]

[0060] Among them, n i represents the number of occurrences of participle i in the title, ∑ k n k Indicates the number of occurrences of all the words in the title.

[0061] Finally, the TF value and IDF value of each word are used to determine the TF-IDF value of the word. The TF-IDF value calculation formula is as follows:

[0062] TF_IDF i =TF i ×IDF i

[0063] Step 102: Determine whether the representative word exists in the index in the index space constructed by the title of the deduplicated text. If not, execute step 103; if yes, execute step 104.

[0064] Before executing step 102, for the construction process of the index space, the word segmentation results of the title of the deduplicated text and the TF-IDF value of each word segmentation are obtained. Based on the assumption that the most influential words between similar semantic texts are the same, the word with the largest TF-IDF value in the title is used as the index, and the title after word segmentation and part-of-speech tagging is used as the key value to construct the index space.

[0065] like Figure 2 As shown, the index space is established based on the title "Purchase Announcement of Experimental Equipment by Water Conservancy Engineering Research Laboratory", the first word "Water Conservancy Engineering" is the index, and the content in [] is the title and the part-of-speech tagging result is the key value.

[0066] In step 102, based on the assumption that the most influential words between similar semantic texts are the same, when the representative word is not in the index in the index space, it can be determined that the text to be deduplicated is a text without semantic repetition. When the representative word is in the index in the index space, deduplication determination is further performed based on the title part of speech.

[0067] Step 103: Determine that the text to be deduplicated is not a duplicate text.

[0068] Step 104: Determine whether the text to be deduplicated is a duplicate text based on the title part of speech.

[0069] In an optional embodiment, an existing title represented by a key value corresponding to a representative word in the index space is obtained, and non-repeating words between the existing title and the title are compared. If there are non-repeating words, whether the text to be deduplicated is a duplicate text is determined based on the part of speech of the non-repeating words. If there are no non-repeating words, it is determined that the text to be deduplicated is a duplicate text.

[0070] The non-repeated words between the existing title and the title of the text to be deduplicated may come from the existing title or the title of the text to be deduplicated.

[0071] It should be noted that, based on a large amount of experience, titles with different product words and quantifiers have different semantics in a high probability. The possible parts of speech of product words are nouns (n) and English (eng). Therefore, when there are non-repeating words in the titles with the parts of speech of nouns, English and quantifiers (m), the input title can be considered a non-repeating title.

[0072] Based on this, for the process of determining whether the text to be deduplicated is a duplicate text based on the part of speech of non-repeating words, it can be determined whether the part of speech of the non-repeating words is the preset part of speech. If it is the preset part of speech, it is determined that the text to be deduplicated is not a duplicate text. If it is not the preset part of speech, it is determined whether the text to be deduplicated is a duplicate text based on the proportion of the number of non-repeating words relative to the total number of word segments in the title. If the proportion exceeds the preset value, it is determined that the text to be deduplicated is not a duplicate text. If the proportion does not exceed the preset value, it is determined that the text to be deduplicated is a duplicate text.

[0073] Among them, the preset parts of speech are noun, English, and quantifier.

[0074] Furthermore, when the number of non-repeating words exceeds a certain value of the total number of words in the title, the titles cannot maintain the same semantics, and the text to be deduplicated is determined not to be a repeated text, otherwise it is a duplicate.

[0075] Optionally, based on a large amount of data statistics, the best effect is achieved when the preset value of the proportion is 1 / 3.

[0076] It should be additionally explained that when it is determined that the text to be deduplicated is a duplicate text, the text to be deduplicated may be discarded.

[0077] Step 105: When it is determined that the text to be deduplicated is not a duplicate text, a representative word is used as an index, and the title and the part-of-speech tagging result of the title are used as key values ​​to add to the index space.

[0078] Among them, when it is determined that the text to be deduplicated is not a duplicate text, it means that the text to be deduplicated is valuable text and is a deduplicated text. Its title needs to be processed and added to the index space for reference in deduplicating subsequent input text.

[0079] With respect to the process from step 101 to step 105 above, in order to verify the efficiency and accuracy of the algorithm proposed in the present invention, the present invention was tested in a large-scale corpus deduplication task of tender short texts, and 5,000 titles in the tender field were deduplicated.

[0080] Experiments show that the present invention can process 100 titles per second on average, and the time complexity does not increase with the increase in the number of titles, meeting actual production needs, and the accuracy of deduplication can reach more than 95%.

[0081] At this point, the above Figure 1 In the deduplication process shown, the title is usually a summary of the text, which contains the core information in the text. The present invention can achieve the same effect as deduplication of the text by deduplicating the title. At the same time, the title is a short text, and the efficiency of processing the title is much higher than that of processing the text. For large-scale text data, the use of title deduplication can greatly reduce the processing time.

[0082] In addition, by obtaining the most influential representative words in the title of the text to be deduplicated, the complexity of deduplication can be further reduced and the efficiency of deduplication can be improved. Based on the assumption that the most influential words between similar semantic texts are the same, when it is determined that this representative word exists in the index space constructed by the title of the deduplicated text, deduplication is performed based on the title part of speech, thereby achieving semantic-aware deduplication.

[0083] It can be seen that this solution is not only able to perceive semantic deduplication, but also has a high deduplication efficiency.

[0084] Corresponding to the above-mentioned embodiment of the text deduplication method, the present invention also provides an embodiment of a text deduplication device.

[0085] Figure 3 FIG. 1 is a schematic diagram of a text deduplication device according to an exemplary embodiment of the present invention. The device is used to execute the text deduplication method provided in any of the above embodiments. Figure 3 As shown, the text deduplication device includes:

[0086] A first determination module 310 is used to determine representative words in the title of the text to be deduplicated;

[0087] A first judgment module 320 is used to judge whether the representative word exists in the index in the index space constructed by the title of the deduplicated text;

[0088] A second determination module 330 is used to determine that the to-be-deduplicated text is not a duplicate text when it is determined that the representative word does not exist;

[0089] The second judgment module 340 is used to determine whether the to-be-deduplicated text is a duplicate text based on the title part of speech method when it is determined that the representative word exists;

[0090] The space adding module 350 is used to add the representative word as an index and the title and the part-of-speech tagging result of the title as key values ​​to the index space when it is determined that the text to be deduplicated is not a duplicate text.

[0091] The implementation process of the functions and effects of each unit in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned method, and will not be repeated here.

[0092] For the device embodiment, since it basically corresponds to the method embodiment, the relevant parts can refer to the partial description of the method embodiment. The device embodiment described above is only schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of the present invention. Ordinary technicians in this field can understand and implement it without paying creative work.

[0093] An embodiment of the present invention further provides an electronic device corresponding to the text deduplication method provided in the above embodiment, so as to execute the above text deduplication method.

[0094] Figure 4 This is a hardware structure diagram of an electronic device according to an exemplary embodiment of the present invention, and the electronic device includes: a communication interface 601, a processor 602, a memory 603 and a bus 604; wherein the communication interface 601, the processor 602 and the memory 603 communicate with each other through the bus 604. The processor 602 can execute the text deduplication method described above by reading and executing the machine executable instructions corresponding to the control logic of the text deduplication method in the memory 603. The specific content of the method is referred to the above embodiment, and will not be repeated here.

[0095] The memory 603 mentioned in the present invention can be any electronic, magnetic, optical or other physical storage device, and can contain storage information, such as executable instructions, data, etc. Specifically, the memory 603 can be RAM (Random Access Memory), flash memory, storage drive (such as hard disk drive), any type of storage disk (such as optical disk, DVD, etc.), or similar storage medium, or a combination thereof. The communication connection between the system network element and at least one other network element is realized through at least one communication interface 601 (which can be wired or wireless), and the Internet, wide area network, local area network, metropolitan area network, etc. can be used.

[0096] The bus 604 may be an ISA bus, a PCI bus or an EISA bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. The memory 603 is used to store programs, and the processor 602 executes the programs after receiving execution instructions.

[0097] The processor 602 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit or software instructions in the processor 602. The above processor 602 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present application can be implemented or executed. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in conjunction with the embodiments of the present application can be directly embodied as a hardware decoding processor to be executed, or the hardware and software modules in the decoding processor are combined to be executed.

[0098] The electronic device provided in the embodiment of the present application and the text deduplication method provided in the embodiment of the present application are based on the same inventive concept and have the same beneficial effects as the methods adopted, run or implemented therein.

[0099] The present application also provides a computer-readable storage medium corresponding to the text deduplication method provided in the above embodiment. Figure 5 As shown, the computer-readable storage medium shown is a CD 30 on which a computer program (ie, a program product) is stored. When the computer program is run by a processor, the text deduplication method provided by any of the aforementioned embodiments will be executed.

[0100] It should be noted that examples of the computer-readable storage medium may also include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other optical or magnetic storage media, which are not listed here one by one.

[0101] The computer-readable storage medium provided by the above-mentioned embodiments of the present application and the text deduplication method provided by the embodiments of the present application are based on the same inventive concept and have the same beneficial effects as the methods adopted, run or implemented by the application programs stored therein.

[0102] Those skilled in the art will readily appreciate other embodiments of the present invention after considering the specification and practicing the invention disclosed herein. The present invention is intended to cover any variations, uses or adaptations of the present invention that follow the general principles of the present invention and include common knowledge or customary techniques in the art not disclosed by the present invention. The description and examples are to be regarded as exemplary only, and the true scope and spirit of the present invention is indicated by the following claims.

[0103] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0104] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A text deduplication method, characterized in that: The method comprises: Determine representative words in the title of the text to be deduplicated; Determine whether the representative word exists in the index in the index space constructed by the title of the deduplicated text; If it does not exist, it is determined that the text to be deduplicated is not a duplicate text; If it exists, then obtain the existing title represented by the key value corresponding to the representative word in the index space, and determine whether the text to be deduplicated is a duplicate text based on whether the non-duplicate word between the existing title and the title is a preset part of speech. If it is a preset part of speech, determine that the text to be deduplicated is not a duplicate text, and the preset part of speech includes noun, English, and quantifier; When it is determined that the text to be deduplicated is not a duplicate text, the representative word is used as an index, and the title and the part-of-speech tagging result of the title are used as key values ​​to add to the index space; The representative words in the title of the text to be deduplicated are determined, including: Perform word segmentation on the title to obtain a word segmentation result; Determine the term frequency-inverse document frequency TF-IDF value of each word in the word segmentation result; The word segment corresponding to the maximum TF-IDF value is determined as the representative word.

2. The method according to claim 1, characterized in that: Determine the word frequency-inverse text frequency TF-IDF value of each word in the word segmentation result, including: Determine the word frequency TF value of each word in the word segmentation result; Get the inverse document frequency (IDF) value corresponding to each word segment from the preset vocabulary; The TF-IDF value of each word is determined using the TF value and IDF value of the word.

3. The method according to claim 1, characterized in that Determine whether the text to be deduplicated is a duplicate text based on the title part of speech, including: Obtaining an existing title represented by a key value corresponding to the representative word in the index space; Compare the existing title with the title for non-repeated words; If there are non-repeating words, determining whether the to-be-deduplicated text is a repeated text according to the part of speech of the non-repeating words; If there is no non-repeated word, it is determined that the text to be deduplicated is a repeated text.

4. The method according to claim 3, characterized in that Determining whether the to-be-deduplicated text is a duplicate text according to the part of speech of the non-duplicate words includes: Determine whether the part of speech of the non-repeated word is the preset part of speech; If it is the preset part of speech, it is determined that the text to be deduplicated is not a duplicate text; If it is not the preset part of speech, determining whether the to-be-deduplicated text is a duplicate text according to the ratio of the number of non-duplicate words to the total number of segmented words in the title; If the proportion exceeds the preset value, it is determined that the text to be deduplicated is not a duplicate text; If the proportion does not exceed the preset value, it is determined that the text to be deduplicated is a duplicate text.

5. A text deduplication device, characterized in that: The device comprises: A first determination module is used to determine representative words in the title of the text to be deduplicated; A first judgment module is used to judge whether the representative word exists in the index in the index space constructed by the title of the deduplicated text; A second determination module is used to determine that the to-be-deduplicated text is not a duplicate text when it is determined that the representative word does not exist; The second judgment module is used to obtain an existing title represented by a key value corresponding to the representative word in the index space when judging that the representative word exists, and determine whether the text to be deduplicated is a duplicate text based on whether the non-duplicate word between the existing title and the title is a preset part of speech, and if it is a preset part of speech, determine that the text to be deduplicated is not a duplicate text, and the preset part of speech includes noun, English, and quantifier; A space adding module, used for adding the representative word as an index, the title and the part-of-speech tagging result of the title as key values ​​to the index space when it is determined that the text to be deduplicated is not a duplicate text; The first determination module is specifically used to segment the title to obtain a segmentation result; determine the term frequency-inverse document frequency TF-IDF value of each segmentation in the segmentation result; and determine the segmentation corresponding to the maximum TF-IDF value as a representative word.

6. The device according to claim 5, characterized in that The second judgment module is specifically used to obtain an existing title represented by a key value corresponding to the representative word in the index space; compare the non-repeating words between the existing title and the title; if there are non-repeating words, determine whether the text to be deduplicated is a duplicate text based on the part of speech of the non-repeating words; if there are no non-repeating words, determine that the text to be deduplicated is a duplicate text.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps of the method according to any one of claims 1 to 4 are implemented.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method according to any one of claims 1 to 4 are implemented.

Images