De-duplication extraction method and device for text elements and storage medium

Abstract

The invention relates to a de-duplication extraction method for text elements, which comprises the following steps of: acquiring text information, preprocessing a text to obtain a text element set without de-duplication, inputting the set into a cell of an excel table, selecting a separation symbol to perform column separation by utilizing a column separation function in the excel, selecting a separator as a comma, and performing de-duplication extraction on the text element. The method comprises the following steps of: dividing a text set into one line, copying data in the line, selectively pasting in excel, checking and transposing in the selective pasting, converting a text in one line into a column, selecting the data in the line, and de-duplicating by utilizing a duplicate item deleting function in the excel to obtain text elements subjected to de-duplicating extraction. Compared with the prior art, the method has the advantages that the dependence on professional software is reduced, the applicability is high, the tedious manual processing process is simplified, the steps are clear, a user can understand and operate the method conveniently, and the accuracy and efficiency of duplicate removal extraction are improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of text processing, and in particular relates to a method, device and storage medium for deduplicating and extracting text elements. Background Art

[0002] With the development of the times and the advancement of technology, the Internet has penetrated into all aspects of people's lives, especially the newly emerging entertainment methods in recent years - online variety shows are also deeply loved by young people. Many young people regard chasing stars as a part of their lives, and most young people often use instant messaging software such as Weibo to check the information of the stars they follow. For most people born in the 1980s, they usually pay attention to sports news to understand current hot topics, while for those born in the 1990s and 2000s, their interests and hobbies are more extensive, and they are more willing to obtain information by browsing various popular video websites. With the rapid development of society and the advent of the information age, the lifestyle of young people is rapidly moving towards diversification. More and more young people are accustomed to posting topics of interest on the Internet, and in such communication processes, a large number of Internet buzzwords will appear, which are often the most popular words or phrases at the moment.

[0003] However, the list of hot words on the Internet obtained through web crawler technology will have duplications. In the prior art, data deduplication is generally achieved in two ways. The first is to use SQL statements in the database to achieve deduplication; the second is to merge multiple data files into one document and then perform deduplication operations. Both methods require the use of computer programming languages ​​​​to be implemented. On the one hand, it will have an impact on the data itself, and on the other hand, it will take a lot of time to complete the deduplication work. In the actual application process, it is often necessary to face the problem of multi-source heterogeneity, that is, it is necessary to collect original data from multiple different channels and perform subsequent processing and analysis to obtain the required results. However, under normal circumstances, there is no unified data format and classification conditions, which greatly increases the difficulty of cleaning and integrating the original data set. Moreover, this large-scale and complex data cleaning process is also difficult to rely on manual implementation, so automatic data cleaning technology based on natural language processing is particularly important. Summary of the invention

[0004] The purpose of the present invention is to provide a method, device and storage medium for deduplicating text elements in order to overcome the defects of the above-mentioned prior art.

[0005] The purpose of the present invention can be achieved by the following technical solutions:

[0006] On one hand, the present invention provides a method for removing duplicate text elements, comprising the following steps:

[0007] Copying a plurality of target cells, wherein the target cells contain one or more text elements to be extracted;

[0008] Paste all the copied target cells into a blank word processing document as unformatted text;

[0009] Remove all carriage returns from a word processing document;

[0010] Copy the processed text into a cell in the Excel table, use the Split into Columns function in Excel, select the cell, select the separator symbol to split into columns, select the separator as the first preset symbol, and divide each text element in the text into a row;

[0011] Copy the row of data, perform special paste in Excel, check Transpose in special paste, and convert the text in one row into one column;

[0012] Select the column of data and use the delete duplicates function in Excel to deduplicate all text elements in the column to obtain a set of text elements that have been deduplicated.

[0013] Furthermore, the target cell includes a cell in an Excel spreadsheet.

[0014] Furthermore, all text elements in the target cell are separated by a first preset symbol.

[0015] Furthermore, the first preset symbols include a period, a comma, a question mark, an exclamation mark, a colon, a semicolon, quotation marks, and brackets.

[0016] Furthermore, dividing each text element in the text into a row specifically divides all text elements in the text into a corresponding cell.

[0017] Furthermore, one text element corresponds to one cell.

[0018] Furthermore, all cells having text elements are located in one row.

[0019] The present invention also provides a device for removing duplicates and extracting text elements, comprising:

[0020] A first processing module is used to copy multiple target cells, paste all the copied target cells into a blank word processing document in a non-formatted text format, and delete all carriage returns in the word processing document;

[0021] The second processing module is used to copy the processed text into a cell of an Excel table, use the column splitting function in Excel to select the cell, select the separator symbol to split into columns, select the separator as the first preset symbol, and divide each text element in the text into a row;

[0022] The third processing module is used to copy the data obtained after being processed by the second processing module, perform selective pasting in Excel, check transpose in the selective paste, and convert a row of text into a column;

[0023] The fourth processing module is used to deduplicate the data processed by the third processing module.

[0024] The present invention also provides an electronic device, comprising: a processor and a memory, wherein the processor is used to execute a processing program for a table in a version document stored in the memory to implement the above-mentioned method for deduplicating and extracting text elements.

[0025] The present invention also provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, the method described above is implemented.

[0026] Compared with the prior art, the present invention has the following advantages:

[0027] The present invention provides an efficient and systematic text element deduplication extraction method, which uses common text processing software and Excel tools to reduce the dependence on professional software and has strong applicability. Through the use of preprocessing and delimiters, the text can be quickly divided into individual elements, simplifying the cumbersome manual processing process. In addition, the steps of the method are clear, easy for users to understand and operate, and improve the accuracy and efficiency of deduplication extraction. Compared with the prior art, the method not only reduces the time cost, but also improves the flexibility of text processing, and is applicable to a variety of text formats. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 The present invention is a flow chart of the method. DETAILED DESCRIPTION

[0029] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0030] Embodiment 1:

[0031] On the one hand, this embodiment provides a method for removing duplicate text elements, including the following steps:

[0032] Copy multiple target cells, wherein the target cells contain one or more text elements to be extracted; the target cells include cells in an Excel spreadsheet, and all text elements in the target cells are separated by a first preset symbol, and the first preset symbol includes a period, a comma, a question mark, an exclamation mark, a colon, a semicolon, quotation marks, and brackets.

[0033] Paste all the copied target cells into a blank word processing document as unformatted text;

[0034] Remove all carriage returns from a word processing document;

[0035] The processed text is copied into a cell of an Excel table, and the column splitting function in Excel is used to select the cell, select the separator symbol to split into columns, and select the separator as the first preset symbol to divide each text element in the text into a row; the dividing each text element in the text into a row specifically divides all the text elements in the text, and each element is divided into a corresponding cell, wherein one text element corresponds to one cell, and all cells having text elements are located in one row.

[0036] Copy the row of data, perform special paste in Excel, check Transpose in special paste, and convert the text in one row into one column;

[0037] Select the column of data and use the delete duplicates function in Excel to deduplicate all text elements in the column to obtain a set of text elements that have been deduplicated.

[0038] Various Internet hot words are captured from different forum communities through web crawler technology. Some of the Internet hot words captured from different communities are the same, which affects the research on users' popular interests. Therefore, it is necessary to deduplicate and extract these Internet hot words.

[0039] like Figure 1 The method for extracting duplicate text elements shown in the figure comprises the following steps: copying multiple target cell contents from a network hot word table obtained by a network crawler, wherein the multiple target cell contents are as shown in Table 1

[0040] Table 1 List of Internet hot words obtained

[0041]

[0042] The text is input into the first processing module for preprocessing. The first processing module pastes all the copied target cell contents into a blank word processing document in the form of unformatted text, deletes all the carriage returns in the word processing document, and obtains a text containing all the text elements of all the target cells: "trend, new energy vehicles, quantum mechanics, breaking defense, double standards, metaverse, cloud cat raising, working people, stalk culture, short video, metaverse, cloud computing, weeding, filters, large models, co-renting, artificial intelligence, blockchain, 5G, quantum computing, metaverse, cloud computing, Internet of Things, deep learning, edge computing, digital twins, reverse Spring Festival travel, star chasing, planting grass, weeding, filters, rollover, private domain traffic, internet celebrity, bringing goods, live streaming bringing goods, social e-commerce, community marketing, comfort zone, hot products, brain holes, artificial intelligence, blockchain, hardcore, reverse Spring Festival travel, star chasing, social animals, comfort zone, hot products, brain holes, krypton gold, large models, co-renting, internet celebrity, bringing goods".

[0043] Input the preprocessed text into the second processing module for processing. The second processing module copies the processed text into a cell of the Excel table. Use the column splitting function in Excel to select the cell, select the separator symbol to split into columns, select the separator as the first preset symbol, and divide each text element in the text into a row.

[0044] The data processed by the second processing module is input into the third processing module for processing. The third processing module copies the data processed by the second processing module, and selectively pastes it in Excel. In the selective paste, check Transpose to convert a row of text into a column. The processed data is shown in Table 2.

[0045] Table 2 List after processing by the third processing module

[0046]

[0047]

[0048]

[0049] The fourth processing module selects the column data and uses the delete duplicates function in Excel to perform a deduplication operation on all text elements in the column to obtain a set of text elements extracted after deduplication, as shown in Table 3.

[0050] Table 3 List of hot words extracted from the Internet after deduplication

[0051]

[0052]

[0053]

[0054] Embodiment 2:

[0055] This embodiment also provides a device for removing duplicate text elements, including:

[0056] A first processing module is used to copy multiple target cells, paste all the copied target cells into a blank word processing document in a non-formatted text format, and delete all carriage returns in the word processing document;

[0057] The second processing module is used to copy the processed text into a cell of an Excel table, use the column splitting function in Excel to select the cell, select the separator symbol to split into columns, select the separator as the first preset symbol, and divide each text element in the text into a row;

[0058] The third processing module is used to copy the data obtained after being processed by the second processing module, perform selective pasting in Excel, check transpose in the selective paste, and convert a row of text into a column;

[0059] The fourth processing module is used to deduplicate the data processed by the third processing module.

[0060] This embodiment also provides an electronic device, including: a processor and a memory, wherein the processor is used to execute a processing program for a table in a version document stored in the memory to implement the above-mentioned method for deduplicating and extracting text elements.

[0061] This embodiment further provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, the above-mentioned method is implemented.

[0062] If the above functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc., which can store program code.

[0063] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present invention, and these modifications or replacements should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A method for removing duplicates and extracting text elements, characterized in that: The following steps are involved: Copying a plurality of target cells, wherein the target cells contain one or more text elements to be extracted; Paste all the copied target cells into a blank word processing document as unformatted text; Remove all carriage returns from a word processing document; Copy the processed text into a cell in the Excel table, use the Split into Columns function in Excel, select the cell, select the separator symbol to split into columns, select the separator as the first preset symbol, and divide each text element in the text into a row; Copy the row of data, perform special paste in Excel, check Transpose in special paste, and convert the text in one row into one column; Select the column of data and use the delete duplicates function in Excel to deduplicate all text elements in the column to obtain a set of text elements that have been deduplicated.

2. The method for removing duplicates and extracting text elements according to claim 1, characterized in that: The target cell includes a cell in an Excel table.

3. The method for removing duplicates and extracting text elements according to claim 1, characterized in that: All text elements in the target cell are separated by a first preset symbol.

4. The method for removing duplicates and extracting text elements according to claim 3, characterized in that: The first preset symbols include a period, a comma, a question mark, an exclamation mark, a colon, a semicolon, quotation marks, and brackets.

5. The method for removing duplicates and extracting text elements according to claim 1, characterized in that: The step of dividing each text element in the text into a row specifically involves dividing all text elements in the text into a corresponding cell.

6. A method for removing duplicates and extracting text elements according to claim 5, characterized in that: The one text element corresponds to one cell.

7. The method for removing duplicates and extracting text elements according to claim 5, characterized in that: All cells having text elements are located in one row.

8. A device for removing duplicates and extracting text elements, characterized in that: include: A first processing module is used to copy multiple target cells, paste all the copied target cells into a blank word processing document in a non-formatted text format, and delete all carriage returns in the word processing document; The second processing module is used to copy the processed text into a cell of an Excel table, use the column splitting function in Excel to select the cell, select the separator symbol to split into columns, select the separator as the first preset symbol, and divide each text element in the text into a row; The third processing module is used to copy the data obtained after being processed by the second processing module, perform selective pasting in Excel, check transpose in the selective paste, and convert a row of text into a column; The fourth processing module is used to deduplicate the data processed by the third processing module.

9. An electronic device, characterized in that: include: A processor and a memory, wherein the processor is used to execute a processing program for a table in a version document stored in the memory, so as to implement the deduplication extraction method for text elements according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images