A text filtering method, device, equipment and storage medium
By calculating the keyword vectors in the text and judging the cosine similarity, the text filtering problem in the prior art is solved, and text with inconsistent themes and contents can be quickly filtered, improving user experience and information acquisition efficiency.
Patent Information
- Application Number
- CN202010845647.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2020-08-20
- Publication Date
- 2024-11-22
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
The prior art is difficult to effectively filter text with inconsistent topics and content, resulting in users wasting time during reading and searching and unable to quickly find the required information.
By extracting keywords in text and inputting them into pre-trained neural network model, calculating the average vector of phrase vectors, judging the cosine similarity, filtering inappropriate text, and saving the filtered text vector to the database to achieve the theme and content matching filtering.
It improves users' experience when reading and searching, quickly filters out text with inconsistent topics and content, and improves the accuracy and efficiency of information acquisition.
Smart Images

Figure CN111966789B_ABST
Abstract
Description
Technical Field
[0001] The embodiments of the present invention relate to text technology, and in particular to a text filtering method, device, equipment and storage medium. Background Art
[0002] In an era of attracting attention and earning clicks, the themes of many texts are inconsistent with their content, or the contents are inconsistent with each other. Merchants gain clicks, but users’ time is wasted.
[0003] In the existing process of users obtaining texts for reading, there are a large number of so-called "clickbait" texts, but there is no way to filter these texts. Users need to screen them one by one and cannot directly find the text they want to read, or when users need to search based on certain keywords, they often cannot find the text they want by searching only by title. Therefore, the need to filter texts that are inconsistent with the subject and its content is becoming more and more urgent to improve users' reading and search experience. Summary of the invention
[0004] The embodiment of the present invention provides a text filtering method, device, equipment and storage medium to filter texts with inconsistent themes and contents.
[0005] To achieve this purpose, an embodiment of the present invention provides a text filtering method, which includes:
[0006] Obtaining a target text and a text title of the target text;
[0007] Extracting a first preset number of first keywords from the target text, and extracting a second preset number of second keywords from the text title;
[0008] Inputting the first keyword into a pre-trained neural network model to obtain a first phrase vector, and inputting the second keyword into a pre-trained neural network model to obtain a second phrase vector;
[0009] Determine an average vector of the first phrase vectors as a first vector, and determine an average vector of the second phrase vectors as a second vector;
[0010] determining a first cosine similarity between the first vector and the second vector;
[0011] The target text is filtered according to the first cosine similarity.
[0012] Further, filtering the target text according to the first cosine similarity includes:
[0013] Determining whether the first cosine similarity is less than a first threshold;
[0014] If the first cosine similarity is less than a first threshold, the target text is filtered.
[0015] Further, the determining the average vector of the first phrase vector as the first vector and determining the average vector of the second phrase vector as the second vector comprises:
[0016] Determine an average vector of the first vector and the second vector as a third vector;
[0017] The target text and the third vector are combined and saved in a preset database.
[0018] Furthermore, the combining of the target text and the third vector and saving them in a preset database comprises:
[0019] Obtaining a fourth vector of the comparison text in the preset database;
[0020] Determining a second cosine similarity between the third vector and the fourth vector;
[0021] The similarity between the target text and the comparison text is determined according to the second cosine similarity.
[0022] Furthermore, the combining of the target text and the third vector and saving them in a preset database comprises:
[0023] Acquire a third preset number of third keywords input by a user, where the third preset number is the sum of the first preset number and the second preset number;
[0024] Inputting the third keyword into a pre-trained neural network model to obtain a third phrase vector;
[0025] Determine an average vector of the third phrase vectors as a fifth vector;
[0026] A search text is determined from the preset database, wherein a third cosine similarity between a sixth vector of the search text and the fifth vector is greater than a second threshold.
[0027] Furthermore, the first keyword is extracted using TF-IDF technology, and the second keyword is extracted using LSA model or LDA model.
[0028] Furthermore, the neural network model is a Word2vec model.
[0029] On the one hand, an embodiment of the present invention further provides a text filtering device, the device comprising:
[0030] A text acquisition module, used to acquire a target text and a text title of the target text;
[0031] A keyword extraction module, used to extract a first preset number of first keywords from the target text, and extract a second preset number of second keywords from the text title;
[0032] a vector generation module, configured to input the first keyword into a pre-trained neural network model to obtain a first phrase vector, and input the second keyword into a pre-trained neural network model to obtain a second phrase vector;
[0033] a vector averaging module, configured to determine an average vector of the first phrase vectors as a first vector, and determine an average vector of the second phrase vectors as a second vector;
[0034] A similarity determination module, configured to determine a first cosine similarity between the first vector and the second vector;
[0035] A text filtering module is used to filter the target text according to the first cosine similarity.
[0036] On the other hand, an embodiment of the present invention further provides a computer device, which includes: one or more processors; a storage device for storing one or more programs, when the one or more programs are executed by the one or more processors, the one or more processors implement a method provided in any embodiment of the present invention.
[0037] On the other hand, an embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements a method provided in any embodiment of the present invention.
[0038] The embodiment of the present invention obtains a target text and a text title of the target text; extracts a first preset number of first keywords from the target text, and extracts a second preset number of second keywords from the text title; inputs the first keyword into a pre-trained neural network model to obtain a first phrase vector, and inputs the second keyword into a pre-trained neural network model to obtain a second phrase vector; determines the average vector of the first phrase vector as the first vector, and determines the average vector of the second phrase vector as the second vector; determines the first cosine similarity between the first vector and the second vector; and filters the target text according to the first cosine similarity, thereby solving the problem that a user cannot directly obtain the text he wants when searching or reading, and achieving the effect of filtering texts with inconsistent themes and contents. BRIEF DESCRIPTION OF THE DRAWINGS
[0039] Figure 1 It is a flowchart of a text filtering method provided in Embodiment 1 of the present invention;
[0040] Figure 2 It is a flowchart of a text filtering method provided in Embodiment 2 of the present invention;
[0041] Figure 3 is a flowchart of a similar text search method provided in Embodiment 2 of the present invention;
[0042] Figure 4 It is a flowchart of a keyword text search method provided in Embodiment 2 of the present invention;
[0043] Figure 5 is a structural schematic diagram of a text filtering device provided in Embodiment 3 of the present invention;
[0044] Figure 6 A schematic diagram of the structure of a computer device provided in Embodiment 4 of the present invention. DETAILED DESCRIPTION
[0045] The present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are used to explain the present invention, rather than to limit the present invention. It should also be noted that, for ease of description, only the parts related to the present invention, rather than all structures, are shown in the accompanying drawings.
[0046] It should be mentioned before discussing the exemplary embodiments in more detail that some exemplary embodiments are described as processes or methods depicted as flow charts. Although the flow charts describe the steps as sequential processes, many of the steps therein can be implemented in parallel, concurrently or simultaneously. In addition, the order of the steps can be rearranged. The process can be terminated when its operation is completed, but can also have additional steps not included in the accompanying drawings. The process can correspond to a method, function, procedure, subroutine, subprogram, etc.
[0047] In addition, the terms "first", "second", etc. can be used in this article to describe various directions, actions, steps or elements, but these directions, actions, steps or elements are not limited by these terms. These terms are only used to distinguish a first direction, action, step or element from another direction, action, step or element. For example, without departing from the scope of the present application, the first module can be referred to as the second module, and similarly, the second module can be referred to as the first module. Both the first module and the second module are modules, but they are not the same module. The terms "first", "second", etc. cannot be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features. Thus, the features defined as "first" and "second" can expressly or implicitly include one or more of the features. In the description of the embodiment of the present invention, the meaning of "multiple" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.
[0048] Embodiment 1
[0049] like Figure 1 As shown, the first embodiment of the present invention provides a text filtering method, the method comprising:
[0050] S110: Acquire a target text and a text title of the target text.
[0051] S120: extract a first preset number of first keywords from the target text, and extract a second preset number of second keywords from the text title.
[0052] In this embodiment, when performing text filtering, it is first necessary to obtain the target text and the text title of the target text, wherein the target text is the text body of the target text, and then extract a first preset number of first keywords from the target text, and extract a second preset number of second keywords from the text title. The first keyword represents the key content of the target text, and the second keyword also represents the key content of the text title of the target text. The first preset number and the second preset number can be defined by the user. If the user has a higher accuracy requirement, the first preset number and the second preset number can be set to be larger. If the user has a higher processing speed requirement, the first preset number and the second preset number can be set to be smaller.
[0053] S130: Input the first keyword into a pre-trained neural network model to obtain a first phrase vector, and input the second keyword into a pre-trained neural network model to obtain a second phrase vector.
[0054] S140: Determine an average vector of the first phrase vectors as a first vector, and determine an average vector of the second phrase vectors as a second vector.
[0055] In this embodiment, after obtaining the first preset number of first keywords and the second preset number of second keywords, the first keyword can be input into a pre-trained neural network model to obtain a first phrase vector, and the second keyword can be input into a pre-trained neural network model to obtain a second phrase vector, wherein the neural network model is pre-trained, that is, by inputting the keyword into the neural network model, a phrase vector corresponding to the keyword can be obtained. Since the first keyword is the first preset number and the second keyword is the second preset number, the first phrase vector is also the first preset number, and the second phrase vector is also the second preset number. Then, the average vector of the first phrase vector is determined as the first vector, and the average vector of the second phrase vector is determined as the second vector, that is, the average value of the first preset number of first phrase vectors is calculated as the first vector, which is used to represent the target text, and the average value of the second preset number of second phrase vectors is calculated as the second vector, which is used to represent the text title of the target text.
[0056] S150: Determine a first cosine similarity between the first vector and the second vector.
[0057] S160: Filter the target text according to the first cosine similarity.
[0058] In this embodiment, after obtaining the first vector and the second vector, the first cosine similarity of the first vector and the second vector can be calculated. The first cosine similarity is used to represent the similarity between the target text and the text title of the target text. Therefore, it can be determined whether the target text needs to be filtered out based on the first cosine similarity. Thus, there is no need for the user to make a judgment by themselves, and texts with inconsistent themes and contents can be directly filtered out.
[0059] Furthermore, the target text can also be a paragraph in the text body of the target text, and a first preset number of first keywords are extracted from the paragraph to generate a first vector and a second vector of the text title to calculate the first cosine similarity. If it is determined based on the first cosine similarity that the paragraph and the text title have a low similarity, the paragraph can be filtered out, and the above method can be continued to be used to calculate the first cosine similarity of each paragraph in the target text, thereby filtering out paragraphs in the target text that are irrelevant to the text title.
[0060] The embodiment of the present invention obtains a target text and a text title of the target text; extracts a first preset number of first keywords from the target text, and extracts a second preset number of second keywords from the text title; inputs the first keyword into a pre-trained neural network model to obtain a first phrase vector, and inputs the second keyword into a pre-trained neural network model to obtain a second phrase vector; determines the average vector of the first phrase vector as the first vector, and determines the average vector of the second phrase vector as the second vector; determines the first cosine similarity between the first vector and the second vector; and filters the target text according to the first cosine similarity, thereby solving the problem that a user cannot directly obtain the text he wants when searching or reading, and achieving the effect of filtering texts with inconsistent themes and contents.
[0061] Embodiment 2
[0062] like Figure 2 As shown, the second embodiment of the present invention provides a text filtering method, which is further explained on the basis of the first embodiment of the present invention. The method includes:
[0063] S210: Acquire a target text and a text title of the target text.
[0064] In this embodiment, the target text may be multiple. For example, two target texts are obtained. Specifically, text A is "The new real estate project has grandly opened. The opportunity cannot be missed and will not come again...", the text title A of text A is "The new stock god is born! The current profit has been 10 times", and text B is "The stock market trading volume has recovered. Following the rapid rise of the securities sector, the pharmaceutical and chip sectors continued to rise. The ChiNext Index hit a 4-year high...", and the text title B of text B is "The rise has exploded across the board, and the stock market opportunity is coming."
[0065] S220: Extract a first preset number of first keywords from the target text, and extract a second preset number of second keywords from the text title.
[0066] In this embodiment, the first keyword is extracted using TF-IDF (Term Frequency–Inverse Document Frequency) technology, and the second keyword is extracted using LSA (Latent semantic analysis) model or LDA (Latent Dirichlet Allocation) model. Exemplarily, the first preset number and the second preset number are both set to 3, and the first keyword of text A is extracted using TF-IDF technology as "real estate, opening, opportunity not to be missed", and the first keyword of text B is extracted as "stock market, sector, rise", and then the second keyword of text A is extracted using the LDA model as "stock god, birth, profit", and the second keyword of text B is extracted as "detonation, rise, stock market".
[0067] S230: Input the first keyword into a pre-trained neural network model to obtain a first phrase vector, and input the second keyword into a pre-trained neural network model to obtain a second phrase vector.
[0068] S240: Determine an average vector of the first phrase vectors as a first vector, and determine an average vector of the second phrase vectors as a second vector.
[0069] In this embodiment, the neural network model is a Word2vec (word to vector) model. Exemplarily, the first keyword of text A is "property, opening, opportunity not to be missed" input into the pre-trained Word2vec model to obtain the first phrase vector, that is, [11, 2.6, ... 23, 13], [7, 47, ... 31, 4.22], [12, 3.1, ... 45, 62], and the second keyword of text A "stock god, birth, profit" is input into the Word2vec model to obtain the second phrase vector, that is, [0...2, 2.1, ... 14, 33], [3.2, 22, ... 20, 11], [6.6, 11, ... 1.2, 33], wherein each phrase vector is a 300-dimensional vector, and similarly processing text B can also obtain 3 first phrase vectors and 3 second phrase vectors. Furthermore, the average values of the three first phrase vectors are calculated as the first vector, namely [10, 17.57, ... 33, 26.4], and the average value of the second preset number of second phrase vectors is calculated as the second vector, namely [3.34, 11.7, ... 11.73, 25.67].
[0070] S250: Determine an average vector of the first vector and the second vector as a third vector.
[0071] S260: combining the target text and the third vector and saving them in a preset database.
[0072] In this embodiment, the average value of the first vector and the second vector can also be calculated as the third vector, that is, [6.67, 14.64, ... 22.37, 26.04]. The third vector is used to represent the entire target text, and the target text and the third vector are combined and saved in a preset database. The text in the preset database includes the third vector, so the preset database can be searched for similar texts and searched by keywords.
[0073] S270: Determine a first cosine similarity between the first vector and the second vector.
[0074] S280: Determine whether the first cosine similarity is less than a first threshold.
[0075] S290: If the first cosine similarity is less than a first threshold, filter the target text.
[0076] In this embodiment, it is necessary to calculate the cosine similarity of the first vector [10, 17.57, ... 33, 26.4] and the second vector [3.34, 11.7, ... 11.73, 25.67], that is, the first cosine similarity, and then determine whether the first cosine similarity is less than the first threshold. If the first cosine similarity of text A is less than the first threshold, text A is filtered. If the first cosine similarity of text B is greater than or equal to the first threshold, text B is retained.
[0077] As preferably, for the target text that is not filtered out, it can be displayed to the user. Before the target text that is not filtered out is displayed to the user, the sentences with low similarity between sentences in the same paragraph in these target texts can also be filtered out. Specifically, the target text is divided into paragraphs, and each paragraph is divided into sentences. The sentences in the same paragraph in the target text are sequentially input into the pre-trained twin neural network (Siamese Network) model in pairs to obtain the similarity between the sentences in the same paragraph, and the similarity of each sentence in the same paragraph is obtained according to the similarity between the sentences in the same paragraph, and finally the sentences are filtered according to the similarity and the target text is displayed. Exemplarily, the target text includes paragraph one, and paragraph one includes sentence A, sentence B and sentence C, and then sentence A and sentence B are taken as a group, sentence A and sentence C are taken as a group, and sentence B and sentence C are taken as a group, and are input into the pre-trained neural network model at one time, wherein the neural network model has been pre-trained, and the similarity of the two sentences can be obtained after inputting two sentences, thereby obtaining the similarity between sentence A and sentence B, the similarity between sentence A and sentence C, and the similarity between sentence B and sentence C. Then, the second and third paragraphs are processed in the same way until the entire target text is processed. Among them, the similarity between sentence A and sentence B is high, the similarity between sentence A and sentence C is low, and the similarity between sentence B and sentence C is low, which means that sentence C is invalid information in the paragraph, so sentence C is filtered out and the target text is displayed.
[0078] As a preferred method, after filtering out sentences with low similarity between sentences in the same paragraph, irrelevant words in the target text that have not been filtered out can also be filtered out. Specifically, the named entity recognition technology (NER, Named Entity Recognition) is used to extract the first entity words in the target text, and the first entity words are input into the pre-trained Word2vec neural network model to obtain the first word vector of each first entity word, determine the center point of the first word vector, and finally obtain the first entity words corresponding to the second word vector as the filter words of the target text, wherein the second word vector is the first word vector of the first preset number farthest from the center point. The first preset number is set by the user, and the user sets the first preset number according to the filtering requirements, and then obtains the first entity words corresponding to the second word vector. Because the second word vector is far from the center point, it means that the first entity words corresponding to the second word vector have a low correlation with the main meaning of the target text, so it is used as the filter word of the target text. Thus, the target text is displayed after filtering out the sentences and filter words with low similarity, and the user can quickly obtain the information he needs.
[0079] In one embodiment, if Figure 3 As shown, after step S260, the following similar text search method may also be performed:
[0080] S310: Obtain a fourth vector of the comparison text in the preset database.
[0081] S320: Determine a second cosine similarity between the third vector and the fourth vector.
[0082] S330: Determine the similarity between the target text and the comparison text according to the second cosine similarity.
[0083] In this embodiment, after obtaining the target text and the third vector of the target text, a text similar to the target text can be obtained from the preset database according to the third vector. Specifically, the fourth vector of the comparison text is obtained from the preset database, wherein the generation method of the fourth vector of the comparison text is the same as the generation method of the third vector of the target text, and the comparison text is all the texts saved in the preset database. The second cosine similarity of the fourth vector and the third vector is calculated one by one, and then the similarity of the target text and the comparison text is determined according to the second cosine similarity, and the fourth vectors whose second cosine similarity is greater than the preset value or whose second cosine similarity ranks first in the preset number from large to small, that is, the fourth vector with high similarity, are found, and then the comparison text corresponding to the fourth vector is obtained as the result of the target text similarity search. After the target text and the third vector of the target text are combined and saved in the preset database, they will be used as the comparison text and the fourth vector of the comparison text to be searched later.
[0084] Exemplarily, the fourth vector of a comparison text is [13, 43.51, ... 12, 9.22], and then the second cosine similarity of the third vector [6.67, 14.64, ... 22.37, 26.04] and the fourth vector [13, 43.51, ... 12, 9.22] is calculated. If the second cosine similarity is greater than a preset value, the comparison text is used as the result of the target text similarity search.
[0085] In one embodiment, if Figure 4 As shown, after step S260, the following keyword text search method may also be performed:
[0086] S410: Acquire a third preset number of third keywords input by a user, where the third preset number is the sum of the first preset number and the second preset number.
[0087] S420: Input the third keyword into a pre-trained neural network model to obtain a third phrase vector.
[0088] S430: Determine an average vector of the third phrase vectors as a fifth vector.
[0089] S440: Determine a search text from the preset database, wherein a third cosine similarity between a sixth vector of the search text and the fifth vector is greater than a second threshold.
[0090] In this embodiment, the texts in the preset database all include the third vector, so the user can perform text search by entering keywords. Specifically, first obtain the third preset number of third keywords input by the user. As a preferred embodiment, the third preset number is the sum of the first preset number and the second preset number. It should be noted that in the process of generating the third vector, the third preset number of keywords are extracted, so the number of keywords input by the user also needs to be the third preset number to improve the accuracy. When the third keywords input by the user are less than the third preset number, one of the third keywords can also be copied to reach the third preset number. Then the third keyword is input into the pre-trained neural network model to obtain the third phrase vector, and the average vector of the third phrase vector is determined as the fifth vector, and then the third cosine similarity between the fifth vector and each third vector in the preset database is calculated, and the sixth vector whose third cosine similarity is greater than the second threshold is found, or the sixth vector whose third cosine similarity is ranked in the first preset number from large to small, and the text corresponding to the sixth vector, that is, the search text, is output as the result.
[0091] Exemplarily, the third keyword input by the user is "stock market, sector, rise, detonation". Because it is less than the third preset number, it will be automatically supplemented to "stock market, sector, rise, detonation, rise, stock market", and the corresponding fifth vector is obtained as [6.67, 14.64, ... 22.37, 26.04]. Then, the third cosine similarity between the fifth vector and each third vector in the preset database is calculated. Finally, the sixth vector whose third cosine similarity is greater than the second threshold is searched, which is also [6.67, 14.64, ... 22.37, 26.04], that is, the search text is text B.
[0092] Embodiment 3
[0093] like Figure 5 As shown, the third embodiment of the present invention provides a text filtering device 100. The text filtering device 100 provided by the third embodiment of the present invention can execute the text filtering method provided by any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method. The text filtering device 100 includes a text acquisition module 200, a keyword extraction module 300, a vector generation module 400, a vector average module 500, a similarity determination module 600 and a text filtering module 700.
[0094] Specifically, the text acquisition module 200 is used to acquire the target text and the text title of the target text; the keyword extraction module 300 is used to extract a first preset number of first keywords from the target text, and to extract a second preset number of second keywords from the text title; the vector generation module 400 is used to input the first keyword into a pre-trained neural network model to obtain a first phrase vector, and to input the second keyword into a pre-trained neural network model to obtain a second phrase vector; the vector averaging module 500 is used to determine the average vector of the first phrase vector as the first vector, and to determine the average vector of the second phrase vector as the second vector; the similarity determination module 600 is used to determine the first cosine similarity of the first vector and the second vector; the text filtering module 700 is used to filter the target text according to the first cosine similarity.
[0095] In this embodiment, the text filtering module 700 is specifically used to determine whether the first cosine similarity is less than a first threshold; if the first cosine similarity is less than the first threshold, the target text is filtered. Preferably, the first keyword is extracted using TF-IDF technology, the second keyword is extracted using LSA model or LDA model, and the neural network model is Word2vec model.
[0096] Furthermore, the text filtering device 100 also includes a database building module 800 , a similarity search module 900 and a keyword search module 1000 .
[0097] Specifically, the database establishment module 800 is used to determine that the average vector of the first vector and the second vector is the third vector; the target text and the third vector are combined and saved in a preset database. The similarity search module 900 is used to obtain the fourth vector of the comparison text in the preset database; determine the second cosine similarity of the third vector and the fourth vector; and determine the similarity between the target text and the comparison text based on the second cosine similarity. The keyword search module 1000 is used to obtain a third preset number of third keywords input by the user, the third preset number being the sum of the first preset number and the second preset number; inputting the third keyword into a pre-trained neural network model to obtain a third phrase vector; determining the average vector of the third phrase vector as a fifth vector; and determining a search text from the preset database, wherein the third cosine similarity between the sixth vector of the search text and the fifth vector is greater than a second threshold.
[0098] Embodiment 4
[0099] Figure 6 This is a schematic diagram of the structure of a computer device 12 provided in Embodiment 4 of the present invention. Figure 6 A block diagram of an exemplary computer device 12 suitable for use in implementing embodiments of the present invention is shown. Figure 6 The computer device 12 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.
[0100] like Figure 6 As shown, the computer device 12 is in the form of a general-purpose computing device. The components of the computer device 12 may include, but are not limited to: one or more processors or processing units 16, a system memory 28, and a bus 18 that connects various system components (including the system memory 28 and the processing unit 16).
[0101] Bus 18 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processor or a local bus using any of a variety of bus architectures. By way of example, these architectures include, but are not limited to, an Industry Standard Architecture (ISA) bus, a Micro Channel Architecture (MAC) bus, an Enhanced ISA bus, a Video Electronics Standards Association (VESA) local bus, and a Peripheral Component Interconnect (PCI) bus.
[0102] The computer device 12 typically includes a variety of computer system readable media. These media can be any available media that can be accessed by the computer device 12, including volatile and non-volatile media, removable and non-removable media.
[0103] The system memory 28 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 30 and / or cache memory 32. The computer device 12 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, the storage system 34 may be used to read and write non-removable, non-volatile magnetic media ( Figure 6 not shown, usually called a "hard drive"). Although Figure 6 Not shown in the figure, a disk drive for reading and writing to a removable non-volatile disk (e.g., a "floppy disk"), and an optical disk drive for reading and writing to a removable non-volatile optical disk (e.g., a CD-ROM, a DVD-ROM, or other optical media) may be provided. In these cases, each drive may be connected to the bus 18 via one or more data medium interfaces. The memory 28 may include at least one program product having a set (e.g., at least one) of program modules that are configured to perform the functions of the various embodiments of the present invention.
[0104] A program / utility 40 having a set (at least one) of program modules 42 may be stored, for example, in the memory 28. Such program modules 42 include, but are not limited to, an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment. The program modules 42 generally perform the functions and / or methods of the embodiments described herein.
[0105] The computer device 12 may also communicate with one or more external devices 14 (e.g., keyboards, pointing devices, displays 24, etc.), may communicate with one or more devices that enable a user to interact with the computer device 12, and / or may communicate with any device that enables the computer device 12 to communicate with one or more other computing devices (e.g., network cards, modems, etc.). Such communication may be performed via an input / output (I / O) interface 22. Furthermore, the computer device 12 may also communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) via a network adapter 20. As shown, the network adapter 20 communicates with other modules of the computer device 12 via a bus 18. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the computer device 12, including, but not limited to, microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.
[0106] The processing unit 16 executes various functional applications and data processing by running the programs stored in the system memory 28, for example, implementing the method provided in the embodiment of the present invention:
[0107] Obtaining a target text and a text title of the target text;
[0108] Extracting a first preset number of first keywords from the target text, and extracting a second preset number of second keywords from the text title;
[0109] Inputting the first keyword into a pre-trained neural network model to obtain a first phrase vector, and inputting the second keyword into a pre-trained neural network model to obtain a second phrase vector;
[0110] Determine an average vector of the first phrase vectors as a first vector, and determine an average vector of the second phrase vectors as a second vector;
[0111] determining a first cosine similarity between the first vector and the second vector;
[0112] The target text is filtered according to the first cosine similarity.
[0113] Embodiment 5
[0114] Embodiment 5 of the present invention further provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the method provided in all the embodiments of the present invention is implemented:
[0115] Obtaining a target text and a text title of the target text;
[0116] Extracting a first preset number of first keywords from the target text, and extracting a second preset number of second keywords from the text title;
[0117] Inputting the first keyword into a pre-trained neural network model to obtain a first phrase vector, and inputting the second keyword into a pre-trained neural network model to obtain a second phrase vector;
[0118] Determine an average vector of the first phrase vectors as a first vector, and determine an average vector of the second phrase vectors as a second vector;
[0119] determining a first cosine similarity between the first vector and the second vector;
[0120] The target text is filtered according to the first cosine similarity.
[0121] The computer storage medium of the embodiment of the present invention can adopt any combination of one or more computer-readable media. The computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples (non-exhaustive list) of computer-readable storage media include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this document, a computer-readable storage medium can be any tangible medium containing or storing a program, which can be used by an instruction execution system, device or device or used in combination with it.
[0122] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, which carry computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.
[0123] Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.
[0124] Computer program code for performing the operations of the present invention may be written in one or more programming languages or a combination thereof, including object-oriented programming languages such as Java, Smalltalk, C++, and conventional procedural programming languages such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).
[0125] Note that the above are only preferred embodiments of the present invention and the technical principles used. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and that various obvious changes, readjustments and substitutions can be made by those skilled in the art without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in more detail through the above embodiments, the present invention is not limited to the above embodiments, and may include more other equivalent embodiments without departing from the concept of the present invention, and the scope of the present invention is determined by the scope of the appended claims.
Claims
1. A text filtering method, It is characterized in that include: Obtaining a target text and a text title of the target text; Extracting a first preset number of first keywords from the target text, and extracting a second preset number of second keywords from the text title, wherein the first preset number and the second preset number are defined by a user; Inputting the first keyword into a pre-trained neural network model to obtain a first phrase vector, and inputting the second keyword into a pre-trained neural network model to obtain a second phrase vector; Determine an average vector of the first phrase vectors as a first vector, and determine an average vector of the second phrase vectors as a second vector; determining a first cosine similarity between the first vector and the second vector; Filter the target text according to the first cosine similarity; The first keyword is extracted by using TF-IDF technology, and the second keyword is extracted by using LSA model or LDA model; For the filtered target text, determining invalid sentences in each paragraph of the filtered target text, and filtering the invalid sentences; For the target text after filtering the invalid sentences, determining invalid words in the target text after filtering the invalid sentences, obtaining a target display text, and displaying the target display text; The determining of invalid sentences in each paragraph of the filtered target text includes: Input two by two sentences in the same paragraph of the filtered target text into the pre-trained twin neural network model in turn to obtain the similarity between the sentences in the same paragraph, and obtain the similarity of each sentence in the same paragraph based on the similarity between the sentences in the same paragraph, and finally determine the invalid sentences based on the similarity of each sentence in the same paragraph; The determining of invalid words in the target text after filtering the invalid sentences includes: For the target text after filtering the invalid sentences, first entity words are extracted based on named entity recognition technology, and the first entity words are input into a pre-trained Word2vec neural network model to obtain a first word vector for each first entity word, and the center point of the first word vector is determined, and the first entity word corresponding to the second word vector is determined as an invalid word, wherein the second word vector is the first preset number of first word vectors that are farthest from the center point.
2. The method according to claim 1, It is characterized in that The filtering the target text according to the first cosine similarity comprises: Determining whether the first cosine similarity is less than a first threshold; If the first cosine similarity is less than a first threshold, the target text is filtered.
3. The method according to claim 1, It is characterized in that The step of determining the average vector of the first word group vectors as the first vector and determining the average vector of the second word group vectors as the second vector comprises: Determine an average vector of the first vector and the second vector as a third vector; The target text and the third vector are combined and saved in a preset database.
4. The method according to claim 3, It is characterized in that The step of combining the target text and the third vector and saving them in a preset database includes: Obtaining a fourth vector of the comparison text in the preset database; Determining a second cosine similarity between the third vector and the fourth vector; The similarity between the target text and the comparison text is determined according to the second cosine similarity.
5. The method according to claim 3, It is characterized in that The step of combining the target text and the third vector and saving them in a preset database includes: Acquire a third preset number of third keywords input by a user, where the third preset number is the sum of the first preset number and the second preset number; Inputting the third keyword into a pre-trained neural network model to obtain a third phrase vector; Determine an average vector of the third phrase vectors as a fifth vector; A search text is determined from the preset database, wherein a third cosine similarity between a sixth vector of the search text and the fifth vector is greater than a second threshold.
6. The method according to claim 1, It is characterized in that The neural network model is a Word2vec model.
7. A text filtering device, It is characterized in that include: A text acquisition module, used to acquire a target text and a text title of the target text; A keyword extraction module, configured to extract a first preset number of first keywords from the target text, and extract a second preset number of second keywords from the text title, wherein the first preset number and the second preset number are defined by a user; The first keyword is extracted by using TF-IDF technology, and the second keyword is extracted by using LSA model or LDA model; a vector generation module, configured to input the first keyword into a pre-trained neural network model to obtain a first phrase vector, and input the second keyword into a pre-trained neural network model to obtain a second phrase vector; a vector averaging module, configured to determine an average vector of the first phrase vectors as a first vector, and determine an average vector of the second phrase vectors as a second vector; A similarity determination module, configured to determine a first cosine similarity between the first vector and the second vector; A text filtering module, used for filtering the target text according to the first cosine similarity; An invalid sentence filtering module, for determining invalid sentences in each paragraph of the filtered target text, and filtering the invalid sentences; An invalid word filtering module, for the target text after filtering the invalid sentences, determines invalid words in the target text after filtering the invalid sentences, obtains a target display text, and displays the target display text; The invalid sentence filtering module is specifically used for: Input two by two sentences in the same paragraph of the filtered target text into the pre-trained twin neural network model in turn to obtain the similarity between the sentences in the same paragraph, and obtain the similarity of each sentence in the same paragraph based on the similarity between the sentences in the same paragraph, and finally determine the invalid sentences based on the similarity of each sentence in the same paragraph; The invalid word filtering module is specifically used for: For the target text after filtering the invalid sentences, first entity words are extracted based on named entity recognition technology, and the first entity words are input into a pre-trained Word2vec neural network model to obtain a first word vector for each first entity word, and the center point of the first word vector is determined, and the first entity word corresponding to the second word vector is determined as an invalid word, wherein the second word vector is the first preset number of first word vectors that are farthest from the center point.
8. A computer device, It is characterized in that include: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 6.
9. A computer-readable storage medium having a computer program stored thereon, It is characterized in that When the program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.
