Text processing method, device, computer equipment and storage medium

Abstract

The present application discloses a text processing method, device, computer equipment and storage medium, which are applied to the field of artificial intelligence technology, wherein the method comprises: after obtaining a reference text sent by a client terminal, parsing the reference text to obtain a part-of-speech sequence of the reference text, and then determining a feature vector of the reference text through a word vector dictionary; then determining N to-be-recalled texts from a text database according to the part-of-speech sequence of the reference text; then determining the feature vectors of the N to-be-recalled texts through a word vector dictionary, and clustering the obtained N feature vectors, and determining a reference feature vector set from the N feature vectors according to the clustering result; then determining a target recall text from the N to-be-recalled texts according to the reference feature vector set and the feature vector of the reference text. By using this method, the efficiency of text recall and the accuracy of text recall can be improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a text processing method, apparatus, computer equipment and storage medium. Background Art

[0002] With the rapid development of the big data era, a large amount of text has emerged on the Internet, including short texts and long articles. Therefore, how to enable users to quickly obtain the required texts has become one of the research hotspots in various fields.

[0003] At present, the text recall strategy usually obtains the target text to be recalled through manual review. For this type of method, there are two problems. On the one hand, the high degree of manual participation leads to low processing efficiency. On the other hand, the accuracy of the recalled text is low. Therefore, how to improve the accuracy and efficiency of text matching has become a technical problem that needs to be solved urgently. Summary of the invention

[0004] The embodiments of the present application provide a text processing method, apparatus, computer device, and storage medium, which can improve the efficiency of text recall and also improve the accuracy of text recall.

[0005] On the one hand, an embodiment of the present application discloses a text processing method, the method comprising:

[0006] Acquire a reference text, and determine a part-of-speech sequence and a feature vector of the reference text;

[0007] Determine N to-be-recalled texts from a text database according to the part-of-speech sequence of the reference text, wherein the part-of-speech sequence corresponding to each of the N to-be-recalled texts matches the part-of-speech sequence of the reference text, wherein N is a positive integer;

[0008] Determine a feature vector of each of the N texts to be recalled;

[0009] Performing clustering processing on the obtained N feature vectors, and determining a reference feature vector set from the N feature vectors according to the clustering processing result;

[0010] According to the reference feature vector set and the feature vector of the reference text, a target recalled text is determined from the N to-be-recalled texts.

[0011] The present application discloses a text processing device, which includes:

[0012] An acquisition unit, used to acquire a reference text and determine a part-of-speech sequence and a feature vector of the reference text;

[0013] A determination unit, configured to determine N to-be-recalled texts from a text database according to the part-of-speech sequence of the reference text, wherein the part-of-speech sequence corresponding to each of the N to-be-recalled texts matches the part-of-speech sequence of the reference text, wherein N is a positive integer;

[0014] The determining unit is further used to determine a feature vector of each of the N to-be-recalled texts;

[0015] A processing unit, configured to perform clustering processing on the obtained N feature vectors, and determine a reference feature vector set from the N feature vectors according to the clustering processing result;

[0016] The determination unit is further configured to determine a target recalled text from the N to-be-recalled texts based on the reference feature vector set and the feature vector of the reference text.

[0017] On the one hand, an embodiment of the present application discloses a computer device, which includes: a processor, suitable for implementing one or more computer programs; and a computer storage medium, wherein the computer storage medium stores one or more computer programs, and the one or more computer programs are suitable for being loaded by the processor and executing the above-mentioned text processing method.

[0018] In one aspect, the present application discloses a computer-readable storage medium storing one or more computer programs, wherein the one or more computer programs are suitable for being loaded by a processor and executing the above-mentioned text processing method.

[0019] In one aspect, the present application discloses a computer program product, which includes a computer program stored in a computer-readable storage medium. A processor of a computer device reads the computer program from the computer-readable storage medium, and the processor executes the computer program, so that the server executes the above-mentioned text processing method.

[0020] In an embodiment of the present application, after obtaining the reference text sent by the client terminal, the computer device parses the reference text to obtain the part-of-speech sequence of the reference text, and determines the feature vector of the reference text through a word vector dictionary; the computer device then determines N to-be-recalled texts that match the part-of-speech sequence of the reference text from a text database based on the part-of-speech sequence of the reference text, and determines the feature vectors of the N to-be-recalled texts through a word vector dictionary, clusters the obtained N feature vectors, and determines a reference feature vector set from the N feature vectors based on the clustering results; and then determines a target recall text from the N to-be-recalled texts based on the reference feature vector set and the feature vector of the reference text. The number of target recall texts can be one or more. Through this method, on the one hand, manpower can be effectively reduced, automation capabilities can be improved, and the efficiency of text recall can be improved; on the other hand, the to-be-recalled texts are screened twice through part-of-speech sequences and clustering, and then the target recall text is determined by comparing the feature vectors with the reference texts, thereby improving the accuracy of text recall. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0022] Figure 1 It is a schematic diagram of the architecture of a text processing system disclosed in an embodiment of the present application;

[0023] Figure 2 It is a flowchart of a text processing method disclosed in an embodiment of the present application;

[0024] Figure 3 is a schematic diagram of a reference text and part-of-speech sequence disclosed in an embodiment of the present application;

[0025] Figure 4 It is a flowchart of another text processing method disclosed in an embodiment of the present application;

[0026] Figure 5 It is a flowchart of another text processing method disclosed in an embodiment of the present application;

[0027] Figure 6 is a structural schematic diagram of a text processing device disclosed in an embodiment of the present application;

[0028] Figure 7 It is a structural schematic diagram of a computer device disclosed in an embodiment of the present application. DETAILED DESCRIPTION

[0029] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0030] At present, the method of obtaining the target text to be recalled completely by manual review has low processing efficiency and accuracy. Based on this, when recalling the target text to be recalled, the computer device can first preliminarily screen the hot events and the matching texts in the database, and then manually verify the matching texts preliminarily screened to determine the final target recall text. This can improve the processing efficiency and accuracy of text recall to a certain extent. In order to further improve the processing efficiency and accuracy of text recall, the embodiment of the present application proposes a text processing method based on natural language processing and machine learning technology in artificial intelligence technology.

[0031] Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that machines have the functions of perception, reasoning and decision-making.

[0032] Artificial intelligence technology is a comprehensive discipline that covers a wide range of fields, including both hardware-level and software-level technologies. The basic technologies of artificial intelligence generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics and other technologies. Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0033] The present application relates to natural language processing technology and machine learning under artificial intelligence technology. Among them, natural language processing (NLP) is an important direction in the fields of computer science and artificial intelligence. It studies various theories and methods that can realize effective communication between people and computers using natural language. Natural language processing is a science that integrates linguistics, computer science, and mathematics. Therefore, research in this field will involve natural language, that is, the language used by people in daily life, so it is closely related to the study of linguistics. Natural language processing technology usually includes technologies such as text processing, semantic understanding, machine translation, robot question and answer, knowledge graph, etc. Machine learning (ML) is a multi-field interdisciplinary subject involving probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory, etc. It specializes in how computers simulate or realize human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications are spread across all fields of artificial intelligence. Machine learning and deep learning usually include technologies such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and self-learning.

[0034] Based on natural language processing and machine learning technologies in artificial intelligence technology, a text processing method proposed in the present application may specifically include: when a user searches for a text based on a keyword, the server first obtains the keyword sent by the client corresponding to the user, that is, the reference text, and then the server uses natural language processing technology to parse the reference text to obtain the part-of-speech sequence corresponding to the reference text, and then uses the word vector dictionary obtained by machine learning to determine the feature vector of the reference text; the server obtains the text matching the subject information from the text database based on the subject information of the reference text, and then compares the part-of-speech sequence corresponding to the matched text with the part-of-speech sequence of the reference text, and then determines the matched text as the text to be recalled, and the number of the text to be recalled is multiple; further, the word vector dictionary is used to determine the feature vector of the text to be recalled, and after the feature vectors of the text to be recalled are clustered, a reference feature vector set is determined from the clustering results, and finally, each reference feature vector in the reference feature vector set is compared with the feature vector of the reference text, and the text to be recalled corresponding to the reference feature vector whose matching value is greater than or equal to the specified threshold is determined as the target recall text. This application uses natural language processing technology and machine learning technology to determine the target recall articles based on the part-of-speech sequence of the reference text, clustering algorithm, etc. Therefore, this method improves the accuracy of the recalled text to a certain extent. At the same time, after clustering, the text is screened, which can effectively reduce labor costs and improve the efficiency of text recall.

[0035] Based on the above brief description of the text processing method, the present application embodiment discloses a schematic diagram of the architecture of a text processing system, see Figure 1 , the text processing system may at least include: one or more client terminals 101 and computer devices 102. The client terminal 101 refers to the user terminal used for searching, and the user inputs the reference text in the interface of the client terminal 101; the computer device 102 refers to the device that is connected to the client terminal 101 for communication, and is mainly used to receive the search request of the client terminal 101, determine the target recall text, and return the target recall text to the client terminal 101. The client terminal 101 and the computer device 102 in the text processing system can achieve communication connection, and the corresponding connection mode includes wired connection and wireless connection, which is not limited.

[0036] It should be noted that the computer equipment mentioned above can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. The client terminals mentioned above can be smart phones, tablet computers, laptops, desktop computers, smart speakers, smart watches, smart cars, etc., but are not limited to them.

[0037] Figure 1 The system architecture of the text processing system is merely exemplified and is not intended to be limiting. Figure 1 The client terminal 101 is directly connected to the computer device 102, but in other embodiments, the client terminal 101 may also be connected to the local server first, and the local server is then connected to the computer device 102, which is not limited to this. For example, in other embodiments, the above-mentioned computer device can also be deployed as a node in the blockchain network, or the computer device can be connected to the blockchain network, so that the computer device can upload internal data to the blockchain network for storage to prevent the internal data from being tampered with, thereby ensuring data security.

[0038] In a specific implementation, the general process of implementing the text processing method proposed in the embodiment of the present application through the above-mentioned text processing system is as follows: after the computer device obtains the reference text sent by the client terminal, it parses the reference text to obtain the part-of-speech sequence of the reference text, and then determines the feature vector of the reference text through the word vector dictionary; then, according to the part-of-speech sequence of the reference text, N to-be-recalled texts are determined from the text database, wherein each of the N to-be-recalled texts has a part-of-speech sequence that matches the reference text; the computer device then determines the feature vectors of the N to-be-recalled texts through the word vector dictionary, and clusters the obtained N feature vectors, and determines a reference feature vector set from the N feature vectors according to the clustering results; then, according to the reference feature vector set and the feature vector of the reference text, a target recall text is determined from the N to-be-recalled texts, the number of target recall texts may be one or more, and the target recall text has a high degree of similarity with the reference text, that is, the event theme described in the target recall text is relatively similar to the event theme described in the reference text. Furthermore, the computer device also sends the target recall text to the client terminal, so that the client terminal can display the target recall text on the interface for the user to view.

[0039] Based on the above description, the text processing method provided in the embodiment of the present application can achieve the following effects: on the one hand, the text to be recalled is determined by the part-of-speech sequence of the reference text, and then the feature vector of the text to be recalled is processed by clustering, and the reference feature vector is determined, which can effectively reduce manpower, improve automation capabilities, and improve the efficiency of text recall; on the other hand, the text to be recalled is screened twice by part-of-speech sequence and clustering, and then the target recalled text is determined by comparing the feature vector with the reference text, thereby improving the accuracy of text recall.

[0040] Based on the text processing system described above, the present application embodiment discloses a text processing method, see Figure 2 , is a flowchart of a text processing method disclosed in an embodiment of the present application, and the text processing method can be executed by a computer device. The flowchart of the text processing method can specifically include steps S201 to S205:

[0041] S201, obtaining a reference text, and determining a part-of-speech sequence and a feature vector of the reference text.

[0042] Among them, the reference text sent by the client is obtained. The reference text can be determined by the client according to the user operation, or according to the characters entered by the user in the terminal interface of the client terminal, indicating that the user wants to actively obtain the article corresponding to the reference text. The characters entered by the user can be determined according to the theme information of the hot articles, hot topics and other article types recommended in the daily information flow. Among them, hot articles and hot articles can refer to articles with a large number of searches on the Internet, such as the blog homepage showing historical searches, the microblog homepage showing historical searches, etc. In some feasible embodiments, the reference text can also be updated in real time by the computing device based on the background data, specifically determined according to the frequency of occurrence of characters in the statistical data. The higher the frequency, the more likely it is to be used as a reference text by the computer device. For example, in some application scenarios, such as real-time news push, the computer device actively pushes it to the client page. Whether it is determined based on the input characters or the reference text is determined in real time based on the background data, the sources of the characters of both characters mostly have a unified or regular speech system, or the law of context association. These features can be similarly represented by part-of-speech sequences. By matching the part-of-speech sequences, the computer device can filter out reference texts from multiple hot articles.

[0043] In one embodiment, after obtaining the reference text sent by the client terminal, the computer device can use a part-of-speech analysis tool to parse the part-of-speech sequence of the reference text. The parts of speech may include noun part-of-speech, verb part-of-speech, adverb part-of-speech, adjective part-of-speech, personal noun part-of-speech, etc., which are not listed here one by one. For example, the reference text and the part-of-speech sequence of the reference text can be as follows: Figure 3 As shown, 301 is a reference text, 302 is a part-of-speech sequence of the reference text, where "nr" is a name, "n" is a noun, "v" is a verb, and "c" is a conjunction. Part-of-speech analysis tools may include jieba ("Jieba" Chinese word segmentation, a Python word segmentation component), SnowNLP (Simplified Chinese Text Processing, a class library written in Python), THULAC (THU Lexical Analyzer for Chinese, a set of Chinese lexical analysis toolkits developed and launched by the Natural Language Processing and Social Humanities Computing Laboratory of Tsinghua University, with Chinese word segmentation and part-of-speech tagging functions), etc.

[0044] Furthermore, the server will also perform word segmentation processing on the reference text to obtain each word of the reference text, and then search the word vector dictionary for the feature vector corresponding to each word segment of the reference text, and then fuse the feature vectors of each word segment of the reference text to obtain the feature vector corresponding to the reference text. The fusion method can be accumulation. For example, after the word segmentation processing of reference text 1, word segmentation A, word segmentation B and word segmentation C are obtained. According to the word vector dictionary, it is determined that the feature vector a corresponding to word segmentation A is a 100-dimensional vector, the feature vector b corresponding to word segmentation B is also a 100-dimensional vector, and the feature vector c corresponding to word segmentation C is also a 100-dimensional vector. Then, the feature vector a, feature vector b and feature vector c are accumulated to obtain a new 100-dimensional vector, and then the vector is normalized to obtain the feature vector of reference text 1.

[0045] Among them, the word vector dictionary can be trained in advance by the server, and its training process may include: first obtain a large number of sample data sets from the text database, the sample data set includes multiple training texts, assuming that the number of training texts is 1 million, then after obtaining 1 million training texts, first perform word segmentation on the training text to obtain the word segmentation results of each training text, and then use word2vec to train the word segmentation results of the 1 million training texts to obtain the vector representation of each word segmentation, and at the same time filter out some useless data, such as useless word segmentation vectors, and finally obtain a version of the word vector dictionary corresponding to the 1 million training texts, and the word vector dictionary includes the vector corresponding to each word segmentation. In an embodiment of the present application, we can take the dimension of the vector as 100 dimensions, so the word vector dictionary obtained is a vector set including many 100-dimensional vectors, one vector corresponding to one word segmentation. The word vector dictionary is the basis for obtaining the feature vectors of each text to be recalled later. Among them, word2vec (word to vector) is a group of related models used to generate word vectors. These models are shallow and double-layer neural networks used for training to reconstruct the word text of linguistics. Under the assumption of the bag-of-words model in word2vec, the order of words is unimportant. After training, the word2vec model can be used to map each word to each vector. In addition to word2vec, you can also use GloVe (Global Vectors for Word Representation), which is a word representation tool based on global word frequency statistics. It can express a word as a vector composed of real numbers. These vectors capture some semantic characteristics between words, such as similarity, analogy, etc. We can calculate the semantic similarity between two words by operating on vectors, such as Euclidean distance or cosine similarity. Since GloVe is more inclined to determine the semantic similarity between two words, the embodiment of the present application chooses to use word2vec to directly obtain the mapping relationship between each word and each vector.

[0046] S202: Determine N texts to be recalled from a text database according to the part-of-speech sequence of the reference text.

[0047] In one embodiment, after obtaining the part-of-speech sequence of the reference text in step S201, in order to realize the text recall of the present application, some articles are needed for testing to check whether the recall effect can be achieved. Therefore, it is also necessary to determine N texts to be recalled from the text database, and the specific implementation method is: first determine the subject information of the reference text, and determine one or more texts to be selected from the text database whose subject information matches the subject information of the reference text. Among them, the subject information here can be a subject word. For example, if the reference text is "Zhang XX has a blind date in a certain place", the corresponding subject words can be the name "Zhang XX", the verb "blind date" and the noun "a certain place". Therefore, one or more texts to be selected can be obtained from the text database according to the subject words. These texts to be selected include at least the same subject words as the reference text. Corresponding examples, the texts to be selected either include "Zhang XX" and "a certain place", or include "blind date" and "a certain place", or include "Zhang XX", "blind date" and "a certain place" at the same time, and so on.

[0048] Furthermore, after determining one or more texts to be selected, in order to reduce the amount of subsequent text processing and improve efficiency, it is necessary to further screen the texts to be selected, that is, the part-of-speech sequence of the reference text can be used to screen the texts to be selected, so as to determine N texts to be recalled, and the part-of-speech sequence corresponding to each of the N texts to be recalled matches the part-of-speech sequence of the reference text. The matching indicated here can refer to the part-of-speech sequence of each text to be recalled and the reference text being the same or having a high degree of similarity. In the embodiment of the present application, N can be any positive integer.

[0049] S203: Determine a feature vector of each of the N texts to be recalled.

[0050] In one embodiment, for any of the N texts to be recalled, the processing process of the computer device can be the same. Specifically, the computer device can perform word segmentation processing on the text to be recalled to obtain a word segmentation set, determine the feature vector of each word segmentation in the word segmentation set according to the word vector dictionary, and fuse the feature vectors of each word segmentation. The specific fusion method can be summation, and then determine the feature vector of the text to be recalled according to the fusion result. Corresponding to the above example, each word segmentation in the word vector dictionary obtained corresponds to a 100-dimensional vector. Therefore, for the word segmentation set of each text to be recalled, a 100-dimensional vector set will be obtained. Specifically, the 100-dimensional vector of each word segmentation in the set can be accumulated to obtain a 100-dimensional feature vector. At this time, each value of this 100-dimensional feature vector is very large. Therefore, it is necessary to normalize the feature vector to obtain a feature vector of the text to be recalled. The processing steps for each text to be recalled in the N texts to be recalled are consistent. Therefore, after processing, N feature vectors can be obtained, and each feature vector corresponds to a text to be recalled.

[0051] S204 , clustering the obtained N feature vectors, and determining a reference feature vector set from the N feature vectors according to the clustering result.

[0052] In one embodiment, in order to reduce the subsequent manual verification of the text, the computer device can cluster the obtained N feature vectors to obtain at least one cluster category, determine the target cluster category from the at least one cluster category, and determine the reference feature vector set according to the feature vectors included in each target cluster category. In the embodiment of the present application, k-means can be used to cluster the N feature vectors. K-means is our most commonly used clustering algorithm based on Euclidean distance. This type of algorithm believes that the closer the distance between two targets, the greater the similarity. Assume k = 100, that is, the N feature vectors are divided into 100 cluster categories, each cluster category can include a different number of feature training, then the computer device determines the number of feature vectors included in each cluster category in at least one cluster category, and sorts the cluster categories in order from high to low according to the number of feature vectors included, and obtains the first sorting result, that is, the cluster category ranked first includes the most feature vectors, and the cluster category ranked 100 includes the least feature vectors. Further, the computer device determines the clustering categories ranked in the top M positions in at least one clustering category as the target clustering category according to the first sorting result. It can be understood that there are multiple target clustering categories, and M can take any positive value. Assuming that M is 20, 20 target clustering categories are selected. Among them, in addition to the k-means clustering algorithm, the clustering algorithm can also adopt mean shift clustering, density-based clustering, maximum expectation clustering using a Gaussian mixture model, or agglomerative hierarchical clustering, etc.

[0053] Furthermore, the computer device uses the feature vectors included in the target cluster category as reference feature vectors. Since the number of reference feature vectors obtained in the embodiment of the present application may be more than one, a reference feature vector set can better represent the number of feature vectors.

[0054] S205 . Determine a target recalled text from N to-be-recalled texts according to the reference feature vector set and the feature vector of the reference text.

[0055] In one embodiment, the computer device may compare each reference feature vector in the reference feature vector set with the feature vector of the reference text, thereby obtaining a matching value between each reference feature vector and the feature vector of the reference text, and then determine from the reference feature vector set a target feature vector whose matching value with the feature vector of the reference text is greater than or equal to a set threshold, and then determine the to-be-recalled text corresponding to the target feature vector as the target recall text. The set threshold may be set a priori or according to business requirements.

[0056] In another embodiment, the computer device can sort the target recall texts in the order of the matching values ​​from large to small according to the matching values ​​between the target feature vectors corresponding to the target recall texts and the feature vectors of the reference texts, and obtain a second sorting result, and then send the second sorting result and the target recall texts to the client, so that the client determines the display order of the target recall texts according to the second sorting result, and displays the target recall texts in the display order. It should be noted that the target recall text can refer to the summary information of the target recall text, or the central idea sentence of the target recall text, or the text title of the target recall text. In order to fully display the number of target recall texts on the display interface of the client terminal, a part of the content of the target recall text is displayed. When the number of target recall texts is too large, it can also be displayed through different pages, that is, one page displays a part of the target recall text, and when the next page is clicked, other target recall texts are displayed.

[0057] In an embodiment of the present application, after obtaining a reference text sent by a client terminal, the computer device parses the reference text to obtain a part-of-speech sequence of the reference text, and then determines a feature vector of the reference text through a word vector dictionary; then, based on the part-of-speech sequence of the reference text, N texts to be recalled are determined from a text database, wherein each of the N texts to be recalled has a part-of-speech sequence that matches the reference text; the computer device then determines the feature vectors of the N texts to be recalled through a word vector dictionary, and clusters the obtained N feature vectors, and determines a reference feature vector set from the N feature vectors based on the clustering results; then, based on the reference feature vector set and the feature vector of the reference text, a target recall text is determined from the N texts to be recalled, the number of target recall texts may be one or more, and the target recall text has a high degree of similarity to the reference text, that is, the event theme described in the target recall text is relatively similar to the event theme described in the reference text. Through this method, on the one hand, manpower can be effectively reduced, automation capabilities can be improved, and the efficiency of text recall can be improved; on the other hand, the recalled text is screened twice through part-of-speech sequence and clustering, and then the target recalled text is determined by comparing the feature vector with the reference text, thereby improving the accuracy of text recall.

[0058] Based on the above-described text processing system and text processing method, the present application embodiment discloses another text processing method, see Figure 4 , is a flowchart of another text processing method disclosed in an embodiment of the present application, and the text processing method can be executed by a computer device. The text processing method can specifically include steps 401 to 412:

[0059] S401: Obtain a reference text, and determine a part-of-speech sequence and a feature vector of the reference text.

[0060] S402: Determine subject information of a reference text, and determine from a text database one or more texts to be selected whose subject information matches the subject information of the reference text.

[0061] S403 : Determine N texts to be recalled from one or more texts to be selected according to the part-of-speech sequence of the reference text.

[0062] S404 . For any one of the N texts to be recalled, perform word segmentation processing on the text to be recalled to obtain a word segmentation set.

[0063] S405. Determine the feature vector of each word in the word segmentation set according to the word vector dictionary.

[0064] S406: Fuse the feature vectors of each word segmentation, and determine the feature vector of the text to be recalled according to the fusion result.

[0065] S407: Cluster the obtained N feature vectors to obtain at least one cluster category.

[0066] S408: Determine a target clustering category from at least one clustering category, and determine a reference feature vector set according to feature vectors included in each target clustering category.

[0067] S409: Compare each reference feature vector in the reference feature vector set with the feature vector of the reference text, and determine a matching value between each reference feature vector and the feature vector of the reference text.

[0068] S410 , determining a target feature vector from a reference feature vector set, the target feature vector having a matching value with a feature vector of a reference text being greater than or equal to a set threshold, and determining the to-be-recalled text corresponding to the target feature vector as the target recalled text.

[0069] S411. According to the matching value between the target feature vector corresponding to each target recall text and the feature vector of the reference text, sort the target recall texts in descending order of the matching value to obtain a second sorting result.

[0070] S412: Send the second sorting result and each target recall text to the client, so that the client determines the display order of each target recall text according to the second sorting result, and displays each target recall text according to the display order.

[0071] Among them, steps 401 to 412 are more detailed descriptions of the relevant steps corresponding to the text processing method provided in the embodiment of the present application. The specific embodiment has been described in Figure 2 The relevant embodiments shown have been described in detail and will not be repeated here.

[0072] In one embodiment, Figure 4 The text processing method described can be specifically divided into three major parts. In the first part, the computer device determines the reference text and the part-of-speech sequence of the reference text, and uses the part-of-speech sequence of the reference text to screen out N texts to be recalled; in the second part, the computer device uses the training text to train a version of the word vector dictionary, and then uses the word vector dictionary to obtain the feature vector of the reference text and the feature vectors of the N texts to be recalled; in the third part, the computer device clusters the N feature vectors of the feature vectors of the N texts to be recalled, and then uses the feature vector of the reference text as a reference standard to determine the target recall text from the clustering results, and sends the target recall text to the client, so that the client can display the target recall text in the interface.

[0073] Specifically, the third part can be Figure 5As shown, it includes N feature vectors of N texts to be recalled, such as for text 1 to be recalled, the corresponding feature vector is feature vector 1, and for text N to be recalled, the corresponding feature vector is feature vector N, and then the N feature vectors are clustered, and then the target clustering category is determined from the clustering results, and then the reference feature vector features included in the target clustering category are compared with the feature vectors of the reference text to determine the target recalled text.

[0074] In an embodiment of the present application, a computer device determines a reference text and determines a part-of-speech sequence of the reference text, and determines N texts to be recalled using the part-of-speech sequence of the reference text; obtains a feature vector of the reference text and a feature vector of the N texts to be recalled using a word vector dictionary; clusters the N feature vectors of the feature vectors of the N texts to be recalled, and then uses the feature vector of the reference text as a reference standard to determine the target recall text from the clustering results. Through this method, on the one hand, the text to be recalled is determined by the part-of-speech sequence of the reference text, and then the feature vector of the text to be recalled is processed by clustering, and a reference feature vector is determined, which can effectively reduce manpower, improve automation capabilities, and improve the efficiency of text recall; on the other hand, the text to be recalled is screened twice by part-of-speech sequence and clustering, and then the target recall text is determined by comparing the feature vector with the reference text, so the accuracy of text recall can be effectively improved.

[0075] Based on the above method embodiment, the present application embodiment also provides a structural diagram of a text processing device. Figure 6 , is a structural diagram of a text processing device provided in an embodiment of the present application. Figure 6 The text processing device 600 shown can run the following units:

[0076] An acquisition unit 601 is used to acquire a reference text and determine a part-of-speech sequence and a feature vector of the reference text;

[0077] A determination unit 602 is used to determine N to-be-recalled texts from a text database according to the part-of-speech sequence of the reference text, wherein the part-of-speech sequence corresponding to each of the N to-be-recalled texts matches the part-of-speech sequence of the reference text, wherein N is a positive integer;

[0078] The determining unit 602 is further configured to determine a feature vector of each of the N to-be-recalled texts;

[0079] A processing unit 603 is used to perform clustering processing on the obtained N feature vectors, and determine a reference feature vector set from the N feature vectors according to the clustering processing result;

[0080] The determining unit 602 is further configured to determine a target recalled text from the N to-be-recalled texts according to the reference feature vector set and the feature vector of the reference text.

[0081] In one embodiment, the processing unit 603 performs clustering processing on the obtained N feature vectors, and determines a reference feature vector set from the N feature vectors according to the clustering processing result, which can be specifically used for:

[0082] Clustering the obtained N feature vectors to obtain at least one cluster category;

[0083] A target cluster category is determined from the at least one cluster category, and a reference feature vector set is determined according to feature vectors included in each target cluster category.

[0084] In an embodiment, the determining unit 602 determines the target cluster category from the at least one cluster category, which may be specifically used to:

[0085] Determining the number of feature vectors included in each cluster category of the at least one cluster category;

[0086] Sorting the at least one clustering category in descending order of the number of the included feature vectors to obtain a first sorting result;

[0087] According to the first sorting result, a cluster category ranked in the top M positions in the at least one cluster category is determined as a target cluster category, where M is a positive integer.

[0088] In one embodiment, the determining unit 602 determines the target recalled text from the N to-be-recalled texts according to the reference feature vector set and the feature vector of the reference text, which can be specifically used to:

[0089] Comparing each reference feature vector in the reference feature vector set with the feature vector of the reference text, and determining a matching value between each reference feature vector and the feature vector of the reference text;

[0090] Determine, from the reference feature vector set, a target feature vector whose matching value with the feature vector of the reference text is greater than or equal to a set threshold;

[0091] The to-be-recalled text corresponding to the target feature vector is determined as the target recalled text.

[0092] In one embodiment, there are multiple target feature vectors, and the processing unit 603 is further specifically configured to:

[0093] According to the matching value between the target feature vector corresponding to each target recall text and the feature vector of the reference text, the target recall texts are sorted in descending order of the matching value to obtain a second sorting result;

[0094] The second sorting result and the target recall texts are sent to the client, so that the client determines the display order of the target recall texts according to the second sorting result, and displays the target recall texts according to the display order.

[0095] In one embodiment, the determining unit 602 determines the feature vector of each of the N to-be-recalled texts, which can be specifically used to:

[0096] For any one of the N to-be-recalled texts, perform word segmentation processing on the to-be-recalled text to obtain a word segmentation set;

[0097] Determine the feature vector of each word in the word set according to the word vector dictionary;

[0098] The feature vectors of the various word segments are fused, and the feature vector of the text to be recalled is determined according to the fusion result.

[0099] In one embodiment, the determining unit 602 determines N to-be-recalled texts from a text database according to the part-of-speech sequence of the reference text, which can be specifically used to:

[0100] Determining subject information of the reference text, and determining one or more texts to be selected whose subject information matches the subject information of the reference text from a text database;

[0101] N texts to be recalled are determined from the one or more texts to be selected according to the part-of-speech sequence of the reference text.

[0102] According to one embodiment of the present application, Figure 2 as well as Figure 4 The steps involved in the text processing method shown can be Figure 6 The text processing device 600 shown in FIG. 6 is executed by various units. For example, Figure 2 In the text processing method shown in FIG. 1 , step S201 can be performed by Figure 6 The acquisition unit 601 in the text processing device 600 shown in FIG. 1 is used to perform the steps S202 to S203 and S205. Figure 6 The text processing device 600 shown in FIG. 6 is used to determine the unit 602, and step S204 can be performed by Figure 6 The processing unit 603 in the text processing device 600 shown in the figure is used for execution; for example, Figure 4In the text processing method shown in FIG. 1 , step S401 can be performed by Figure 6 The acquisition unit 601 in the text processing device 600 shown in FIG. 1 is used to perform the steps S402-S403, S405-S406, and S408-S411. Figure 6 The text processing device 600 shown in FIG. 6 is executed by the determining unit 602, and steps S404, S407 and S412 can be performed by Figure 6 The text processing unit 603 in the text processing device 600 is shown to execute.

[0103] According to another embodiment of the present application, Figure 6 The various units in the text processing device 600 shown can be separately or completely combined into one or several other units to form, or one (some) of the units can be further divided into multiple functionally smaller units to form, which can achieve the same operation without affecting the realization of the technical effects of the embodiments of the present application. The above-mentioned units are divided based on logical functions. In practical applications, the functions of one unit can also be implemented by multiple units, or the functions of multiple units can be implemented by one unit. In other embodiments of the present application, other units can also be included based on the text processing device 600. In practical applications, these functions can also be implemented with the assistance of other units, and can be implemented by the collaboration of multiple units.

[0104] According to another embodiment of the present application, the program can be executed by running on a general computing device such as a computer including a central processing unit (CPU), a random access memory medium (RAM), a read-only memory medium (ROM), and other processing elements and storage elements. Figure 2 as well as Figure 4 A computer program (including program code) for each step involved in the corresponding method shown in FIG. Figure 6 The text processing apparatus 600 shown in the figure can be used to implement the text processing method of the embodiment of the present application. The computer program can be recorded on a computer-readable storage medium, for example, and loaded into the computing device through the computer-readable storage medium and run therein.

[0105] In an embodiment of the present application, after obtaining the reference text sent by the client terminal, the reference text is parsed to obtain the part-of-speech sequence of the reference text, and then the feature vector of the reference text is determined through a word vector dictionary; then, N texts to be recalled are determined from a text database based on the part-of-speech sequence of the reference text, wherein each of the N texts to be recalled has a part-of-speech sequence that matches the reference text; the computer device then determines the feature vectors of the N texts to be recalled through a word vector dictionary, and clusters the obtained N feature vectors, and determines a reference feature vector set from the N feature vectors based on the clustering results; then, based on the reference feature vector set and the feature vector of the reference text, a target recall text is determined from the N texts to be recalled, and the number of target recall texts can be one or more. Through this method, on the one hand, manpower can be effectively reduced, automation capabilities can be improved, and the efficiency of text recall can be improved; on the other hand, the text to be recalled is screened twice through part-of-speech sequence and clustering, and then the target recall text is determined by comparing the feature vector with the reference text, thereby improving the accuracy of text recall.

[0106] Based on the above method and device embodiments, the present application embodiment provides a computer device. Figure 7 , is a structural diagram of a computer device provided in an embodiment of the present application. Figure 7 The computer device 700 shown includes at least a processor 701, an input interface 702, an output interface 703, a computer storage medium 704, and a memory 705. The processor 701, the input interface 702, the output interface 703, the computer storage medium 704, and the memory 705 may be connected via a bus or other means.

[0107] The computer storage medium 704 may be stored in the memory 705 of the computer device 700, the computer storage medium 704 is used to store a computer program, the computer program includes program instructions, and the processor 701 is used to execute the program instructions stored in the computer storage medium 704. The processor 701 (or CPU (Central Processing Unit)) is the computing core and control core of the computer device 700, which is suitable for implementing one or more instructions, and is specifically suitable for loading and executing one or more computer instructions to implement the corresponding method flow or corresponding function.

[0108] The embodiment of the present application also provides a computer storage medium (Memory), which is a memory device in the computer device 700 for storing programs and data. It is understandable that the computer storage medium here can include both the built-in storage medium in the computer device 700 and the extended storage medium supported by the computer device 700. The computer storage medium provides a storage space, which stores the operating system of the computer device 700. In addition, one or more instructions suitable for being loaded and executed by the processor are also stored in the storage space, and these instructions can be one or more computer programs (including program codes). It should be noted that the computer storage medium here can be a high-speed RAM memory, or a non-volatile memory (non-volatile memory), such as at least one disk storage; optionally, it can also be at least one computer storage medium located away from the aforementioned processor.

[0109] In one embodiment, the computer storage medium can be loaded by the processor 701 and execute one or more instructions stored in the computer storage medium to implement the above-mentioned Figure 2 as well as Figure 4 In a specific implementation, one or more instructions in the computer storage medium are loaded by the processor 701 and execute the following steps:

[0110] Acquire a reference text, and determine a part-of-speech sequence and a feature vector of the reference text;

[0111] Determine N to-be-recalled texts from a text database according to the part-of-speech sequence of the reference text, wherein the part-of-speech sequence corresponding to each of the N to-be-recalled texts matches the part-of-speech sequence of the reference text, wherein N is a positive integer;

[0112] Determine a feature vector of each of the N texts to be recalled;

[0113] Performing clustering processing on the obtained N feature vectors, and determining a reference feature vector set from the N feature vectors according to the clustering processing result;

[0114] According to the reference feature vector set and the feature vector of the reference text, a target recalled text is determined from the N to-be-recalled texts.

[0115] In one embodiment, the processor 701 performs clustering processing on the obtained N feature vectors, and determines a reference feature vector set from the N feature vectors according to the clustering processing result, which can be specifically used for:

[0116] Clustering the obtained N feature vectors to obtain at least one cluster category;

[0117] A target cluster category is determined from the at least one cluster category, and a reference feature vector set is determined according to feature vectors included in each target cluster category.

[0118] In one embodiment, the processor 701 determines the target cluster category from the at least one cluster category, which may be specifically used to:

[0119] Determining the number of feature vectors included in each cluster category of the at least one cluster category;

[0120] Sorting the at least one clustering category in descending order of the number of the included feature vectors to obtain a first sorting result;

[0121] According to the first sorting result, a cluster category ranked in the top M positions in the at least one cluster category is determined as a target cluster category, where M is a positive integer.

[0122] In one embodiment, the processor 701 determines the target recalled text from the N to-be-recalled texts according to the reference feature vector set and the feature vector of the reference text, which can be specifically used to:

[0123] Comparing each reference feature vector in the reference feature vector set with the feature vector of the reference text, and determining a matching value between each reference feature vector and the feature vector of the reference text;

[0124] Determine, from the reference feature vector set, a target feature vector whose matching value with the feature vector of the reference text is greater than or equal to a set threshold;

[0125] The to-be-recalled text corresponding to the target feature vector is determined as the target recalled text.

[0126] In one embodiment, there are multiple target feature vectors, and the processor 701 is further specifically configured to:

[0127] According to the matching value between the target feature vector corresponding to each target recall text and the feature vector of the reference text, the target recall texts are sorted in descending order of the matching value to obtain a second sorting result;

[0128] The second sorting result and the target recall texts are sent to the client, so that the client determines the display order of the target recall texts according to the second sorting result, and displays the target recall texts according to the display order.

[0129] In one embodiment, the processor 701 determines the feature vector of each of the N to-be-recalled texts, which can be specifically used to:

[0130] For any one of the N to-be-recalled texts, perform word segmentation processing on the to-be-recalled text to obtain a word segmentation set;

[0131] Determine the feature vector of each word in the word set according to the word vector dictionary;

[0132] The feature vectors of the various word segments are fused, and the feature vector of the text to be recalled is determined according to the fusion result.

[0133] In one embodiment, the processor 701 determines N to-be-recalled texts from a text database according to the part-of-speech sequence of the reference text, which can be specifically used to:

[0134] Determining subject information of the reference text, and determining from a text database one or more texts to be selected whose subject information matches the subject information of the reference text;

[0135] N texts to be recalled are determined from the one or more texts to be selected according to the part-of-speech sequence of the reference text.

[0136] In the example of the present application, the computer device 700 determines the reference text and determines the part-of-speech sequence of the reference text, and determines N texts to be recalled using the part-of-speech sequence of the reference text; obtains the feature vector of the reference text and the feature vectors of the N texts to be recalled using the word vector dictionary; clusters the N feature vectors of the feature vectors of the N texts to be recalled, and then uses the feature vector of the reference text as a reference standard to determine the target recall text from the clustering results. Through this method, on the one hand, the text to be recalled is determined by the part-of-speech sequence of the reference text, and then the feature vector of the text to be recalled is processed by clustering, and the reference feature vector is determined, which can effectively reduce manpower, improve automation capabilities, and improve the efficiency of text recall; on the other hand, the text to be recalled is screened twice by the part-of-speech sequence and clustering, and then the target recall text is determined by comparing it with the feature vector of the reference text, so the accuracy of text recall can be improved.

[0137] According to one aspect of the present application, an embodiment of the present application further provides a computer product, the computer product includes a computer program, the computer program is stored in a computer-readable storage medium. The processor 701 reads the computer program from the computer-readable storage medium, and the processor 701 executes the computer program, so that the computer device 700 executes Figure 2 as well as Figure 4 The text processing method shown.

[0138] It should be noted that, for the above-mentioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the order of the actions described, because according to the present application, some steps may be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application.

[0139] In the several embodiments provided in this application, it should be understood that the disclosed device can be implemented in other ways. For example, the device embodiments described above are only schematic, and the division of the modules described above is only a logical function division. There may be other division methods in actual implementation, such as multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed.

[0140] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A text processing method, characterized in that: The method comprises: Acquire a reference text, and determine a part-of-speech sequence and a feature vector of the reference text; Determine the subject information of the reference text, determine one or more to-be-selected texts whose subject information matches the subject information of the reference text from a text database; determine N to-be-recalled texts from the one or more to-be-selected texts according to the part-of-speech sequence of the reference text; match the part-of-speech sequence corresponding to each of the N to-be-recalled texts with the part-of-speech sequence of the reference text, wherein N is a positive integer; Determine a feature vector of each of the N texts to be recalled; The obtained N feature vectors are clustered to obtain at least one cluster category; M cluster categories are determined from the at least one cluster category as target cluster categories according to the number of feature vectors included in each cluster category, and a reference feature vector set is determined according to the feature vectors included in each target cluster category, where M is a positive integer; According to the reference feature vector set and the feature vector of the reference text, a target recalled text is determined from the N to-be-recalled texts.

2. The method according to claim 1, characterized in that The determining M clustering categories as target clustering categories from the at least one clustering category according to the number of feature vectors included in each clustering category includes: Determining the number of feature vectors included in each cluster category of the at least one cluster category; Sorting the at least one clustering category in descending order of the number of the included feature vectors to obtain a first sorting result; According to the first sorting result, a cluster category ranked in the top M positions in the at least one cluster category is determined as a target cluster category.

3. The method according to claim 1, characterized in that The step of determining a target recalled text from the N to-be-recalled texts according to the reference feature vector set and the feature vector of the reference text comprises: Comparing each reference feature vector in the reference feature vector set with the feature vector of the reference text, and determining a matching value between each reference feature vector and the feature vector of the reference text; Determine, from the reference feature vector set, a target feature vector whose matching value with the feature vector of the reference text is greater than or equal to a set threshold; The to-be-recalled text corresponding to the target feature vector is determined as the target recalled text.

4. The method according to claim 3, characterized in that There are multiple target feature vectors, and the method further includes: According to the matching value between the target feature vector corresponding to each target recall text and the feature vector of the reference text, the target recall texts are sorted in descending order of the matching value to obtain a second sorting result; The second sorting result and the target recall texts are sent to the client, so that the client determines the display order of the target recall texts according to the second sorting result, and displays the target recall texts according to the display order.

5. The method according to claim 1, characterized in that The step of determining a feature vector of each of the N to-be-recalled texts comprises: For any one of the N to-be-recalled texts, perform word segmentation processing on the to-be-recalled text to obtain a word segmentation set; Determine the feature vector of each word in the word set according to the word vector dictionary; The feature vectors of the various word segments are fused, and the feature vector of the text to be recalled is determined according to the fusion result.

6. A text processing device, characterized in that: The device comprises: An acquisition unit, used to acquire a reference text and determine a part-of-speech sequence and a feature vector of the reference text; A determination unit, configured to determine the subject information of the reference text, determine one or more to-be-selected texts whose subject information matches the subject information of the reference text from a text database; determine N to-be-recalled texts from the one or more to-be-selected texts according to the part-of-speech sequence of the reference text; the part-of-speech sequence corresponding to each to-be-recalled text in the N to-be-recalled texts matches the part-of-speech sequence of the reference text, wherein N is a positive integer; The determining unit is further used to determine a feature vector of each of the N to-be-recalled texts; A processing unit, configured to perform clustering processing on the obtained N feature vectors to obtain at least one cluster category; determine M cluster categories from the at least one cluster category as target cluster categories according to the number of feature vectors included in each cluster category, and determine a reference feature vector set according to the feature vectors included in each target cluster category, wherein M is a positive integer; The determination unit is further configured to determine a target recalled text from the N to-be-recalled texts based on the reference feature vector set and the feature vector of the reference text.

7. A computer device, characterized in that: The computer device comprises: a processor adapted to implement one or more computer programs; and, A computer storage medium storing one or more computer programs, wherein the one or more computer programs are suitable for being loaded by the processor and executing the text processing method according to any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores one or more computer programs, and the one or more computer programs are suitable for being loaded by a processor and executing the text processing method according to any one of claims 1 to 5.

Images