Keyword extraction method, device, storage medium and computer equipment

Abstract

The present application relates to a keyword extraction method, device, computer-readable storage medium and computer equipment, the method comprising: sliding and extracting words from a target text according to at least two co-occurrence windows respectively, obtaining multiple word combinations under each co-occurrence window; determining the co-occurrence probability between different words in each word combination based on a word vector model; in each word combination, determining the words whose co-occurrence probability meets the screening condition as candidate words; determining the contribution probability of each candidate word to the explanation of the target text in the candidate word set composed of the candidate words of each word combination according to the conditional probability of the candidate word appearing in the corpus; determining the candidate word whose contribution probability meets the threshold condition as a keyword. The scheme provided by the present application can improve the accuracy of keyword extraction.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a keyword extraction method, device, storage medium and computer equipment. Background Art

[0002] Keywords are a group of words that represent the important content of a text, and they play an important role in text clustering, classification, automatic summarization, etc. At present, there are many mainstream keyword extraction algorithms, such as TF-IDF, TextRank, etc. Among them, TF-IDF mainly judges the importance of each word segment based on word frequency and inverse text frequency. This method only considers the information volume of a single word, which will lose the correlation information between words and reduce the accuracy of keyword extraction. TextRank judges the importance of each word segment based on the co-occurrence relationship between words in the target text. Although this method considers the correlation information between words, it only uses the information of the target text itself to achieve keyword extraction, so the accuracy of the extracted keywords is still not high. Summary of the invention

[0003] Based on this, it is necessary to provide a keyword extraction method, apparatus, computer-readable storage medium and computer device to address the technical problem of poor keyword extraction accuracy.

[0004] A keyword extraction method, comprising:

[0005] Slide and extract words from the target text according to at least two co-occurrence windows to obtain multiple word combinations under each co-occurrence window;

[0006] Determine the co-occurrence probability between different words in each word combination based on the word vector model;

[0007] In each word combination, the words whose co-occurrence probability meets the screening condition are determined as candidate words;

[0008] According to the conditional probability of the candidate words appearing in the corpus, in the candidate word set composed of the candidate words of each word combination, determining the probability of contribution of each candidate word to the explanation of the target text;

[0009] The candidate words whose contribution probabilities meet the threshold condition are determined as keywords.

[0010] A keyword extraction device, comprising:

[0011] A sliding word extraction module is used to slide words from the target text according to at least two co-occurrence windows to obtain multiple word combinations under each co-occurrence window;

[0012] A candidate word screening module is used to determine the co-occurrence probability between different words in each word combination based on the word vector model; in each word combination, the words whose co-occurrence probability meets the screening condition are determined as candidate words;

[0013] A contribution rate calculation module is used to determine the contribution probability of each candidate word to the explanation of the target text in the candidate word set composed of the candidate words of each word combination according to the conditional probability of the candidate word appearing in the corpus;

[0014] The keyword extraction module is used to determine the candidate words whose contribution probability meets the threshold condition as keywords.

[0015] A computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the processor executes the steps of the keyword extraction method.

[0016] A computer device comprises a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the keyword extraction method.

[0017] The above keyword extraction method, device, computer-readable storage medium and computer equipment can make each word in the target text appear in multiple word combinations by sliding and extracting words from the target text based on multiple co-occurrence windows. The probability of each word co-occurring with other words is measured based on the word vector model in units of word combinations, and words with large co-occurrence probability are selected as candidate words, which fully considers the relationship between words in the target text. Compared with simply considering the information content of the word itself, the accuracy of candidate word extraction can be improved; combined with the conditional probability of the candidate word appearing in the entire corpus, its contribution probability to the interpretation of the target text is evaluated, so that the calculation of the contribution probability is guided by the semantic information of the entire corpus, which is equivalent to having a benchmark conditional probability as a reference. Compared with simply considering the information content of the target text itself, it can reduce the contribution probability accuracy deviation caused by the extreme nature of the target text content, and thus improve the accuracy of selecting keywords in the candidate word set according to the contribution probability. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 An application environment diagram of a keyword extraction method in an embodiment;

[0019] Figure 2 Schematic diagram of a flow chart of a keyword extraction method in one embodiment;

[0020] Figure 3 is a schematic diagram of co-occurrence windows with different window lengths in one embodiment;

[0021] Figure 4A schematic diagram of a process for calculating co-occurrence probability based on a word vector model in one embodiment;

[0022] Figure 5 is a schematic diagram of a first Huffman tree based on a target text in one embodiment;

[0023] Figure 6 Schematic diagram of the principle of a keyword extraction method in one embodiment;

[0024] Figure 7 A schematic diagram of a process of extracting keywords from a candidate word set based on contribution probability in one embodiment;

[0025] Figure 8 Schematic diagram of a flow chart of a keyword extraction method in a specific embodiment;

[0026] Fig. 9 is a structural block diagram of a keyword extraction device in one embodiment;

[0027] Fig.10 is a structural block diagram of a keyword extraction device in one embodiment;

[0028] Fig.11 is a structural block diagram of a computer device in one embodiment;

[0029] Fig.12 It is a structural block diagram of a computer device in another embodiment. DETAILED DESCRIPTION

[0030] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0031] Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that machines have the functions of perception, reasoning and decision-making.

[0032] Artificial intelligence technology is a comprehensive discipline that covers a wide range of fields, including both hardware-level and software-level technologies. The basic technologies of artificial intelligence generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics and other technologies. Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0033] Computer Vision (CV) is a science that studies how to make machines "see". To put it more specifically, it refers to using cameras and computers to replace human eyes to identify, follow and measure targets, and further process graphics so that the computer processes the images into images that are more suitable for human observation or transmission to instruments for detection. As a scientific discipline, computer vision studies related theories and technologies, and attempts to establish an artificial intelligence system that can obtain information from images or multi-dimensional data. Computer vision technology usually includes image processing, image recognition, image semantic understanding, image retrieval, OCR, video processing, video semantic understanding, video content / behavior recognition, three-dimensional object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous positioning and map construction, and other technologies, as well as common biometric recognition technologies such as face recognition and fingerprint recognition.

[0034] The key technologies of speech technology include automatic speech recognition technology (ASR), text-to-speech technology (TTS) and voiceprint recognition technology. Enabling computers to listen, see, speak and feel is the future development direction of human-computer interaction, among which speech has become one of the most promising human-computer interaction methods in the future.

[0035] Natural language processing (NLP) is an important direction in the fields of computer science and artificial intelligence. It studies various theories and methods that can achieve effective communication between people and computers using natural language. Natural language processing is a science that integrates linguistics, computer science, and mathematics. Therefore, research in this field will involve natural language, that is, the language people use in daily life, so it is closely related to the study of linguistics. Natural language processing technology usually includes text processing, semantic understanding, machine translation, robot question answering, knowledge graph and other technologies.

[0036] Machine Learning (ML) is a multi-disciplinary subject that involves probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory and other disciplines. It specializes in studying how computers simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications are spread across all areas of artificial intelligence. Machine learning and deep learning usually include artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and self-learning.

[0037] Autonomous driving technology usually includes high-precision maps, environmental perception, behavioral decision-making, path planning, motion control and other technologies. Autonomous driving technology has broad application prospects.

[0038] With the research and advancement of artificial intelligence technology, artificial intelligence technology has been studied and applied in many fields, such as common smart homes, smart wearable devices, virtual assistants, smart speakers, smart marketing, unmanned driving, automatic driving, drones, robots, smart medical care, smart customer service, etc. I believe that with the development of technology, artificial intelligence technology will be applied in more fields and play an increasingly important role.

[0039] The solution provided in the embodiments of the present application involves technologies such as natural language processing of artificial intelligence, which is specifically illustrated by the following embodiments.

[0040] Figure 1 FIG. 1 is an application environment diagram of a keyword extraction method in an embodiment. Figure 1 , the keyword extraction method is applied to a keyword extraction system. The keyword extraction system includes a terminal 110 and a server 120. The terminal 110 and the server 120 are connected via a network. The above keyword extraction method can be executed in the terminal 110 or in the server 120. After the terminal 110 obtains the target text, the above keyword extraction method is used to extract the keywords in the target text. Alternatively, after the terminal 110 obtains the target text, the target file is sent to the server 120, and the server 120 extracts the keywords in the target text using the above keyword extraction method. Based on the extracted keywords, the server 120 can determine the similarity between the target text and other texts, push non-similar texts to the terminal 110, or migrate comments between similar texts. The terminal 110 can specifically be a desktop terminal or a mobile terminal, and the mobile terminal can specifically be at least one of a mobile phone, a tablet computer, a laptop computer, etc. The server 120 can be implemented as an independent server or a server cluster consisting of multiple servers.

[0041] like Figure 2As shown, in one embodiment, a keyword extraction method is provided. This embodiment mainly applies this method to the above Figure 1 The terminal 110 or the server 120 in FIG. Figure 2 , the keyword extraction method specifically includes the following steps:

[0042] S202, slidingly extracting words from the target text according to at least two co-occurrence windows respectively, to obtain a plurality of word combinations under each co-occurrence window.

[0043] The target text refers to the text from which keywords are to be extracted, which may be a sentence, a paragraph, or an article, etc. The target text may be stored locally on the computer device, or may be obtained from other computer devices via a network or USB (Universal Serial Bus).

[0044] A co-occurrence window is a sliding window defined based on a double pointer and including multiple panes arranged in sequence. Different types of co-occurrence windows can have different window lengths. The window length refers to the number of panes contained in the co-occurrence window. Figure 3 , Figure 3 A schematic diagram of co-occurrence windows with different window lengths in one embodiment is shown. Figure 3 The figure shows a process of sliding word extraction with a co-occurrence window of a window length of 4 and a process of sliding word extraction with a co-occurrence window of a window length of 3. Figure 3 As shown, each pane is used to extract a word in the target text. The double pointer includes a start pointer and an end pointer. The start pointer is used to indicate the position of the first sequential pane in the co-occurrence window relative to the target text; the end pointer is used to indicate the position of the last sequential pane in the co-occurrence window relative to the target text. In other words, the start pointer and the end pointer point to a word in the target text respectively. By changing the start pointer or the end pointer, the window length of the co-occurrence window can be adjusted, and the co-occurrence window can also be controlled to slide relative to the target text.

[0045] As the co-occurrence window slides based on the target text, the words framed by each pane in the co-occurrence window in the target text change. Thus, each co-occurrence window can frame multiple word combinations in the target text. A word combination is a combination consisting of more than one word.

[0046] Specifically, the computer device obtains the target text, performs word segmentation on the target text, and removes the stop words therein. Stop words refer to words that have little effect on expressing the theme of the target text, such as punctuation marks, subjects such as "you", "we", "she", and modal particles such as "ah", "le", and "ne". After removing the stop words, the multiple words are arranged in the order of appearance in the target text to form a word sequence corresponding to the target text. For example, the word sequence corresponding to the target text "I especially like watching the Brazilian Football World Cup!" can be [like, watch, Brazil, football, World Cup].

[0047] The computer device slides and extracts words in the word sequence based on each co-occurrence window. For example, in the target text "I like to watch the Brazilian Football World Cup", based on the co-occurrence window with a window length of 3, the computer device points the starting pointer of the co-occurrence window to the first order word "like" in the word sequence, and obtains a word combination [like, watch, Brazil]. The computer device points the starting pointer to the second order word "watch" at a sliding speed of 1, and obtains another word combination [watch, Brazil, football]. In this way, multiple word combinations corresponding to each co-occurrence window can be obtained.

[0048] In one embodiment, the co-occurrence window may be pre-defined and configured to be stored in the computer device, or may be dynamically determined by the computer device according to the text length of the target text and the accuracy of keyword extraction. It is understood that the more types of co-occurrence windows used for sliding word extraction, the higher the accuracy of the corresponding extracted keywords. The number of types of co-occurrence windows may be comprehensively determined based on the amount of calculated data and the accuracy of keyword extraction.

[0049] S204, determining the co-occurrence probability between different words in each word combination based on the word vector model.

[0050] Among them, the word vector model is a neural network model based on word vectors, which can efficiently learn the prediction model of word space vectors from the corpus. Specifically, it can be a CBOW model or Skip-Gram model based on the framework of Hierarchical Softmax (hierarchical classification) and Negative Sampling (NEG) in Word2Vec (WordEmbedding). Corpus is a basic resource that carries language knowledge with computer devices as the carrier. It is used to store corpus resources obtained after analyzing and processing text materials that have actually appeared in actual language use. Different word vector models can be trained based on different corpora.

[0051] The co-occurrence probability refers to the probability of different things co-occurring. In the embodiment of the present application, the co-occurrence probability refers to the probability that one word in a word combination and another word in the word combination co-appear in a target text in a certain order.

[0052] Specifically, the computer device mines valuable P (w1, w2, w3...w in the target text based on the statistical language model (Statistical Language Model, SLM) of natural language processing (NLP) technology. n )=P(w1)P(w2)P(w3)...P(w n ), such as the conditional probability of each word appearing in the target text. Assume Indicates that the target text consists of n words w1, w2, w3...w n According to the text content constructed in sequence, the joint probability of the target text is Based on the naive Bayes formula, the naive Bayes hypothesis is made between words in the target text, and the above formula can be converted to, then the conditional probability of each word in the target text appearing in the corpus can be:

[0053]

[0054] Among them, 1≤i≤n, 1≤j≤n, P(w j |w i ) is the word w j With the word w i The transition probability between them is also the model parameter of the language model. In the embodiment provided in the present application, the computer device calculates these model parameters based on the word vector model, and then obtains the conditional probability of each word appearing in the target text. The computer device determines the co-occurrence probability of each word in the target text based on different word combinations and other words according to the conditional probability.

[0055] It is worth noting that the word vector model based on neural network has its own smoothing function. Even if two words co-occur in any word combination in the target text, a reasonable co-occurrence probability corresponding to each word can be obtained.

[0056] S206: In each word combination, words whose co-occurrence probability meets the screening condition are determined as candidate words.

[0057] The screening condition refers to the pre-set condition that the co-occurrence probability of a word in a word combination should meet when determining the word as a candidate word, which can be the maximum co-occurrence probability in the word combination, or the co-occurrence probability reaches a threshold, etc. It can be understood that when the screening condition is "the maximum co-occurrence probability in the word combination", each word combination will produce a candidate word, and when the screening condition is "the co-occurrence probability reaches the threshold", whether each word group can produce a candidate word and the number of candidate words produced are uncertain.

[0058] Specifically, the computer device selects candidate words in each word combination according to the preset screening condition. For example, when the screening condition is "the co-occurrence probability in the word combination is the largest", for the target text "I particularly like to watch the Brazilian Football World Cup!", the conditional probabilities of "like", "watch", "Brazil", "football", and "World Cup" appearing in the target text are 65%, 21%, 8%, 33%, and 19% respectively. Then, the corresponding co-occurrence probability combination in the word combination A1 [like, watch, Brazil] extracted based on the co-occurrence window A with a window length of 3 can be [65%, 21%, 8%], so that "like" can be used as a candidate word for the word combination A1, and the corresponding co-occurrence probability combination in the word combination A2 [watch, Brazil, football] can be [21%, 8%, 33%], so that "football" can be used as a candidate word for the word combination A2. The corresponding co-occurrence probability combination in the word combination B1 [like, watch, Brazil, football] extracted based on the co-occurrence window B with a window length of 4 can be [65%, 21%, 8%, 33%], so "like" can be used as a candidate word for the word combination B1, and the corresponding conditional probability combination in the word combination B2 [watch, Brazil, football, World Cup] can be [21%, 8%, 33%, 19%], so "football" can be used as a candidate word for the word combination B2.

[0059] S208, determining the probability of contribution of each candidate word to the explanation of the target text in the candidate word set composed of the candidate words of each word combination according to the conditional probability of the candidate word appearing in the corpus.

[0060] The contribution probability refers to the probability value used to characterize the explanation strength of the candidate word on the subject expressed in the target text, which can be calculated based on the term frequency (TF) of the candidate word in the candidate word set. The computer device processes the corpus based on the word vector model in the above manner, and calculates the conditional probability of each word in the corpus appearing in the corpus.

[0061] For the convenience of description, the conditional probability of a leaf node appearing in the target text calculated according to the first transition probability of all adjacent nodes on the first Huffman path is recorded as the first conditional probability. The conditional probability of a leaf node appearing in the corpus calculated according to the second transition probability of all adjacent nodes on the second Huffman path is recorded as the second conditional probability.

[0062] Specifically, the computer device classifies the candidate words generated by each word combination, and classifies the same candidate words into the same candidate word. The computer device calculates the sum of the co-occurrence probabilities of each candidate word with other words in different word combinations to obtain the comprehensive co-occurrence probability corresponding to each candidate word. The computer device obtains the conditional probability of each candidate word appearing in the entire corpus, calculates the ratio of the comprehensive co-occurrence probability to the conditional probability of appearing in the corpus, and determines the contribution probability of the corresponding candidate word to the interpretation of the target text based on the ratio.

[0063] It is worth noting that the second conditional probability is a probability value calculated based on all the corpus in the corpus, which can be considered as a baseline probability. Since the contribution vector of each candidate word will be affected by all the words in the corpus, that is to say, it will be corrected by a baseline probability, compared with the TextRank algorithm, which only judges the importance of each word based on the amount of information in the target text itself, the calculation method of the contribution probability in this application will not have too much deviation due to the extreme nature of the target text content, which greatly improves the accuracy of keyword extraction. The conditional probability and co-occurrence probability are calculated based on the word vector model, which fully considers the semantic information of the target text itself and all the corpus in the corpus. Although the IF-IDF algorithm also refers to the information of the corresponding word in the corpus for judging the importance of each word, IF-IDF only refers to the frequency of the word appearing in the corpus, and completely ignores the semantic correlation information between words. Therefore, the calculation method of the contribution probability in this application also greatly improves the accuracy of keyword extraction compared with the IF-IDF algorithm.

[0064] S210: Determine candidate words whose contribution probabilities meet the threshold condition as keywords.

[0065] Among them, the threshold condition refers to the condition that the contribution probability corresponding to the candidate word should meet when the candidate word is determined as a keyword. Specifically, it can be that the contribution probability is ranked in the front preset number in descending order, or the occurrence probability reaches the threshold, etc. It can be understood that when the threshold condition is "the contribution probability reaches the threshold", the number of keywords extracted from the target text is uncertain. When the threshold condition is "the contribution probability is ranked in the front preset number in descending order", the number of keywords extracted from the target text can be determined, that is, the preset number. For example, when the preset number is 1, the candidate word with the highest contribution probability is extracted from the target text as the keyword that best reflects the expressed theme.

[0066] In one embodiment, the preset number may be a fixed value set in advance, or a value dynamically determined according to the text length of the target text. For example, as the text length of the target text increases, the preset number of corresponding extracted keywords may increase accordingly.

[0067] It should be noted that the word vector model takes into account the internal association of the target text by considering the transition probability between words. It is a binary model that evaluates keywords from two dimensions: word frequency and semantic association between words. Compared with the unigram model of the TF-IDF algorithm that only considers the information content of the word itself, it can further improve the accuracy of keyword extraction.

[0068] Although the TextRank algorithm is also a binary model, it also constructs a directed graph G(V, E) with each word in the target text as a node, and uses a voting mechanism to score each node in the directed graph, sorts multiple words according to the scores, and selects one or more words with high scores as keywords. Since the number of nodes required to build a directed graph has a certain threshold, such as at least 50 nodes, there are certain requirements for the text length of the target text for extracting keywords, which makes the TextRank algorithm only applicable to common texts, such as only being able to extract keywords from an article or a long paragraph, and not from a sentence. The keyword extraction method provided in this application is not subject to this limitation and is applicable to texts of any length.

[0069] The TextRank algorithm can achieve keyword extraction using only the information of a single document itself, without the need to learn and train multiple documents in advance. It is widely used because of its simplicity and effectiveness. This unsupervised approach is both its advantage and disadvantage, which makes the TextRank algorithm only consider the amount of information in the target text itself. For extreme samples, the accuracy of keyword extraction is often extremely low. In comparison, this application uses the conditional probability of the word appearing in the corpus as a reference, and has better stability in the accuracy of keyword extraction.

[0070] The above keyword extraction method slides and extracts words from the target text based on multiple co-occurrence windows, so that each word in the target text can appear in multiple word combinations. Based on the word vector model, the probability of each word co-occurring with other words is measured in units of word combinations, and words with large co-occurrence probability are selected as candidate words. The association between words in the target text is fully considered. Compared with simply considering the information content of the word itself, the accuracy of candidate word extraction can be improved; combined with the conditional probability of the candidate word appearing in the entire corpus, its contribution probability to the interpretation of the target text is evaluated, so that the calculation of the contribution probability is guided by the semantic information of the entire corpus, which is equivalent to having a benchmark conditional probability as a reference. Compared with simply considering the information content of the target text itself, it can reduce the contribution probability accuracy deviation caused by the extreme nature of the target text content, and thus improve the accuracy of selecting keywords in the candidate word set based on the contribution probability.

[0071] In one embodiment, Figure 4As shown, step S204, i.e., the step of determining the co-occurrence probability between different words in each word combination based on the word vector model, includes:

[0072] S402, obtaining a first Huffman tree constructed based on the word vector model and the target text; the first Huffman tree includes a plurality of nodes, and the nodes include a root node and a leaf node corresponding to each word in the target text.

[0073] S404, determining the first Huffman path of each leaf node in the first Huffman tree.

[0074] S406: Determine a first transition probability of the root node transitioning to a corresponding leaf node based on the first Huffman path.

[0075] S408: Determine the co-occurrence probability between different words in each word combination according to the first transition probability.

[0076] Among them, the Huffman tree, also known as the optimal binary tree, is a binary tree with the shortest weighted path length. The Huffman tree includes multiple levels of nodes, including leaf nodes and non-leaf nodes. The weight of a node refers to a numerical value with a certain meaning assigned to the node. A path refers to the path between child nodes that a node can reach by extending downward. The path length refers to the number of branches in the path. If the layer number of the root node in the Huffman tree is specified as 1, the path length from the root node to the Lth layer node is L-1. The weighted path length of a node refers to the product of the path length from the root node to the node and the weight of the node. The weighted path length of a tree refers to the sum of the weighted path lengths of all leaf nodes in the tree.

[0077] The Huffman tree is an ordered tree in which each node has at most two subtrees, which are usually referred to as the "left subtree" and the "right subtree". The order of the left subtree and the right subtree cannot be reversed. In the embodiment of the present application, the node with a large word frequency is uniformly used as the left child node, and the node with a small word frequency is uniformly used as the right child node.

[0078] The Huffman tree can be constructed based on the word vector model. The word vector model is input to obtain a first Huffman tree with each word in the target text as a leaf node and the corresponding word frequency as the weight of the corresponding leaf node, and n-1 non-leaf nodes are added.

[0079] refer to Figure 5 , Figure 5FIG. 1 shows a schematic diagram of the first Huffman tree based on the target text in one embodiment. Assume that during the World Cup, several target texts related to football are captured from the network platform. According to statistics, the number of occurrences of the six words "I", "like", "watch", "Brazil", "football" and "World Cup" are 15, 8, 6, 5, 3 and 1 respectively. The Huffman tree constructed with these six words as nodes and the corresponding word frequencies as weights is as follows: Figure 5 As shown. Leaf nodes are nodes corresponding to words that appear in the target text used to construct the first Huffman tree, such as the nodes corresponding to "I", "like", "watch", "Brazil", "football", and "World Cup". Non-leaf nodes are nodes added to connect leaf nodes when constructing the Huffman tree, such as nodes with word frequencies of 4, 9, 14, 23, and 38. The first Huffman path refers to the path extending from the root node to a leaf node with the shortest path length. For example, the first Huffman path of the leaf node "football" is Figure 5 The path formed by connecting the non-leaf nodes and the leaf node "football" in series with bold edges has a corresponding path length of 4.

[0080] Starting from the root node to reach a leaf node, one or more branches are required in the middle, and each branch can be regarded as a binary classification. For the convenience of description, the two adjacent nodes on the Huffman path are divided into pre-order nodes and post-order nodes. Each non-leaf node has two corresponding post-order nodes, namely the left child node and the right child node. For example, the left child node corresponding to the non-leaf node with a word frequency of 38 is the node with a word frequency of 23, and the right child node is the leaf node corresponding to "I".

[0081] Specifically, the computer device obtains a pre-constructed first Huffman tree. Each leaf node in the first Huffman tree has a uniquely determined first Huffman path. The computer device calculates the probability of a node being classified as a branch node, and determines the classification probability as a first transition probability between corresponding adjacent nodes on the first Huffman path. The first transition probability can be calculated based on a pre-trained classification model, such as a logistic regression model, a decision tree model, a naive Bayes model, a multiple Bernoulli model, a support vector machine (Support Vector Machine, SVM), etc. The computer device calculates the conditional probability of the corresponding word of the corresponding leaf node appearing in the corpus based on the first transition probability between all adjacent nodes on the first Huffman path.

[0082] Further, the computer device determines the conditional probability of the occurrence of the leaf node corresponding to each word in the target text in the first Huffman tree as the co-occurrence probability of the corresponding word with other words in each word combination. In an embodiment of the present application, the co-occurrence probability of the same word with different words in different word combinations is the same. It can be understood that other methods can also be selected according to needs to calculate the co-occurrence probability of each word in each word combination.

[0083] In this embodiment, the transfer probability between words is calculated based on the word vector model, and the relationship between words is fully considered. Compared with simply considering the amount of information of a single word, the accuracy of co-occurrence probability calculation can be improved, thereby improving the accuracy of candidate word extraction.

[0084] In one embodiment, a word combination includes a central word and corresponding context words; determining the co-occurrence probability between different words in each word combination according to the first transition probability includes: calculating the conditional probability of the corresponding word appearing in the target text according to the first transition probability; calculating the co-occurrence probability of each context word and the central word in each word combination according to the conditional probability; in each word combination, determining the words whose co-occurrence probability meets the screening conditions as candidate words includes: in each word combination, determining the context words whose co-occurrence probability meets the screening conditions as candidate words.

[0085] Among them, the central word refers to the word at a preset position in the word combination. A central word can be pre-specified in each word combination based on the co-occurrence window. The context word refers to the previous word or the following word of the central word. In an embodiment of the present application, the words other than the central word in the word combination are the context words of the central word. In a word combination, the central word has at least one of the corresponding previous words or following words. The previous words refer to the words that appear before the central word in the word combination, and the following words refer to the words that appear after the central word in the word combination.

[0086] Specifically, the computer device calculates the conditional probability of the word corresponding to the leaf node appearing in the target text according to the first transition probability between all adjacent nodes on the first Huffman path corresponding to the leaf node. The computer device calculates the co-occurrence probability of each context word in each word combination with the central word according to the conditional probability of each word appearing in the target text. The co-occurrence probability can be understood as the degree of effect of the context word on the prediction of the central word.

[0087] In one embodiment, according to the conditional probability, the co-occurrence probability of each context word and the central word in each word combination is calculated separately, including: determining the conditional probability of the context word appearing in the target text as the co-occurrence probability of the context word and the central word in each word combination; in each word combination, determining the context words whose co-occurrence probability meets the screening conditions as candidate words includes: in each word combination, determining the context word with the largest co-occurrence probability as the candidate word.

[0088] Specifically, the computer device uses the conditional probability of the context words appearing in the first Huffman tree as the co-occurrence probability with the central word. In an embodiment of the present application, the conditional probability of the same word in different word combinations predicting different central words is the same. It can be understood that other methods can also be selected according to needs to calculate the co-occurrence probability of each context word in each word combination predicting the central word.

[0089] For instance, in the above example, the corresponding co-occurrence probability combination in word combination A1 [like, watch, Brazil] can be [65%, 21%, 8%]. Since the context words with the largest co-occurrence probability are screened, the conditional probability 21% corresponding to the central word "watch" can be ignored, so "like" can be used as a candidate word for word combination A1. The corresponding co-occurrence probability combination in word combination A2 [watch, Brazil, football] can be [21%, 8%, 33%]. When the conditional probability of the central word "Brazil" is ignored, "football" can be used as a candidate word for word combination A2.

[0090] In one embodiment, the computer device performs a linear conversion operation on the conditional probability of the context word appearing in the first Huffman tree, and uses the operation result as the co-occurrence probability with the central word.

[0091] In this embodiment, the co-occurrence probability between words in the target text is calculated based on the word vector model. Since the word vector can reflect the similarity between words, that is, the word vector model has established the association between similar words, compared with TF-IDF which only considers the relationship between the same words, it can fully combine the semantic information of the target text, thereby improving the accuracy of keyword extraction.

[0092] In one embodiment, according to the first transition probability, calculating the conditional probability that the corresponding word appears in the target text includes: determining the node vector and node code corresponding to each node on the first Huffman path; the first Huffman path includes a plurality of nodes connected in series; inputting the node vector of the preceding node among the adjacent nodes on the first Huffman path into a pre-trained classification model to obtain a classification probability;

[0093] According to the classification probability of the preceding node and the node code of the succeeding node in the adjacent nodes, the transition probability between the corresponding adjacent nodes is determined; according to the transition probability between all adjacent nodes on the first Huffman path, the conditional probability of the corresponding word of the corresponding leaf node appearing in the target text is calculated.

[0094] Among them, the node vector is a vector defined for non-leaf nodes to assist in completing the conditional probability. The node vector of the non-leaf node has the same vector length as the word vector corresponding to the leaf node. Since calculating the transition probability is a binary classification process, it is necessary to assign a node code to each node. The node code is a code assigned to the node for distinguishing the left child node and the right child node as a positive class and a parent class. For example, the node with the node code 1 can be positioned as a positive class, and the node with the node code 0 can be defined as a negative class. In the embodiment provided in the present application, it is stipulated that the node code of the left child node is 1, and the node code of the right child node is 0. It can be understood that in another embodiment, the node code of the left child node can also be 0, and the node code of the right child node can be 1. It should be noted that the root node does not have a corresponding node code. The node code of the subsequent node can be used as a classification label for classifying the previous node.

[0095] For the convenience of description, two adjacent nodes on the Huffman path are divided into a pre-order node and a post-order node. It can be understood that the post-order node is the left child node or the right child node of the pre-order node. The Huffman path includes multiple nodes connected in series.

[0096] Specifically, the computer device obtains a pre-constructed first Huffman tree, and determines the first Huffman path corresponding to each leaf node in the first Huffman tree. For two adjacent nodes on the first Huffman path, when the post-order node is the left child node of the pre-order node, the computer device calculates the classification probability of the pre-order node extending downward to the left child node, and determines the classification probability as the first transition probability between the corresponding adjacent nodes. When the post-order node is the right child node of the pre-order node, the computer device calculates the classification probability of the pre-order node extending downward to the right child node, and determines the classification probability as the first transition probability between the corresponding adjacent nodes.

[0097] According to the word2vec formula, the probability logarithm of each node is:

[0098]

[0099]

[0100] Among them, θ is the node vector, x is the input word vector, and d is the node code. In other words, on the first Huffman path of the leaf node w, the classification probability of a preceding node being classified as a left child node can be The probability of being classified as a right child node can be Among them, θ is the node vector, which is also a parameter to be determined. The input word vector is the vector obtained by summing up the word vectors of the input layer in the word vector model. The word vector of the input layer refers to the word vector corresponding to each word in the target text.

[0101] In one embodiment, based on the first transition probabilities between all adjacent nodes on the first Huffman path, calculating the conditional probability that the corresponding word of the corresponding leaf node appears in the corpus includes: performing a multiplication operation on the first transition probabilities of all adjacent nodes on the first Huffman path; and determining the multiplication result as the conditional probability that the corresponding word of the corresponding leaf node appears in the corpus.

[0102] For each word w in the corpus, there is a first Huffman path from the root node to the node corresponding to word w in the first Huffman tree. There are one or more branches on the first Huffman path. Each branch is regarded as a binary classification. Each classification generates a transition probability. These transition probabilities are multiplied together to obtain the conditional probability of word w appearing in the corpus. For example, in Figure 4 In the first Huffman tree shown, on the first Huffman path corresponding to the leaf node "football", it takes four binary classifications to trigger from the root node with a word frequency of 38 to reach the leaf node "football". The probability of the first classification result is The probability of the second classification result is The probability of the third classification result is The probability of the fourth classification result is

[0103]

[0104] The computer device multiplies the transition probabilities between all adjacent nodes on the first Huffman path, and uses the multiplication result as the conditional probability of the corresponding word of the corresponding leaf node appearing in the target text. For example, in the above example, the conditional probability of the leaf node "soccer" appearing in the first Huffman tree is

[0105]

[0106] Based on the hierarchical softmax framework, we build the word vector model Skip-gram or CBOW, and use this word vector model to calculate the transfer probability of each word in the sentence to its context words, and select the words with the highest probability of explaining the text, so as to achieve keyword extraction.

[0107] In this embodiment, the Huffman tree is constructed based on the word vector model, which can clearly locate the role of each word in the entire target text and the relationship between it and other words. The co-occurrence probability between words in the target text is calculated based on the Huffman tree, which can fully consider the association between different words. Compared with TF-IDF, which only considers the frequency of occurrence of a single word, it can improve the accuracy of keyword extraction.

[0108] In one embodiment, the above-mentioned keyword extraction method also includes: obtaining a second Huffman tree constructed based on a word vector model and a corpus; the second Huffman tree includes multiple nodes; the nodes include a root node and a leaf node corresponding to each word in the corpus; determining a second Huffman path of the leaf node corresponding to each word in the target text in the second Huffman tree; and calculating the conditional probability of the corresponding word appearing in the corpus based on the second transition probability of the root node transferring to the corresponding leaf node based on the second Huffman path.

[0109] The second Huffman tree is a Huffman tree constructed based on the word vector model and the corpus. The word vector model is input to obtain a second Huffman tree with each word in the text material as a leaf node and the corresponding word frequency as the weight of the corresponding leaf node, and m-1 non-leaf nodes are added to construct a full second Huffman tree.

[0110] Specifically, the computer device determines the second Huffman path of the leaf node corresponding to each word in the corpus in the second Huffman tree in the above manner, and calculates the second transition probability of the root node transferring to the corresponding leaf node based on the second Huffman path, and calculates the second conditional probability of the corresponding word of the corresponding leaf node appearing in the target text according to the second transition probability between all adjacent nodes on the second Huffman path.

[0111] In one embodiment, the second Huffman tree of the entire amount can be pre-constructed, and the second conditional probability based on the second Huffman tree can also be pre-calculated. When it is necessary to extract keywords from the target text, the corresponding second conditional probability value can be directly obtained, which greatly improves the efficiency of keyword extraction. The second Huffman tree and the second conditional probability of each corresponding leaf node only need to be updated accordingly when the corpus is updated.

[0112] In one embodiment, when a candidate word in the target text does not exist in the corpus, the computer device queries the second conditional probability of a similar word corresponding to the word appearing in the corpus, and calculates the probability of the corresponding candidate word contributing to the interpretation of the target text based on the second conditional probability of the similar word.

[0113] In this embodiment, the second conditional probability is a probability value calculated based on all the corpus in the corpus, which can be considered as a baseline probability. Since the contribution vector of each candidate word will be affected by all the words in the corpus, that is, it will be corrected by a baseline probability, the contribution probability calculation deviation caused by the extreme nature of the target text content can be reduced, thereby improving the accuracy of keyword extraction.

[0114] In one embodiment, based on the conditional probability of the candidate words appearing in the corpus, in the candidate word set composed of the candidate words of each word combination, determining the contribution probability of each candidate word to the interpretation of the target text includes: constructing a candidate word set based on the candidate words of each word combination; determining the word frequency of each candidate word in the candidate word set; and determining the contribution probability of the corresponding candidate word to the interpretation of the target text based on the word frequency and the conditional probability.

[0115] The candidate word set is a set of candidate words generated by all word combinations corresponding to the target text. The candidate words in the candidate word set may be repeated, that is, the same candidate word may appear multiple times in the candidate word set.

[0116] In one embodiment, determining the contribution probability of a corresponding candidate word to explaining a target text based on word frequency and conditional probability includes: calculating the ratio of word frequency to the conditional probability of a candidate word appearing in a corpus; and determining the contribution probability of a corresponding candidate word to explaining the target text based on the ratio.

[0117] Specifically, the computer device counts the frequency of each candidate word in the candidate word set, calculates the ratio of the frequency to the conditional probability of the candidate word appearing in the corpus, and determines the ratio as the contribution probability of the corresponding candidate word to the explanation of the target text. For example, assuming that the target text "I particularly like watching the Brazilian Football World Cup!" corresponds to the candidate word set [like, football, like, like, football, like, watch, like, World Cup, World Cup], the frequency of the candidate word "like" is 5. If the conditional probability of "like" appearing in the corpus is 50%, then the contribution probability of the candidate word "like" to the explanation of the target text "I particularly like watching the Brazilian Football World Cup!" can be 10%.

[0118] In one embodiment, the computer device counts the total number of each candidate word in the candidate word set and the word frequency of each candidate word, and calculates the first proportion of the word frequency corresponding to each candidate word relative to the total number of candidate words. The computer device calculates the second proportion of the first conditional probability of the candidate word appearing in the target text relative to the second conditional probability appearing in the corpus. The computer device determines the contribution probability of the corresponding candidate word to the interpretation of the target text according to the first proportion and the second proportion, such as taking the product of the first proportion and the second proportion as the contribution probability of the corresponding candidate word to the interpretation of the target text. For example, in the above example, the candidate word set corresponding to the target text has a total of 10 words, and the word frequency of the candidate word "like" is 5, then the first proportion of the corresponding word frequency of the candidate word "like" relative to the total number of candidate words is 5 / 10=50%; the first conditional probability of the candidate word "like" appearing in the target text is 65% relative to the second conditional probability of 50% appearing in the corpus. The second proportion is 65% / 50%=1.3, then the contribution probability of the candidate word "like" to the interpretation of the target text can be 50%*1.3=65%.

[0119] refer to Figure 6 , Figure 6 FIG. 2 shows a schematic diagram of the principle of a keyword extraction method in an embodiment. Figure 6 As shown, when it is necessary to extract keywords from the target text, the target text is preprocessed by word segmentation, stop word removal, etc., and the word sequence obtained by the preprocessing is input into the word vector model to construct the first Huffman tree corresponding to the target text. The first Huffman tree includes leaf nodes corresponding to each word in the word sequence. Each leaf node has a unique first Huffman path in the first Huffman tree. Based on the word vector model, the first conditional probability of each leaf node based on the first Huffman path can be calculated. Based on the preset multiple co-occurrence windows, the word sequence is slid to obtain multiple word combinations corresponding to each co-occurrence window, such as [a, b, c], [b, c, d], [a, b, c, d], [b, c, d, e], etc. According to the first conditional probability of each word in the word combination, the co-occurrence probability between different words in the word combination can be determined. For example, the first conditional probability corresponding to words a, b, c is A%, B%, C%, and the corresponding co-occurrence probability combination in the word combination [a, b, c] is [A%, B%, C%. In each word combination, a word with the highest co-occurrence probability is selected as a candidate word. The candidate words corresponding to each word combination can form a candidate word set.

[0120] After extracting the candidate word set, the second Huffman tree corresponding to the entire corpus constructed based on the word vector model can be obtained. The second Huffman tree includes leaf nodes corresponding to each word in the corpus. Each leaf node has a unique second Huffman path in the second Huffman tree. Based on the word vector model, the second conditional probability of each leaf node based on the second Huffman path can be calculated. By calculating the ratio of the word frequency of the candidate word in the candidate word set to the corresponding second conditional probability, the co-occurrence probability corresponding to each candidate word can be obtained. For example, the contribution probability corresponding to the candidate word b can be P(b)=3 / B%=B'%. Arrange the various candidate words in the candidate word set in descending order according to the contribution probability, and take one or more candidate words with the highest ranking as keywords.

[0121] In this embodiment, the conditional probability of the candidate word appearing in the entire corpus is combined to evaluate its contribution probability to the interpretation of the target text, so that the calculation of the contribution probability is guided by the semantic information of the entire corpus, which is equivalent to having a benchmark conditional probability as a reference. Compared with simply considering the amount of information in the target text itself, it can reduce the contribution probability accuracy deviation caused by the extreme nature of the target text content, thereby improving the accuracy of screening keywords in the candidate word set based on the contribution probability.

[0122] In one embodiment, before determining the word frequency of each candidate word in the candidate word set, the method also includes: determining the similarity distance between the word vectors corresponding to each candidate word in the candidate word set based on the word vector model; and determining multiple words whose similarity distance is less than or equal to a predetermined threshold as the same candidate word.

[0123] Among them, the similarity between words can be reflected by word vectors. In the word vector model based on neural networks, the word vectors corresponding to semantically similar words are also similar. Similarity distance is an indicator that can reflect the degree of similarity between word vectors, such as cosine distance, Euclidean distance, Jaccard distance, etc.

[0124] Specifically, the computer device determines the similarity distance between the word vectors corresponding to each candidate word in the candidate word set based on the word vector model, and determines multiple words whose similarity distance is less than or equal to a predetermined threshold as the same word or similar words. It can be understood that when the similarity distance is equal to zero, the corresponding two words are the same word, and when the similarity distance is greater than zero and less than a specific non-zero threshold, the corresponding two words are similar words. For example, the candidate word set constructed based on the target text is [like, football, like, like, favorite, like, watch, like, event, World Cup], then like and favorite are similar words.

[0125] Furthermore, the computer device obtains the conditional probability of each candidate word constituting a similar word appearing in the corpus, and averages the conditional probability of each candidate word constituting a similar word appearing in the corpus, and uses the average as the comprehensive conditional probability of the corresponding similar word appearing in the corpus. For example, in the above example, assuming that the conditional probability of the candidate word "like" appearing in the corpus is 50%, and the conditional probability of the candidate word "like" appearing in the corpus is 52%, then the comprehensive conditional probability of the candidate words "like" and "like" appearing in the corpus can be 51%.

[0126] The computer device calculates the ratio of the word frequency of each candidate word in the candidate word set to the comprehensive conditional probability of appearing in the corpus in the above manner, and determines the ratio as the contribution probability of each candidate word constituting a similar word. For example, in the above candidate word set, the candidate words "like" and "like" appear 6 times as the same keyword in the candidate word set, where the contribution probability corresponding to the candidate words "like" and "like" is 6 / 51%. In this way, when selecting keywords from the candidate words according to the contribution probability, as long as one of the candidate words constituting the similar words is selected as a keyword, the similar words selected as keywords will also be selected as keywords.

[0127] In this embodiment, similar words express similar themes. Counting similar words as a kind of word can enhance the explanatory power of similar words on the target text, so that each word constituting the similar words has the same contribution probability, thereby extracting a class of words with strong explanatory power on the target text as keywords with a higher probability.

[0128] In one embodiment, Figure 7 As shown, step 210, that is, the step of determining the candidate words whose contribution probabilities meet the threshold conditions as keywords, includes:

[0129] S702: Arrange multiple candidate words in descending order according to contribution probability.

[0130] S704, filtering candidate words with a number corresponding to a preset threshold and having a high ranking.

[0131] S706, determining the similarity distance between the word vectors corresponding to each word in the target text based on the word vector model.

[0132] S708, obtaining similar words in the target text corresponding to the screened candidate words according to the similarity distance.

[0133] S710: Determine the screened candidate words and corresponding similar words as keywords.

[0134] The preset threshold refers to a preset maximum number of keywords to be extracted. The preset threshold can be freely set according to requirements, such as 1.

[0135] Specifically, the computer device sorts the multiple candidate words in the candidate word set according to the contribution probability. In this embodiment, the computer device sorts the candidate words in the order of contribution probability from large to small, and selects the candidate words corresponding to the number of the preset thresholds ranked at the top as keywords. It can be understood that the candidate words can also be sorted in the order of contribution probability from small to large, and the candidate words corresponding to the number of the preset thresholds ranked at the bottom are selected as keywords.

[0136] Furthermore, the computer device identifies whether there are similar words corresponding to the keyword but not selected as the keyword in the target text based on the similarity distance of the word vector. For example, in the above example, "like" is selected as the keyword in the target text, and its similar word "like" is not selected as the keyword according to the contribution probability, but "like" and the similar word "like" can still be extracted as keywords at the same time.

[0137] In one embodiment, the computer device compares the number of candidate words corresponding to the preset threshold to see whether there are candidate words whose contribution probability reaches the specified value. If the preset threshold number of candidate words with high contribution probability screened out have corresponding contribution probabilities all less than the specified value, indicating that the interpretation strength of each candidate word for the target text is not high, the computer device can dynamically increase the preset threshold to increase the number of candidate words selected as keywords, so that the theme expressed by the target text can be more comprehensively and accurately reflected through more keywords.

[0138] In this embodiment, similar words express similar subject meanings, and a category of segmented words with high contribution probability are extracted as keywords, which can help users understand the subject expressed by the target text more quickly and accurately based on the keywords.

[0139] In one embodiment, the step of training a word vector model includes: obtaining a word vector model to be trained; the word vector model includes a word embedding layer and a neural network layer; calculating the similarity distance between word vectors corresponding to each word in the corpus based on the word embedding layer; calculating the conditional probability of each word appearing in the corpus based on the neural network layer; calculating the difference between the conditional probabilities corresponding to two words whose similarity distance is less than or equal to a first threshold; adjusting the parameters of the neural network layer according to the difference between the difference and the second threshold and continuing training until the training is terminated when the difference is less than or equal to the second threshold.

[0140] The word vector model to be trained may be a neural network model, such as a CNN (Convolutional Neural Networks) model, an RNN (Recurrent Neural Networks) model, and the like.

[0141] Specifically, the computer device constructs training samples based on the corpus. After obtaining the training samples, the training samples can be segmented to obtain a word sequence composed of each word. It is understandable that the training samples obtained by the computer device can come from more than one corpus, and more than one set of training data can be constructed. Generally speaking, the more sets of training data there are, the wider the range of text content covered, and the more beneficial it is to the training of the model.

[0142] The computer device can convert a discrete word sequence into a corresponding initial vector sequence through a word embedding layer. Then the initial vector sequence is input into a pre-trained word vector model, and then the initial vector sequence is processed by a hidden layer included in the word vector model to obtain a hidden layer vector corresponding to the initial vector sequence. Among them, the hidden layer is a term in the neural network model, which is an intermediate layer relative to the input layer and the output layer. The hidden layer includes model parameters obtained by training the neural network model. Here, the hidden layer of the word vector model is an intermediate layer relative to the input layer of the word vector model and the output layer of the word vector model. All intermediate layers between the input layer and the output layer of the word vector model can be collectively referred to as hidden layers, and these intermediate layers can also be divided, that is, multiple hidden layers. The hidden layer of the word vector model may include a multi-layer neural network structure. Each layer of the neural network structure may include one or more neural network layers.

[0143] Furthermore, after obtaining the training sample, the computer device can train the word vector model through the training sample. That is, the initial vector sequence corresponding to the training sample is input into the word vector model, and the data of the hidden layer input included in the word vector model is processed and then the Huffman tree and the conditional probability corresponding to each leaf node in the Huffman tree are output. The computer device can identify similar words in the corpus according to the similarity distance between the word vectors. The computer device calculates the difference in conditional probabilities corresponding to multiple words constituting similar words, adjusts the model parameters in the direction of reducing the difference and continues training until the training stop condition is reached. The training stop condition is the condition for ending the model training. The training stop condition can be reaching a preset number of iterations, or the performance index of the word vector model after adjusting the model parameters reaches a preset index.

[0144] Subsequently, the conditional probability of each word in different target texts is calculated based on the trained word vector model, which can also make the conditional probabilities of similar words in the target texts similar.

[0145] In this embodiment, the word vector model trained in this way can establish connections between similar words. Extracting keywords based on the word vector model is actually superimposing "similar words" for keyword evaluation. Compared with the TF-IDF algorithm which only superimposes "same words" for keyword evaluation, extracting keywords based on the word vector model can preliminarily evaluate the importance of each word in combination with semantics, thereby improving the accuracy of keyword extraction.

[0146] In a specific application scenario, the keyword extraction method provided in this application can be used for text similarity calculation. With the development of Internet technology, users can obtain a large amount of data through the network, but users need to find the data they are interested in in the huge data, which reduces the efficiency of data acquisition. In order to improve the efficiency of data acquisition, some Internet platforms actively predict the data that users may be interested in, so that only the predicted data can be accurately pushed to users. For example, when the Internet platform pushes articles, the similarity between the articles to be pushed and the historical articles that the user has read is compared. The computer device extracts the keywords of the articles to be pushed and the historical articles respectively in the above manner, and evaluates the similarity of the two articles by calculating the similarity distance of the keywords of the two articles. The computer device can push or recall similar articles according to the preferences of different users. For example, for users who like to read similar articles, the computer device pushes the articles to be pushed that are highly similar to the historical articles read by the user to the user. For users who like to read articles in different fields, the computer device can recall the articles to be pushed that are highly similar to the historical articles read by the user, which can effectively avoid repeated recommendations. The computer device can also migrate comments between similar articles.

[0147] In another specific application scenario, the keyword extraction method provided by the present application can be used for text tag generation. When publishing an article, it is often necessary to add corresponding tags to the article, or even to extract the corresponding abstract. At present, the extraction of tags or abstracts mainly relies on the author's manual calibration, which makes the tag generation efficiency low. Even if some online platforms for article publishing can realize automatic tag extraction, they are simply extracted according to the word frequency statistics, which reduces the accuracy of the extracted tags. Based on the keyword extraction method provided by the present application, a computer device can perform keyword extraction while comprehensively considering the relationship between each word in the article and all the expected semantic information in the corpus, and use the extracted keywords as text tags, which not only improves the efficiency of tag generation, but also improves the accuracy of tag generation.

[0148] In another specific application scenario, the keyword extraction method provided by the present application can be used for text retrieval. With the development of Internet technology, users can retrieve data of interest through the network. There can be multiple search conditions, such as searching by author, publisher, title, content, etc. when searching for articles. Among them, keyword matching is a very important way when searching by content. The computer device can pre-extract the keywords of each article in the search library based on the keyword extraction method provided by the present application, and then only the keywords of the article need to be matched with the search conditions set by the user, which not only improves the efficiency of text retrieval, but also improves the text retrieval matching degree because the keywords are extracted under the comprehensive consideration of the association between each word in the article and all the expected semantic information in the corpus. The keyword extraction method provided by each embodiment of the present application can also be applied to any other scene with keyword extraction requirements, such as automatic labeling of sample labels during model training, etc., which will not be listed one by one here.

[0149] like Figure 8 As shown, in a specific embodiment, the keyword extraction method includes the following steps:

[0150] S802, slidingly extracting words from the target text according to at least two co-occurrence windows respectively, to obtain a plurality of word combinations under each co-occurrence window.

[0151] S804, determining the co-occurrence probability between different words in each word combination based on the word vector model; obtaining a first Huffman tree constructed based on the word vector model and the target text; the first Huffman tree includes multiple nodes, and the nodes include a root node and a leaf node corresponding to each word in the target text.

[0152] S806, determining the first Huffman path of each leaf node in the first Huffman tree.

[0153] S808, determining a first transition probability of the root node transferring to the corresponding leaf node based on the first Huffman path; the word combination includes a central word and a corresponding context word.

[0154] S810, determining a node vector and a node code corresponding to each node on a first Huffman path; the first Huffman path includes a plurality of nodes connected in series.

[0155] S812, input the node vector of the preceding node among the adjacent nodes on the first Huffman path into the pre-trained classification model to obtain the classification probability.

[0156] S814, determining a first transition probability between corresponding adjacent nodes according to the classification probability of the preceding node and the node code of the succeeding node among the adjacent nodes.

[0157] S816, calculating a first conditional probability that a word corresponding to a corresponding leaf node appears in the target text according to the first transition probabilities between all adjacent nodes on the first Huffman path.

[0158] S818: Determine the first conditional probability of the context word appearing in the target text as the co-occurrence probability of the context word and the central word in each word combination.

[0159] S820: In each word combination, determine the context word with the highest co-occurrence probability as a candidate word.

[0160] S822, obtaining a second Huffman tree constructed based on the word vector model and the corpus; the second Huffman tree includes a plurality of nodes; the nodes include a root node and a leaf node corresponding to each word in the corpus.

[0161] S824, determining the second Huffman path in the second Huffman tree corresponding to the leaf node of each word in the target text.

[0162] S826, calculating a second conditional probability of the corresponding word appearing in the corpus according to the second transition probability of the root node transferring to the corresponding leaf node based on the second Huffman path.

[0163] S828, constructing a candidate word set based on the candidate words of each word combination.

[0164] S830, determining the similarity distance between the word vectors corresponding to each candidate word in the candidate word set based on the word vector model.

[0165] S832, determining multiple words whose similarity distance is less than or equal to the threshold as the same type of candidate words.

[0166] S834, determining the frequency of each candidate word in the candidate word set.

[0167] S836, calculating the ratio of the word frequency to the second conditional probability of the candidate word appearing in the corpus.

[0168] S838, determining the contribution probability of the corresponding candidate word to the explanation of the target text according to the ratio.

[0169] S840, arranging the plurality of candidate words in descending order according to the contribution probability.

[0170] S842, filtering candidate words with a number corresponding to a preset threshold and having a high ranking.

[0171] S844, determining the screened candidate words as keywords.

[0172] The above keyword extraction method slides and extracts words from the target text based on multiple co-occurrence windows, so that each word in the target text can appear in multiple word combinations. Based on the word vector model, the probability of each word co-occurring with other words is measured in units of word combinations, and words with large co-occurrence probability are selected as candidate words. The association between words in the target text is fully considered. Compared with simply considering the information content of the word itself, the accuracy of candidate word extraction can be improved; combined with the conditional probability of the candidate word appearing in the entire corpus, its contribution probability to the interpretation of the target text is evaluated, so that the calculation of the contribution probability is guided by the semantic information of the entire corpus, which is equivalent to having a benchmark conditional probability as a reference. Compared with simply considering the information content of the target text itself, it can reduce the contribution probability accuracy deviation caused by the extreme nature of the target text content, and thus improve the accuracy of selecting keywords in the candidate word set based on the contribution probability.

[0173] Figure 2 , 4 , 7 and 8 are flowcharts of a keyword extraction method in one embodiment. It should be understood that although Figure 2 , 4 The steps in the flowcharts of , 7 and 8 are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 2 , 4 At least part of the steps in , 7 and 8 may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily performed at the same time, but can be performed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be performed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0174] like Fig. 9 As shown, the present application also provides a keyword extraction device 900, which includes a sliding word extraction module 902, a candidate word screening module 904, a contribution rate calculation module 906 and a keyword extraction module 908.

[0175] The sliding word extraction module 902 is used to slide the words of the target text according to at least two co-occurrence windows to obtain multiple word combinations under each co-occurrence window.

[0176] The candidate word screening module 904 is used to determine the co-occurrence probability between different words in each word combination based on the word vector model; in each word combination, the words whose co-occurrence probability meets the screening conditions are determined as candidate words.

[0177] The contribution rate calculation module 906 is used to determine the contribution probability of each candidate word to the explanation of the target text in the candidate word set composed of the candidate words of each word combination according to the conditional probability of the candidate word appearing in the corpus.

[0178] The keyword extraction module 908 is used to determine the candidate words whose contribution probabilities meet the threshold condition as keywords.

[0179] In one embodiment, the candidate word screening module 904 is also used to obtain a first Huffman tree constructed based on a word vector model and a target text; the first Huffman tree includes multiple nodes, including a root node and a leaf node corresponding to each word in the target text; determining a first Huffman path for each leaf node in the first Huffman tree; determining a first transition probability of the root node transferring to the corresponding leaf node based on the first Huffman path; and determining the co-occurrence probability between different words in each word combination based on the first transition probability.

[0180] In one embodiment, a word combination includes a central word and corresponding context words; the candidate word screening module 904 is also used to calculate the conditional probability of the corresponding word appearing in the target text according to the first transfer probability; according to the conditional probability, the co-occurrence probability of each context word and the central word in each word combination is calculated respectively; in each word combination, the words whose co-occurrence probability meets the screening conditions are determined as candidate words, including: in each word combination, the context words whose co-occurrence probability meets the screening conditions are determined as candidate words.

[0181] In one embodiment, the candidate word screening module 904 is also used to determine the node vector and node code corresponding to each node on the first Huffman path; the first Huffman path includes multiple nodes connected in series; the node vector of the preceding node among the adjacent nodes on the first Huffman path is input into a pre-trained classification model to obtain a classification probability; the transition probability between the corresponding adjacent nodes is determined based on the classification probability of the preceding node among the adjacent nodes and the node code of the succeeding node; based on the transition probability between all adjacent nodes on the first Huffman path, the conditional probability of the corresponding word of the corresponding leaf node appearing in the target text is calculated.

[0182] In one embodiment, the candidate word screening module 904 is also used to determine the conditional probability of the context word appearing in the target text as the co-occurrence probability of the context word and the central word in each word combination; in each word combination, the context word whose co-occurrence probability meets the screening conditions is determined as a candidate word, including: in each word combination, the context word with the largest co-occurrence probability is determined as a candidate word.

[0183] In one embodiment, Fig.10As shown, the above-mentioned keyword extraction device 900 also includes a conditional probability calculation module 910, which is used to obtain a second Huffman tree constructed based on a word vector model and a corpus; the second Huffman tree includes multiple nodes; the nodes include a root node and a leaf node corresponding to each word in the corpus; determine the second Huffman path of the leaf node corresponding to each word in the target text in the second Huffman tree; according to the second transition probability of the root node transferring to the corresponding leaf node based on the second Huffman path, calculate the conditional probability of the corresponding word appearing in the corpus.

[0184] In one embodiment, the contribution rate calculation module 906 is also used to construct a candidate word set based on the candidate words of each word combination; determine the word frequency of each candidate word in the candidate word set; and determine the contribution probability of the corresponding candidate word to the explanation of the target text based on the word frequency and conditional probability.

[0185] In one embodiment, the contribution rate calculation module 906 is also used to determine the similarity distance between the word vectors corresponding to each candidate word in the candidate word set based on the word vector model; and determine multiple words whose similarity distance is less than or equal to a threshold as the same candidate word.

[0186] In one embodiment, the contribution rate calculation module 906 is further used to calculate the ratio of the word frequency to the conditional probability of the candidate word appearing in the corpus; and determine the contribution probability of the corresponding candidate word to the explanation of the target text according to the ratio.

[0187] In one embodiment, the keyword extraction module 908 is also used to arrange multiple candidate words in descending order according to contribution probability; filter the top-ranked candidate words corresponding to a preset threshold; determine the similarity distance between the word vectors corresponding to each word in the target text based on the word vector model; obtain similar words in the target text corresponding to the filtered candidate words based on the similarity distance; and determine the filtered candidate words and corresponding similar words as keywords.

[0188] In one embodiment, Fig.10 As shown, the above-mentioned keyword extraction device 900 also includes a word vector model training module 912, which is used to obtain the word vector model to be trained; the word vector model includes a word embedding layer and a neural network layer; based on the word embedding layer, the similarity distance between the word vectors corresponding to each word in the corpus is calculated; based on the neural network layer, the conditional probability of each word appearing in the corpus is calculated; the difference between the conditional probabilities corresponding to two words whose similarity distance is less than or equal to a first threshold is calculated; according to the difference between the difference and the second threshold, the parameters of the neural network layer are adjusted and the training is continued until the difference is less than or equal to the second threshold, and the training is terminated.

[0189] The above-mentioned keyword extraction device slides and extracts words from the target text based on multiple co-occurrence windows, so that each word in the target text can appear in multiple word combinations. Based on the word vector model, the probability of each word co-occurring with other words is measured in units of word combinations, and words with large co-occurrence probability are selected as candidate words. The association between words in the target text is fully considered. Compared with simply considering the information content of the word itself, the accuracy of candidate word extraction can be improved; combined with the conditional probability of the candidate word appearing in the entire corpus, its contribution probability to the interpretation of the target text is evaluated, so that the calculation of the contribution probability is guided by the semantic information of the entire corpus, which is equivalent to having a benchmark conditional probability as a reference. Compared with simply considering the information content of the target text itself, it can reduce the contribution probability accuracy deviation caused by the extreme nature of the target text content, and thus improve the accuracy of selecting keywords in the candidate word set according to the contribution probability.

[0190] Fig.11 The internal structure diagram of a computer device in one embodiment is shown. The computer device may specifically be Figure 1 The terminal 110 in FIG. Fig.11 As shown, the computer device includes a processor, a memory, a network interface, an input device and a display screen connected through a system bus. Among them, the memory includes a non-volatile storage medium and an internal memory. The non-volatile storage medium of the computer device stores an operating system and may also store a computer program. When the computer program is executed by the processor, the processor can implement the keyword extraction method. The internal memory may also store a computer program. When the computer program is executed by the processor, the processor can execute the keyword extraction method. The display screen of the computer device may be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device may be a touch layer covered on the display screen, or a button, trackball or touchpad provided on the computer device housing, or an external keyboard, touchpad or mouse, etc.

[0191] Fig.12 The internal structure diagram of a computer device in one embodiment is shown. The computer device may specifically be Figure 1 The server 120 in FIG. Fig.12 As shown, the computer device includes a processor, a memory, a network interface, an input device and a display screen connected through a system bus. Among them, the memory includes a non-volatile storage medium and an internal memory. The non-volatile storage medium of the computer device stores an operating system and may also store a computer program, which, when executed by the processor, enables the processor to implement the keyword extraction method. The internal memory may also store a computer program, which, when executed by the processor, enables the processor to execute the keyword extraction method.

[0192] Those skilled in the art will understand that Fig.11 and 12 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0193] In one embodiment, the keyword extraction device provided by the present application can be implemented in the form of a computer program. The computer program can be Fig.11 and 12 The computer device shown in the figure is run. The memory of the computer device can store various program modules constituting the keyword extraction device, such as: Fig. 9 The sliding word extraction module, candidate word screening module, contribution rate calculation module and keyword extraction module shown in the figure. The computer program composed of each program module enables the processor to execute the steps of the keyword extraction method of each embodiment of the present application described in this specification.

[0194] For example, Fig.11 and 12 The computer device shown can be Fig. 9 The sliding word extraction module in the keyword extraction device shown executes step S202. The computer device can execute steps S204 and S206 through the candidate word screening module. The computer device can execute step S208 through the contribution rate calculation module. The computer device can execute step S210 through the keyword extraction module.

[0195] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the above-mentioned keyword extraction method. The steps of the keyword extraction method here can be the steps of the keyword extraction method in each of the above-mentioned embodiments.

[0196] In one embodiment, a computer-readable storage medium is provided, which stores a computer program. When the computer program is executed by a processor, the processor executes the steps of the keyword extraction method described above. The steps of the keyword extraction method here can be the steps of the keyword extraction method in each of the above embodiments.

[0197] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be completed by instructing the relevant hardware through a computer program, and the program can be stored in a non-volatile computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in the present application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0198] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0199] The above embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several modifications and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A keyword extraction method, comprising: Slide and extract words from the target text according to at least two co-occurrence windows to obtain multiple word combinations under each co-occurrence window; Obtain the first Huffman tree constructed based on the word vector model and the target text; The first Huffman tree includes a plurality of nodes, wherein the nodes include a root node and a leaf node corresponding to each word in the target text; Determine a first Huffman path of each leaf node in the first Huffman tree; Determine a first transition probability of the root node transferring to the corresponding leaf node based on the first Huffman path; Determining the co-occurrence probability between different words in each word combination according to the first transition probability; In each word combination, the words whose co-occurrence probability meets the screening condition are determined as candidate words; For each candidate word, determine the word frequency of the candidate word in the candidate word set composed of candidate words of each word combination, calculate the ratio of the word frequency to the conditional probability of the candidate word appearing in the corpus, and determine the contribution probability of the corresponding candidate word to the explanation of the target text according to the ratio; The candidate words whose contribution probabilities meet the threshold condition are determined as keywords.

2. The method according to claim 1, characterized in that: The word combination includes a central word and corresponding context words; and determining the co-occurrence probability between different words in each word combination according to the first transition probability includes: According to the first transition probability, calculating the conditional probability of the corresponding word appearing in the target text; According to the conditional probability of appearing in the target text, the co-occurrence probability of each context word and the central word in each word combination is calculated; In each word combination, determining the words whose co-occurrence probability meets the screening condition as candidate words includes: In each word combination, the context words whose co-occurrence probability meets the screening condition are determined as candidate words.

3. The method according to claim 1, characterized in that Determining a first transition probability of the root node transferring to the corresponding leaf node based on the first Huffman path includes: Determine a node vector and a node code corresponding to each node on the first Huffman path; the first Huffman path includes a plurality of nodes connected in series; Inputting the node vector of the preceding node among the adjacent nodes on the first Huffman path into the pre-trained classification model to obtain a classification probability; According to the classification probability of the preceding node and the node code of the succeeding node among the adjacent nodes, a first transition probability between corresponding adjacent nodes is determined.

4. The method according to claim 2, characterized in that: The step of calculating the co-occurrence probability of each context word and the central word in each word combination according to the conditional probability of the words appearing in the target text comprises: Determine the conditional probability of the context word appearing in the target text as the co-occurrence probability of the context word and the central word in each word combination; In each word combination, the context words whose co-occurrence probability meets the screening conditions are determined as candidate words, including: In each word combination, the context word with the greatest co-occurrence probability is determined as a candidate word.

5. The method according to claim 1, characterized in that: Before determining the word frequency of each candidate word in a candidate word set composed of candidate words of each word combination, calculating the ratio of the word frequency to the conditional probability of the candidate word appearing in the corpus, and determining the contribution probability of the corresponding candidate word to the explanation of the target text according to the ratio, the method further includes: Obtain a second Huffman tree constructed based on the word vector model and the corpus; the second Huffman tree includes a plurality of nodes; the nodes include a root node and a leaf node corresponding to each word in the corpus; Determine the second Huffman path of the second Huffman tree corresponding to the leaf node of each word in the target text; According to the second transition probability of the root node transferring to the corresponding leaf node based on the second Huffman path, the conditional probability of the corresponding word appearing in the corpus is calculated.

6. The method according to claim 1, characterized in that Before determining, for each candidate word, the word frequency of the candidate word in the candidate word set formed by the candidate words of each word combination, calculating the ratio of the word frequency to the conditional probability of the candidate word appearing in the corpus, and determining the contribution probability of the corresponding candidate word to the interpretation of the target text according to the ratio, the method further comprises: Determine the similarity distance between the word vectors corresponding to each candidate word in the candidate word set based on the word vector model; A plurality of words whose similarity distance is less than or equal to a predetermined threshold are determined as the same candidate words.

7. The method according to claim 1, characterized in that The step of determining the candidate words whose contribution probabilities meet the threshold condition as keywords comprises: Arrange the plurality of candidate words in descending order according to the contribution probability; Filter the candidate words with the highest ranking corresponding to the preset threshold; Determine the similarity distance between the word vectors corresponding to each word in the target text based on the word vector model; According to the similarity distance, obtaining similar words in the target text corresponding to the screened candidate words; The screened candidate words and corresponding similar words are determined as keywords.

8. The method according to any one of claims 1 to 7, characterized in that: The steps of training the word vector model include: Obtain a word vector model to be trained; the word vector model includes a word embedding layer and a neural network layer; Calculating the similarity distance between the word vectors corresponding to each word in the corpus based on the word embedding layer; Calculate the conditional probability of each word appearing in the corpus based on the neural network layer; Calculate the difference between the conditional probabilities corresponding to the two words whose similarity distance is less than or equal to the first threshold; According to the difference between the difference and the second threshold, the parameters of the neural network layer are adjusted and training is continued until the training is terminated when the difference is less than or equal to the second threshold.

9. A keyword extraction device, characterized in that: The device comprises: A sliding word extraction module is used to slide words from the target text according to at least two co-occurrence windows to obtain multiple word combinations under each co-occurrence window; A candidate word screening module is used to obtain a first Huffman tree constructed based on a word vector model and a target text; the first Huffman tree includes a plurality of nodes, the nodes including a root node and a leaf node corresponding to each word in the target text; determine a first Huffman path of each leaf node in the first Huffman tree; determine a first transition probability of the root node transferring to the corresponding leaf node based on the first Huffman path; determine the co-occurrence probability between different words in each word combination according to the first transition probability; in each word combination, determine the words whose co-occurrence probability meets the screening condition as candidate words; A contribution probability calculation module, for determining the contribution probability of each candidate word to the explanation of the target text according to the ratio of the word frequency of each candidate word in the candidate word set composed of the candidate words of each word combination to the conditional probability of each candidate word appearing in the corpus; The keyword extraction module is used to determine the candidate words whose contribution probability meets the threshold condition as keywords.

10. The device according to claim 9, characterized in that The word combination includes a central word and corresponding context words; the candidate word screening module is also used to calculate the conditional probability of the corresponding word appearing in the target text based on the first transfer probability; based on the conditional probability of appearing in the target text, the co-occurrence probability of each context word and the central word in each word combination is calculated respectively; in each word combination, the context words whose co-occurrence probability meets the screening conditions are determined as candidate words.

11. The device according to claim 9, characterized in that The candidate word screening module is also used to determine the node vector and node code corresponding to each node on the first Huffman path; the first Huffman path includes a plurality of nodes connected in series; the node vector of the preceding node among the adjacent nodes on the first Huffman path is input into a pre-trained classification model to obtain a classification probability; and the first transition probability between corresponding adjacent nodes is determined based on the classification probability of the preceding node among the adjacent nodes and the node code of the succeeding node.

12. The device according to claim 10, characterized in that The candidate word screening module is also used to determine the conditional probability of the context word appearing in the target text as the co-occurrence probability of the context word and the central word in each word combination; in each word combination, the context word with the largest co-occurrence probability is determined as the candidate word.

13. The device according to claim 9, characterized in that The device also includes: a conditional probability calculation module, which is used to obtain a second Huffman tree constructed based on a word vector model and a corpus before determining the contribution probability of each candidate word to the explanation of the target text based on the ratio of the word frequency of each candidate word in the candidate word set composed of candidate words in each word combination to the conditional probability of each candidate word appearing in the corpus; the second Huffman tree includes a plurality of nodes; the nodes include a root node and a leaf node corresponding to each word in the corpus; determine a second Huffman path of each word in the target text corresponding to the leaf node in the second Huffman tree; and calculate the conditional probability of the corresponding word appearing in the corpus according to the second transition probability of the root node transferring to the corresponding leaf node based on the second Huffman path.

14. The device according to claim 9, characterized in that The contribution probability calculation module is also used to determine the similarity distance between the word vectors corresponding to each candidate word in the candidate word set based on the word vector model; and determine multiple words whose similarity distance is less than or equal to a predetermined threshold as the same candidate word.

15. The device according to claim 9, characterized in that The keyword extraction module is also used to arrange the multiple candidate words in descending order according to the contribution probability; filter the top-ranked candidate words corresponding to a preset threshold; determine the similarity distance between the word vectors corresponding to each word in the target text based on the word vector model; obtain similar words in the target text corresponding to the filtered candidate words according to the similarity distance; and determine the filtered candidate words and the corresponding similar words as keywords.

16. The device according to any one of claims 9 to 15, characterized in that The device also includes: a word vector model training module, used to obtain a word vector model to be trained; the word vector model includes a word embedding layer and a neural network layer; based on the word embedding layer, the similarity distance between the word vectors corresponding to each word in the corpus is calculated; based on the neural network layer, the conditional probability of each word appearing in the corpus is calculated; the difference between the conditional probabilities corresponding to two words whose similarity distance is less than or equal to a first threshold is calculated; according to the difference between the difference and a second threshold, the parameters of the neural network layer are adjusted and the training is continued until the training is terminated when the difference is less than or equal to the second threshold.

17. A computer-readable storage medium storing a computer program, wherein when the computer program is executed by a processor, the processor is caused to perform the steps of the method according to any one of claims 1 to 8.

18. A computer device comprising a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the method according to any one of claims 1 to 8.

Images