Keyword extraction method, device, equipment, storage medium and computer program

Abstract

This disclosure relates to the field of text processing technology, and particularly to a keyword extraction method, apparatus, device, storage medium, and computer program. The method includes: inputting text data to be processed into a pre-trained word vector model for vector mapping processing to obtain word vectors; inputting the word vectors into a bidirectional long short-term memory network for global feature extraction to obtain global features, wherein the global features represent the global features corresponding to the sentence; inputting the word vectors into a text convolutional neural network for local feature extraction to obtain local features, wherein the local features represent the local features corresponding to the word; constructing a probability transition matrix based on the global features and the local features, wherein the probability transition matrix is ​​used to calculate the importance score corresponding to the word; and selecting keywords from the words based on the importance score. This method can efficiently and accurately extract keywords from text data.

Description

Technical Field

[0001] This disclosure relates to the field of text processing technology, and in particular to a keyword extraction method, apparatus, device, storage medium, and computer program. Background Technology

[0002] Keyword extraction refers to the process of automatically identifying words or phrases from text that best represent the text's theme. Keywords are typically concise, highly informative words that effectively summarize the core content of the text, helping readers quickly understand its topic. Keyword extraction is widely used in information retrieval, text classification, and recommender systems.

[0003] Traditional approaches to keyword extraction include frequency-based extraction, which sorts the words by their frequency of occurrence in the text and selects the most frequent words as keywords; and statistical extraction, which calculates the relationships between words and constructs a word graph to assess their importance.

[0004] These methods rely on word frequency as the primary basis, which leads to the neglect of words with semantic relationships, the inability to accurately capture important information in the context, and the lack of in-depth mining of semantic information. As a result, the extracted keywords have certain limitations in terms of accuracy and semantic representativeness. They also often fail to consider the different meanings of words in different contexts, which may cause the same word to be incorrectly identified as a keyword due to different contexts. Summary of the Invention

[0005] This disclosure provides a keyword extraction method, apparatus, device, storage medium, and computer program that can solve the technical problem of how to efficiently and accurately extract keywords from text data.

[0006] Firstly, this disclosure provides a keyword extraction method, including:

[0007] The text data to be processed is input into a pre-trained word vector model for vector mapping to obtain word vectors. The text data to be processed includes sentences, which are composed of words.

[0008] Word vectors are input into a bidirectional long short-term memory network for global feature extraction to obtain global features, which are used to represent the global features corresponding to the sentences.

[0009] Word vectors are input into a text convolutional neural network for local feature extraction, resulting in local features that represent the local features corresponding to words.

[0010] A probability transition matrix is ​​constructed based on global and local features. The probability transition matrix is ​​used to calculate the importance score of the corresponding words.

[0011] Keywords are selected from words based on their importance scores, and the importance scores of the keywords must meet the preset score conditions.

[0012] In some embodiments, word vectors are input into a bidirectional long short-term memory network for global feature extraction to obtain global features, including:

[0013] Using the forward propagation algorithm, the forward hidden state of the current time step corresponding to the word vector of the current time step is calculated based on the forward hidden state of the previous time step corresponding to the word vector of the current time step.

[0014] Using the backpropagation algorithm, the reverse hidden state of the current time step corresponding to the word vector at the current time step is calculated based on the reverse hidden state of the next time step corresponding to the word vector at the current time step.

[0015] The global features are calculated based on the forward hidden state and the reverse hidden state at the current time step.

[0016] In some embodiments, the global features are obtained by combining the forward hidden state of the current time step and the backward hidden state of the current time step, including:

[0017] The forward hidden state and the reverse hidden state of the current time step are concatenated to obtain the target hidden state of the current time step.

[0018] Perform average pooling on the hidden state of the target at the current time step to obtain global features.

[0019] In some embodiments, word vectors are input into a text convolutional neural network for local feature extraction to obtain local features, including:

[0020] Perform a convolution operation on the word vectors to obtain the feature maps corresponding to the word vectors;

[0021] Max pooling is performed on the feature map to obtain local features.

[0022] In some embodiments, a probability transition matrix is ​​constructed based on global features and local features, including:

[0023] The global and local features are concatenated to obtain the target features;

[0024] Similarity is calculated based on target features to obtain a similarity coefficient, which is used to represent the relevance between words.

[0025] The similarity coefficients are normalized to obtain the probability transition matrix.

[0026] In some embodiments, keywords are selected from words based on importance scores, including:

[0027] Using a text ranking algorithm, the importance score corresponding to each word is calculated iteratively based on the probability transition matrix;

[0028] The words are sorted according to their importance scores to obtain a word sorting sequence;

[0029] Keywords are selected from words based on their word order sequence.

[0030] Secondly, this disclosure provides a keyword extraction device, comprising:

[0031] The mapping unit is used to perform vector mapping processing on the text data to be processed according to the pre-trained word vector model to obtain word vectors. The text data to be processed includes sentences, which are composed of words.

[0032] The extraction unit is used to input word vectors into a bidirectional long short-term memory network for global feature extraction, thereby obtaining global features. These global features are used to represent the global features corresponding to the sentences.

[0033] The extraction unit is also used to input word vectors into a text convolutional neural network for local feature extraction, thereby obtaining local features, which are used to represent the local features corresponding to the words.

[0034] The calculation unit is used to construct a probability transition matrix based on the global features of the sentence and the local features of the words. The probability transition matrix is ​​used to calculate the importance score of the corresponding words.

[0035] The filtering unit is used to filter keywords from words based on their importance scores, where the importance scores of the keywords meet the preset score conditions.

[0036] Thirdly, this disclosure provides a computer device including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the method described in the foregoing aspects.

[0037] Fourthly, this disclosure provides a computer-readable storage medium having a computer program stored thereon that, when executed by a processor, implements the steps of the method described in the above aspects.

[0038] Fifthly, this disclosure provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the steps of the methods described in the foregoing aspects.

[0039] This disclosure provides a keyword extraction method, apparatus, device, storage medium, and computer program. It converts words in the text data to be processed into word vectors using a pre-trained word vector model, enhancing semantic understanding. High-quality word vectors effectively understand the relationships between words, aiding subsequent steps in better processing text information. A bidirectional long short-term memory network extracts global features from the context, considering not only the preceding and following relationships of the current word but also a comprehensive understanding of the entire sentence, avoiding semantic biases that may result from relying solely on local information. A text convolutional neural network extracts effective local features from neighboring words, reflecting the importance of words in a specific context and improving the accuracy of keyword extraction. A probability transition matrix is ​​generated by combining global and local features, reflecting the relevance and transition probabilities between words and capturing their dependencies. The importance score of each word is obtained by calculating the probability transition matrix, quantitatively representing the relevance and similarity of words. This representation helps identify important core words, rather than simply relying on frequency statistics, providing a precise metric for keyword extraction. It effectively distinguishes between primary and secondary words, ensuring high relevance and effectiveness of the results, and significantly improving the efficiency and accuracy of keyword extraction. Attached Figure Description

[0040] The present disclosure will be described in more detail below based on embodiments and with reference to the accompanying drawings:

[0041] Figure 1 This is a flowchart illustrating a keyword extraction method provided in an embodiment of this disclosure;

[0042] Figure 2 This is a flowchart illustrating a keyword extraction method provided in an embodiment of this disclosure;

[0043] Figure 3 This is a schematic diagram of the structure of a keyword extraction device provided in an embodiment of this disclosure;

[0044] Figure 4 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this disclosure.

[0045] In the accompanying drawings, the same parts are referred to by the same reference numerals, and the drawings are not drawn to scale. Detailed Implementation

[0046] To enable those skilled in the art to better understand the technical solutions of this disclosure, and to fully understand and implement the process of how this disclosure applies technical means to solve technical problems and achieve corresponding technical effects, the technical solutions in the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this disclosure, not all embodiments. The embodiments of this disclosure and the various features within them can be combined with each other without conflict, and the resulting technical solutions are all within the protection scope of this disclosure. Based on the embodiments of this disclosure, all other embodiments obtained by those skilled in the art without creative effort should fall within the protection scope of this disclosure.

[0047] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this disclosure described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0048] It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0049] Example 1

[0050] Figure 1 This is a flowchart illustrating a keyword extraction method provided in an embodiment of this disclosure. For example... Figure 1 As shown, the process includes the following steps S101 to S105.

[0051] S101. Input the text data to be processed into the pre-trained word vector model for vector mapping processing to obtain word vectors.

[0052] The text data to be processed includes sentences, which are composed of words. It can also be understood as text content that needs to be analyzed or processed, typically consisting of one or more sentences.

[0053] A pre-trained word vector model refers to a model that has been trained on a large-scale text dataset and can convert words into vectors of a fixed dimension. That is, by capturing the semantic relationships between words, the model can map words with similar semantics to a nearby vector space.

[0054] For example, the Robustly Optimized BERT Pretraining Approach (RoBERTa) can be adopted. It can be understood from the relevant principles that RoBERTa is a pre-trained model based on Bidirectional Encoder Representations from Transformers (BERT).

[0055] Vector mapping processing refers to the process of converting words into numerical vectors through a pre-trained word vector model. Each word corresponds to a vector of a fixed dimension, representing the position of the word in the semantic space.

[0056] It can also be understood that the pre-trained word vector model learns the relationships between words by analyzing a large amount of text data, and these relationships are encoded in the model in the form of vectors. The mapping process is to calculate the representations of words in each text in the vector space through the model weights. The similarity in the vector space reflects semantic similarity, so these vectors can be used for subsequent text analysis tasks.

[0057] As an example but not a limitation, assume the text to be processed is: "Today's weather is cloudy turning to sunny".

[0058] After word segmentation, the words obtained are: ["Today", "the", "weather", "is", "cloudy", "turning", "to", "sunny"].

[0059] Input each word into the pre-trained word vector model for vector mapping, and the results may be:

[0060] "Today" → [0.1, 0.2, 0.3,...].

[0061] "the" → [0.0, 0.1, 0.4,...].

[0062] "weather" → [0.5, 0.3, 0.2,...].

[0063] "is" → [0.2, 0.0, 0.1,...].

[0064] "cloudy" → [0.4, 0.6, 0.1,...].

[0065] "Turn" → [0.3, 0.5, 0.2, ...].

[0066] "Sunny" → [0.7, 0.2, 0.4, ...].

[0067] In other words, each word is mapped to a fixed-dimensional word vector through a pre-trained word vector model (RoBERTa). After the vector mapping is completed, a word vector matrix is ​​obtained, where each row corresponds to the vector of a word (i.e., word vector). Let the word vector dimension be d and the sentence length be n, then the input matrix X is obtained with a dimension of n×d.

[0068] RoBERT generates context-sensitive word vectors, meaning that the representation of each word takes into account its position and context within a sentence, thus better capturing the semantics of the words. Secondly, RoBERT's multi-layered transformer structure deeply understands the complexity of language, enabling it to identify and distinguish subtle differences in words across different contexts. Unlike traditional one-way language models, RoBERT improves the accuracy of keyword extraction results by providing high-quality word vector representations.

[0069] S102. Input the word vectors into a bidirectional long short-term memory network for global feature extraction to obtain global features.

[0070] Global features are used to represent the global features corresponding to a statement.

[0071] Bidirectional Long Short-Term Memory (Bi-LSTM) is an improved recurrent neural network (RNN) that can simultaneously consider both past and future information in sequence data. LSTM is a special RNN structure capable of effectively handling and predicting long-range dependencies in time series data. Bidirectional LSTM processes information through two LSTM layers: one processes the sequence forward (left to right) and the other backward (right to left). This structure allows the model to capture bidirectional dependencies based on contextual information.

[0072] Global features refer to features extracted from the entire input sequence when processing text or sequence data. These features reflect the semantic and structural information of the entire sequence. In contrast to local features, which typically focus only on information from a certain part or specific locations, global features can be used to more comprehensively understand and represent the meaning of the entire input sequence.

[0073] Global features corresponding to a statement can be understood as global features extracted from a given statement, representing the overall semantics and important information of that statement. By utilizing these global features, the statement can be better understood and processed to facilitate keyword extraction.

[0074] Using the example above, we input the word vectors from that example into a bidirectional LSTM. The bidirectional LSTM will process each word, for example:

[0075] The forward LSTM processing order is: "Today" → "of" → "weather" → "for" → "cloudy" → "turn" → "sunny".

[0076] The backward LSTM processing order is: "Sunny" → "Turn" → "Cloudy" → "For" → "Weather" → "Of" → "Today".

[0077] In the final layer output of the bidirectional LSTM, by combining the forward and backward information, we can obtain the contextual representation of each word. Then, through pooling (such as max pooling or average pooling), these word vectors are merged into a fixed-dimensional global feature vector, representing the semantic information of the entire sentence. This is how we obtain the global features corresponding to the sentence. This global feature vector can reflect the overall meaning of the sentence "Today's weather is cloudy turning sunny," including information such as weather conditions and time.

[0078] Bidirectional LSTM can capture features in sequence data more comprehensively by considering both forward and backward information from the context. Global features are extracted from the entire sequence and can effectively represent the overall semantics of the sentence. In other words, using bidirectional LSTM to extract global features can integrate and capture the semantic information of all words in the sentence, providing a global perspective for subsequent keyword extraction and improving the ability to understand the whole text and grasp the context.

[0079] S103. Input the word vectors into the text convolutional neural network to extract local features and obtain local features.

[0080] Local features are used to represent the local features corresponding to words.

[0081] Text-CNN is a convolutional neural network architecture specifically designed for text data. It extracts local features from text through convolution operations and uses a combination of convolutional and pooling layers to capture important information in the text.

[0082] Local features refer to features extracted from local regions or specific segments of data during processing. These features typically reflect the local structure or patterns of the data, rather than overall information. In text processing, local features can be phrases, word groups, or information within a specific context.

[0083] Local features corresponding to a word refer to the semantic information and characteristics contained in a word or phrase within a specific context. These features are usually related to the word's position in the sentence, the influence of surrounding words, etc., and can help to better understand the meaning of the word.

[0084] Building upon the example above, we use convolutional layers to perform convolution operations on the input matrix (word vector matrix). Assuming a convolution kernel with a window size of 2, the convolution operation will extract combined features from two adjacent words, for example:

[0085] For "today's weather", the convolution kernel will extract the local features of "today" and "of".

[0086] For the condition "the weather is cloudy", the convolution kernel will extract the local features of "weather" and "is".

[0087] Similarly, the convolution operation slides across the entire sentence, extracting all possible local features.

[0088] Following convolutional layers, pooling layers (such as max pooling) are typically used to reduce feature dimensionality and preserve the most salient local features. These local features can be phrases or information important in a specific context. After convolution and pooling, we obtain a feature vector that represents the local features of each word in the sentence. For example, the extracted local features might include phrases such as "the weather is cloudy" or "cloudy turning sunny".

[0089] TextCNN captures local patterns in text, such as n-grams (consecutive combinations of n words), through convolutional layers. This helps in understanding the semantics and contextual relationships of words, effectively capturing the relationships between adjacent words and extracting information from phrases and local context. Because convolutional layers are translation-invariant, they can identify similar patterns appearing in different positions, improving the accuracy of keyword extraction from text. Furthermore, convolutional operations can process large amounts of data in parallel, improving computational efficiency.

[0090] S104. Construct a probability transition matrix based on global and local features.

[0091] The probability transition matrix is ​​used to calculate the importance score corresponding to a word.

[0092] A probability transition matrix is ​​a matrix used to represent the probability of transitions between states. In Natural Language Processing (NLP), probability transition matrices can be used to represent the relationships and transition probabilities between words, and are often used to calculate the importance or relevance of words in a specific context.

[0093] Importance score is a score that measures how important a word is in a particular text or context. Importance score can help identify keywords, themes, or important information in text.

[0094] Based on the above, obtaining global and local features allows us to construct a probability transition matrix. As an example, not a limitation, we can calculate the similarity between words using global and local features, then construct the probability transition matrix based on the similarity, and obtain the probability by normalizing the correlation coefficient. Based on the constructed probability transition matrix, the TextRank algorithm is used to iteratively calculate the importance scores of the words.

[0095] TextRank is an undirected graph-based ranking algorithm used for keyword extraction and summary generation in text. The algorithm treats words in the text as nodes in a graph, and the similarity or relationship between words as edges. By iteratively calculating the importance scores of nodes, TextRank can effectively identify keywords and important sentences in text. Based on a graph structure, TextRank can effectively handle relationships between words and has a certain degree of robustness. Even with some missing information, the algorithm can still assess the importance of words well. As an unsupervised learning method, TextRank does not require manually labeled data and can be effectively applied to various text data scenarios, making its application more flexible.

[0096] By calculating the similarity between words and constructing a probability transition matrix, the TextRank algorithm can identify keywords and important concepts in text. Utilizing the relationship between global and local features, it can more comprehensively reflect the semantics of the text; this correlation enhances the accuracy of text analysis, especially when dealing with complex text. Through the above steps, the TextRank algorithm can automatically select the most important words as features, making feature selection more efficient and objective. Normalizing the correlation coefficients ensures that the importance score of each word is within a uniform range, facilitating comparison and analysis.

[0097] S105. Select keywords from the words based on their importance scores.

[0098] The importance scores corresponding to the keywords meet the preset score conditions.

[0099] Importance scores, based on factors such as the contextual relationships of words, the similarity between words, and the frequency of word occurrence in the text, can be understood as a numerical measure of the importance of a word in a specific text or context. It reflects the relevance and salience of a word within the text. Generally, a higher importance score indicates that the word is more important in the text.

[0100] Keywords refer to the words that best represent the theme, content, or core idea of a text. Keywords are usually the focus of readers or analysts, and they help to quickly understand the main content of the text.

[0101] The preset score condition refers to a threshold or standard set when screening keywords. Only when the importance score of a word exceeds a certain preset score will the word be selected as a keyword. This condition can be adjusted according to specific application scenarios to ensure that the extracted keywords are representative and relevant enough.

[0102] Combined with the example in the above text, calculate the importance score of each word through the TextRank algorithm. Suppose the scores obtained are as follows:

[0103] "Today": 0.06.

[0104] "Of": 0.0.

[0105] "Weather": 0.3.

[0106] "Is": 0.0.

[0107] "Cloudy": 0.36.

[0108] "Turn": 0.28.

[0109] "Sunny": 0.4.

[0110] Suppose the preset score condition is set at 0.2. This means that only when the importance score of a word is greater than or equal to 0.2 will it be considered a keyword.

[0111] According to the above importance scores, conduct screening:

[0112] "Today": 0.06 (does not meet the condition).

[0113] "Of": 0.0 (does not meet the condition).

[0114] "Weather": 0.3 (meets the condition, selected as a keyword).

[0115] "Is": 0.0 (does not meet the condition).

[0116] "Cloudy": 0.36 (meets the condition, selected as a keyword).

[0117] "Turn": 0.28 (meets the condition, selected as a keyword).

[0118] "Sunny": 0.4 (meets the condition, selected as a keyword).

[0119] After screening, the obtained keywords are: "Weather", "Cloudy", "Turn", "Sunny".

[0120] By filtering keywords based on their importance scores, key information in text can be effectively extracted. By setting preset score conditions, the keyword extraction process can be automated, improving efficiency.

[0121] By converting words in the text data to be processed into word vectors through a pre-trained word vector model, semantic understanding is enhanced. High-quality word vectors can effectively understand the relationships between words, helping subsequent steps to better process text information. A bidirectional long short-term memory network can extract global features from the context, considering not only the relationship between the current word but also the comprehensive understanding of the entire sentence, avoiding semantic biases that may be caused by relying solely on local information. A text convolutional neural network can extract effective local features from neighboring words, reflecting the importance of words in specific contexts and improving the accuracy of keyword extraction. A probability transition matrix is ​​generated by combining global and local features to reflect the relevance and transition probability between words, capturing the dependencies between words. By calculating the probability transition matrix, the importance score of each word is obtained, so that the relevance and similarity of words are quantitatively represented. This representation method can help identify important core words, rather than just simple statistics based on frequency, providing a precise metric for keyword extraction, effectively distinguishing between primary and secondary words, ensuring high relevance and effectiveness of the results, and significantly improving the efficiency and accuracy of keyword extraction.

[0122] Example 2

[0123] Based on the above embodiments, word vectors are input into a bidirectional long short-term memory network for global feature extraction to obtain global features, including: using the forward propagation algorithm, calculating the current time step forward hidden state corresponding to the word vector at the current time step based on the previous time step forward hidden state corresponding to the word vector at the current time step; using the back propagation algorithm, calculating the current time step back hidden state corresponding to the word vector at the current time step based on the next time step back hidden state corresponding to the word vector at the current time step; and calculating the global features based on the current time step forward hidden state and the current time step back hidden state.

[0124] The current time step refers to the specific moment (or word) that the model is processing during sequence processing. For example, when processing the sentence "Today's weather is cloudy turning sunny," the moment the model processes the word "weather" is the current time step.

[0125] During forward propagation, the forward hidden state of the previous time step refers to the hidden state of the model when processing the previous time step before processing the current time step. It contains information about the previous word, which helps the model understand the context of the current word.

[0126] The forward hidden state at the current time step refers to the hidden state calculated through forward propagation when processing the current time step. It combines the word vector at the current time step and the hidden state at the previous time step, representing the information of the current word in the sequence.

[0127] During backpropagation, the backward hidden state at the next time step refers to the hidden state of the model when processing the next time step after processing the current time step. It contains information about subsequent words, which helps the model understand the context of the current word.

[0128] The backward hidden state at the current time step refers to the hidden state calculated through backpropagation when processing the current time step. It combines the word vector at the current time step and the hidden state at the next time step, representing the information of the current word in the backward sequence.

[0129] In some embodiments, the global features are obtained by combining the forward hidden state and the reverse hidden state of the current time step, including: concatenating the forward hidden state and the reverse hidden state of the current time step to obtain the target hidden state of the current time step; and performing an average pooling operation on the target hidden state of the current time step to obtain the global features.

[0130] In bidirectional long short-term memory networks (also known as Bilstm), the model performs forward and backward sequence processing simultaneously to capture contextual information. The relevant principles are as follows.

[0131] The preprocessed word vector matrix X is input into a bidirectional long short-term memory network. Bilstm processes the sentence from both the forward and backward directions.

[0132] During forward propagation, for each time step t (t = 1, 2, ..., n), the forward hidden state is... The calculation formula is:

[0133]

[0134] Where, x t It is the word vector of the t-th word. It is the positive hidden state of the previous time step. This represents the forward computation process of an LSTM unit, which internally includes forget gates, input gates, output gates, and memory unit update operations. The specific formulas are as follows:

[0135]

[0136] Where W and U are weight matrices, b is a bias vector, σ is the sigmoid activation function, and ° represents element-wise multiplication.

[0137] Backpropagation, back hidden state The calculation formula is:

[0138]

[0139] in, It is the reverse hidden state at the next time step. This represents the reverse LSTM cell computation process, whose formula is similar to the forward one, but the computation direction is opposite.

[0140] Finally, the forward and reverse hidden states are concatenated to obtain the fused representation of each word:

[0141]

[0142] At this point, the dimension of H is n×2h (let the dimension of the hidden layer be h).

[0143] Then, average pooling is performed on the hidden states of all words to obtain the global feature vector s of the entire sentence. global The calculation formula is:

[0144]

[0145] Where n is a positive integer.

[0146] By using a bidirectional LSTM, the model can simultaneously consider contextual information, thus better understanding the meaning of the current word. By combining forward and backward hidden states, the resulting global features are richer and can capture more semantic information.

[0147] Example 3

[0148] Based on the above embodiments, word vectors are input into a text convolutional neural network for local feature extraction to obtain local features, including: performing convolution operations on word vectors to obtain feature maps corresponding to word vectors; and performing max pooling operations on the feature maps to obtain local features.

[0149] Convolution operations involve converting word vectors into a two-dimensional matrix (typically a matrix the size of a window / convolution kernel) and then using convolution to extract features. By sliding a convolution kernel (a small filter) across the word vector matrix and performing a dot product, a feature map is obtained. The convolution result at each location represents the features within the context of that location. The result of the convolution operation is a feature map representing the key features extracted from the input word vectors. The feature map highlights patterns that are relevant and important to the target task. Next, max pooling is performed on the feature map. The purpose of max pooling is to extract the most salient features from the feature map, reducing dimensionality and computational cost. During pooling, the maximum value within each convolution region is taken to form a new feature vector, called a local feature. This feature vector can better represent the important features of that region. The specific principle is as follows, based on the example above.

[0150] Using the preprocessed word vector matrix X, we design multiple convolution kernels of different sizes to perform convolution operations on the word vector matrix. If we use a convolution kernel with width k and the number of kernels is m, then the feature map obtained after the convolution operation has a dimension of (n-k+1)×m.

[0151] For each convolution kernel j (j = 1, 2, ..., m), the convolution at position i (i = 1, 2, ..., n-k+1) is calculated as follows:

[0152]

[0153] Among them, w jl It is the l-th weight vector of the j-th convolutional kernel, x i+l It is the word vector at the corresponding position, b j f is the bias term, and f is the activation function (ReLU).

[0154] Max pooling is performed on each feature map to obtain the local feature vector extracted by each convolutional kernel. For the feature map of the j-th convolutional kernel, the max pooling formula is:

[0155] p j =max 1≤i≤n-k+1 (c ij (7)

[0156] These local feature vectors are concatenated to obtain the local feature representation matrix C for each word, with a dimension of n×p (assuming the total dimension after concatenation is p).

[0157] By employing convolution and pooling operations, the most relevant features can be automatically identified from word vectors, finding words with high information content within the context, rather than relying solely on word frequency. Since the semantics of many keywords depend on their surrounding words, convolution operations effectively capture the relationships between multiple adjacent words, facilitating the understanding of the local context of words in text. Max pooling, as a dimensionality reduction technique, can filter out smaller, less important features, retaining only conceptually prominent features, thereby reducing the influence of irrelevant information in the text and improving keyword accuracy. In other words, local features extracted through convolutional neural networks can help to more accurately identify and extract keywords; the use of pooling operations significantly reduces the dimensionality of feature maps, thereby improving computational efficiency and making it more efficient when processing large-scale data.

[0158] Example 4

[0159] Based on the above embodiments, a probability transition matrix is ​​constructed according to global features and local features, including: concatenating global features and local features to obtain target features; calculating similarity based on target features to obtain similarity coefficients, which are used to represent the correlation between words; and normalizing the similarity coefficients to obtain the probability transition matrix.

[0160] Global features are the outputs of a bidirectional long short-term memory network (Bi-LSTM), representing the overall information of a word in its context.

[0161] Local features are the output of a text convolutional neural network (CNN) and represent the specific information of words in their local context.

[0162] By combining these two features, the target features are obtained, indicating that the overall and local information of the words are combined, thus forming a richer feature representation.

[0163] Similarity coefficients are calculated based on target features to assess the relevance between different words. A higher similarity coefficient indicates that the two words are closer in the feature space, meaning they are also semantically related.

[0164] The similarity coefficients are normalized so that the values ​​in the entire matrix are between 0 and 1, which facilitates subsequent calculations and comparisons. The resulting probability transition matrix after normalization can be used to represent the transition probabilities between words.

[0165] Based on the example above, the global feature vector s extracted by Bilstm for the sentence... global The local feature matrix C of each word extracted by TextCNN is concatenated to obtain the comprehensive feature vector F of each word, with a dimension of n×(2h+p).

[0166] To calculate the similarity between words, the Pearson correlation coefficient is used. For word i and word j, the Pearson correlation coefficient r... ij The calculation is as follows:

[0167]

[0168] in, and The combined feature vectors F for words i and j are respectively. i and F j The mean.

[0169] Construct a probability transition matrix M based on similarity, where M ij This represents the probability of transitioning from word i to word j. The probability is obtained by normalizing the correlation coefficient, using the following formula:

[0170]

[0171] By combining global and local features, the model provides a richer representation of each word, enabling it to capture the meaning of the text more comprehensively and thus making keyword extraction more accurate. Calculating similarity coefficients quantifies the relationships between words, helping to identify the words most relevant to the text's topic and laying a solid foundation for further keyword extraction. The normalized probability transition matrix provides a relatively consistent standard for calculating and comparing word importance. This allows the model to evaluate the status of each word more fairly.

[0172] In some embodiments, selecting keywords from words based on importance scores includes: using a text ranking algorithm to iteratively calculate the importance scores corresponding to words based on a probability transition matrix; sorting the words according to their importance scores to obtain a word ranking sequence; and selecting keywords from the words based on the word ranking sequence.

[0173] Using the pre-constructed probability transition matrix, text ranking algorithms (such as TextRank) can be implemented, iteratively calculating the importance score of each word through random walks. The importance score of each word reflects its centrality within the overall text structure.

[0174] After calculating the importance scores of all words, the words are sorted according to their scores, generating a word ranking sequence. Words with higher scores in the ranking sequence are considered more important.

[0175] Based on the word sorting sequence, the top N words with the highest importance scores are selected as the keywords of the text.

[0176] Based on the example above, and using the constructed probability transition matrix M, the TextRank algorithm is employed to iteratively calculate the importance score of words. In the t-th iteration, the importance score of word i is... The updated formula is:

[0177]

[0178] Where d is the damping coefficient, the iterative calculation continues until convergence, the words are sorted according to the final importance score, and the top-ranked words are selected as the extracted keywords.

[0179] By utilizing the TextRank algorithm, implicit connections between words can be effectively uncovered, ensuring that the extracted keywords are highly relevant to the text's theme, thus enhancing the accuracy of extraction. By automatically assessing the importance of keywords without human intervention, the efficiency of keyword extraction is greatly improved.

[0180] In other words, by combining Bilstm and TextCNN to extract global features of sentences and local features of words respectively, a probability transition matrix can be constructed more accurately. This improves the accuracy and semantic relevance of keyword extraction, providing higher-quality keywords for natural language processing tasks such as information retrieval, text summarization, and knowledge graph construction. Using a text ranking algorithm, importance scores are further calculated. After sorting and filtering, the final keywords effectively represent the core content of the text, making keyword extraction more efficient and accurate.

[0181] Example 5

[0182] Based on the above embodiments, this embodiment provides an application example.

[0183] Figure 2 This is a flowchart illustrating a keyword extraction method provided in an embodiment of this disclosure. For example... Figure 2 As shown, the process includes the following steps S201 to S205.

[0184] S201. Input text preprocessing.

[0185] The system receives text data to be processed, performs preprocessing operations such as word segmentation and stop word removal, and converts the text into a sequence of words. Each word is mapped to a fixed-dimensional word vector through a pre-trained word vector model (RoBERTa). Let the word vector dimension be d and the sentence length be n, to obtain the input matrix X with a dimension of n×d.

[0186] S202 and Bilstm extract global features of sentences.

[0187] The preprocessed word vector matrix X is input into a bidirectional long short-term memory network (Bilstm). Bilstm processes the sentence from both the forward and backward directions.

[0188] During forward propagation, for each time step t (t = 1, 2, ..., n), the forward hidden state is... The calculation formula is given in formula (1) above, and will not be repeated here. It includes the forget gate, input gate, output gate and memory unit update operations. The specific formula is given in formula (2) above, and will not be repeated here.

[0189] Backpropagation, back hidden state The calculation formula is shown in formula (3) above, and will not be repeated here. Its formula is similar to that of the positive direction, but the calculation direction is reversed.

[0190] Finally, the forward and reverse hidden states are concatenated to obtain the fused representation of each word, as shown in formulas (4) to (5) above, which will not be repeated here.

[0191] S203, TextCNN extracts local features of words.

[0192] Using the preprocessed word vector matrix X, we design multiple convolution kernels of different sizes to perform convolution operations on the word vector matrix. If we use a convolution kernel with width k and the number of kernels is m, then the feature map obtained after the convolution operation has a dimension of (n-k+1)×m.

[0193] For each convolution kernel j (j = 1, 2, ..., m), the convolution calculation formula at position i (i = 1, 2, ..., n-k+1) is given in formulas (6) to (7) above, and will not be repeated here.

[0194] These local feature vectors are concatenated to obtain the local feature representation matrix C for each word, with a dimension of n×p (assuming the total dimension after concatenation is p).

[0195] S204. Construct the probability transition matrix.

[0196] The global feature vector s of the sentence extracted by Bilstm global The local feature matrix C of each word extracted by TextCNN is concatenated to obtain the comprehensive feature vector F of each word, with a dimension of n×(2h+p).

[0197] To calculate the similarity between words, the Pearson correlation coefficient is used. For word i and word j, the Pearson correlation coefficient r... ij The calculation formula is shown in formula (8) above, and will not be repeated here.

[0198] The probability is obtained by normalizing the correlation coefficient. The calculation formula is shown in formula (9) above, and will not be repeated here.

[0199] S205, TextRank iterative calculation of keywords.

[0200] Based on the constructed probability transition matrix M, the TextRank algorithm is used to iteratively calculate the importance score of words. In the t-th iteration, the importance score of word i is... The updated formula is shown in formula (10) above, and will not be repeated here.

[0201] The calculation is iterated until convergence. The words are then ranked according to their final importance scores, and the top-ranked words are selected as the extracted keywords.

[0202] By extracting global sentence features and local word features using Bilstm and TextCNN respectively, we can capture the semantic information of the text more comprehensively. This allows the constructed probability transition matrix to more accurately reflect the semantic relationships between words, effectively improving the accuracy of keyword extraction compared to the traditional TextRank method. This leads to a more precise construction of the probability transition matrix, enhancing the accuracy and semantic relevance of sentence keyword extraction, and providing higher-quality keywords for natural language processing tasks such as information retrieval, text summarization, and knowledge graph construction.

[0203] The above method first preprocesses the input text, transforming it into a word vector matrix. Then, Bilstm is used to extract global features of the sentence, and a fused representation of each word is calculated through bidirectional propagation. Pooling operations are then performed to obtain the global feature vector of the sentence. Simultaneously, TextCNN is used to perform convolution and pooling operations on the word vector matrix to extract the local feature matrix of each word. The global feature vector and the local feature matrix are then concatenated, and the similarity between words is calculated to construct a probability transition matrix. Finally, based on this probability transition matrix, the TextRank algorithm is used to iteratively calculate word importance scores, selecting the top-ranked words as keywords. This method can more accurately extract sentence keywords, improving the accuracy and semantic relevance of keyword extraction, and has broad application prospects and good results.

[0204] Example 6

[0205] The keyword extraction apparatus of this application embodiment will now be described with reference to the accompanying drawings. For the sake of brevity, appropriate omissions will be made in the following description of the apparatus; relevant content can be referred to in the relevant description of the method above, and will not be repeated.

[0206] Figure 3 This is a schematic diagram of a keyword extraction device provided in an embodiment of this disclosure.

[0207] like Figure 3 As shown, the device 1000 includes the following units.

[0208] The mapping unit 1001 is used to perform vector mapping processing on the text data to be processed according to the pre-trained word vector model to obtain word vectors. The text data to be processed includes sentences, which are composed of words.

[0209] Extraction unit 1002 is used to input word vectors into a bidirectional long short-term memory network for global feature extraction to obtain global features, which are used to represent the global features corresponding to the sentence.

[0210] The extraction unit 1002 is also used to input word vectors into a text convolutional neural network for local feature extraction to obtain local features, which are used to represent the local features corresponding to words;

[0211] The calculation unit 1003 is used to construct a probability transition matrix based on the global features of the sentence and the local features of the words. The probability transition matrix is ​​used to calculate the importance score of the corresponding words.

[0212] The filtering unit 1004 is used to filter keywords from words based on their importance scores, where the importance scores of the keywords meet the preset score conditions.

[0213] In some embodiments, the extraction unit 1002 is further configured to input word vectors into a bidirectional long short-term memory network for global feature extraction to obtain global features, including:

[0214] Using the forward propagation algorithm, the forward hidden state of the current time step corresponding to the word vector of the current time step is calculated based on the forward hidden state of the previous time step corresponding to the word vector of the current time step.

[0215] Using the backpropagation algorithm, the reverse hidden state of the current time step corresponding to the word vector at the current time step is calculated based on the reverse hidden state of the next time step corresponding to the word vector at the current time step.

[0216] The global features are calculated based on the forward hidden state and the reverse hidden state at the current time step.

[0217] In some embodiments, the extraction unit 1002 is further configured to combine the forward hidden state of the current time step and the reverse hidden state of the current time step to obtain global features, including:

[0218] The forward hidden state and the reverse hidden state of the current time step are concatenated to obtain the target hidden state of the current time step.

[0219] Perform average pooling on the hidden state of the target at the current time step to obtain global features.

[0220] In some embodiments, the extraction unit 1002 is further configured to input word vectors into a text convolutional neural network for local feature extraction to obtain local features, including:

[0221] Perform a convolution operation on the word vectors to obtain the feature maps corresponding to the word vectors;

[0222] Max pooling is performed on the feature map to obtain local features.

[0223] In some embodiments, the computing unit 1003 is further configured to construct a probability transition matrix based on global features and local features, including:

[0224] The global and local features are concatenated to obtain the target features;

[0225] Similarity is calculated based on target features to obtain a similarity coefficient, which is used to represent the relevance between words.

[0226] The similarity coefficients are normalized to obtain the probability transition matrix.

[0227] In some embodiments, the filtering unit 1004 is further configured to filter keywords from the words based on importance scores, including:

[0228] Using a text ranking algorithm, the importance score corresponding to each word is calculated iteratively based on the probability transition matrix;

[0229] The words are sorted according to their importance scores to obtain a word sorting sequence;

[0230] Keywords are selected from words based on their word order sequence.

[0231] It should be noted that the information interaction and execution process between the above-mentioned units are based on the same concept as the method embodiments of this application. For details on their specific functions and technical effects, please refer to the method embodiments section, which will not be repeated here.

[0232] Example 6

[0233] Based on the above embodiments, this embodiment provides a computer device 3000, including a memory 3200, a processor 3100, and a computer program 3210 stored in the memory. The processor 3100 executes the computer program 3210 to implement the steps of the method described in the above embodiments.

[0234] In some embodiments of this example, a computer-readable storage medium is provided, on which a computer program is stored, characterized in that the computer program, when executed by a processor, implements the steps of the method described in the above embodiments.

[0235] In some embodiments of this example, a computer program product is provided, including a computer program / instructions, characterized in that the computer program, when executed by a processor, implements the steps of the method described in the above embodiments.

[0236] The processor 3100 may include, but is not limited to, one or more processors or microprocessors. Each processor may be implemented as an Application Specific Integrated Circuit (ASIC), a Digital Signal Processor (DSP), a Digital Signal Processing Device (DSPD), a Programmable Logic Device (PLD), a Field Programmable Gate Array (FPGA), a controller, a microcontroller, a microprocessor, or other electronic components, for performing the methods described in the above embodiments.

[0237] Computer-readable storage media can be implemented by any type of volatile or non-volatile storage device or a combination thereof. Computer-readable storage media may include, but are not limited to, random access memory (RAM), read-only memory (ROM), flash memory, EPROM memory, EEPROM memory, registers, and computer storage media (e.g., hard disks, floppy disks, solid-state drives, removable disks, CD-ROMs, DVD-ROMs, Blu-ray discs, etc.).

[0238] Computer-readable storage media may also store at least one computer-executable program / instruction, such as computer-readable instructions. Computer-readable storage media include, but are not limited to, volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and / or cache memory. Computer-readable storage media may include, for example, read-only memory (ROM), hard disk, flash memory, etc. For example, a non-transitory computer-readable storage medium may be connected to a computing device such as a computer, and then, when the computing device executes the computer-readable instructions stored on the computer-readable storage medium, the various methods described above can be performed.

[0239] In addition, the computer device 3000 may also include (but is not limited to) a data bus, an input / output (I / O) bus, a display, and input / output devices (e.g., keyboard, mouse, speakers, etc.).

[0240] The processor 3100 can communicate with external devices via wired or wireless networks through the I / O bus.

[0241] In one embodiment, the at least one computer-executable instruction may also be compiled into or comprise a software product / computer program product, wherein one or more computer-executable instructions are executed by a processor to perform the steps of the various functions and / or methods in the embodiments described herein.

[0242] In the embodiments provided in this disclosure, it should be understood that the disclosed apparatus and methods can also be implemented in other ways. The apparatus embodiments described above are merely illustrative; for example, the flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of apparatus, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram and / or flowchart, and combinations of blocks in block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0243] It should be noted that, in this disclosure, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element limited by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0244] While the embodiments disclosed herein are as described above, the foregoing content is merely for the purpose of facilitating understanding of this disclosure and is not intended to limit this disclosure. Any person skilled in the art to which this disclosure pertains may make any modifications and changes in form and detail of the implementation without departing from the spirit and scope of this disclosure; however, the scope of patent protection of this disclosure shall still be determined by the scope defined in the appended claims.

Claims

1. A keyword extraction method, characterized in that, include: The text data to be processed is input into a pre-trained word vector model for vector mapping to obtain word vectors. The text data to be processed includes sentences, and the sentences are composed of words. The word vectors are input into a bidirectional long short-term memory network for global feature extraction to obtain global features, which are used to represent the global features corresponding to the statement. The word vectors are input into a text convolutional neural network for local feature extraction to obtain local features, which are used to represent the local features corresponding to the words. A probability transition matrix is ​​constructed based on the global features and the local features, and the probability transition matrix is ​​used to calculate the importance score corresponding to the word. Keywords are selected from the words based on their importance scores, and the importance scores of the keywords meet preset score conditions.

2. The method according to claim 1, characterized in that, The word vectors are input into a bidirectional long short-term memory network for global feature extraction to obtain global features, including: Using the forward propagation algorithm, the forward hidden state of the current time step corresponding to the word vector of the current time step is calculated based on the forward hidden state of the previous time step corresponding to the word vector of the current time step. Using the backpropagation algorithm, the reverse hidden state of the current time step corresponding to the word vector at the current time step is calculated based on the reverse hidden state of the next time step corresponding to the word vector at the current time step. The global feature is calculated based on the forward hidden state and the reverse hidden state at the current time step.

3. The method according to claim 2, characterized in that, The calculation of the global features based on the forward hidden state and the reverse hidden state at the current time step includes: The forward hidden state and the reverse hidden state of the current time step are concatenated to obtain the target hidden state of the current time step. The global features are obtained by performing average pooling on the target hidden state at the current time step.

4. The method according to any one of claims 1-3, characterized in that, The step of inputting the word vectors into a text convolutional neural network for local feature extraction to obtain local features includes: Perform a convolution operation on the word vectors to obtain the feature maps corresponding to the word vectors; The local features are obtained by performing max pooling on the feature map.

5. The method according to any one of claims 1-3, characterized in that, The step of constructing the probability transition matrix based on the global features and the local features includes: The global features and the local features are concatenated to obtain the target features; Similarity is calculated based on the target features to obtain a similarity coefficient, which is used to represent the correlation between the words. The similarity coefficients are normalized to obtain the probability transition matrix.

6. The method according to any one of claims 1-3, characterized in that, The process of selecting keywords from the words based on importance scores includes: Using a text ranking algorithm, the importance score corresponding to the word is iteratively calculated based on the probability transition matrix. The words are sorted according to their importance scores to obtain a word sorting sequence; Keywords are selected from the words based on the word ranking sequence.

7. A keyword extraction device, characterized in that, include: The mapping unit is used to perform vector mapping processing on the text data to be processed according to the pre-trained word vector model to obtain word vectors. The text data to be processed includes sentences, and the sentences are composed of words. The extraction unit is used to input the word vectors into a bidirectional long short-term memory network for global feature extraction to obtain global features, which are used to represent the global features corresponding to the statement. The extraction unit is further configured to input the word vector into a text convolutional neural network for local feature extraction to obtain local features, wherein the local features are used to represent the local features corresponding to the word. The calculation unit is used to construct a probability transition matrix based on the global features of the statement and the local features of the word, and the probability transition matrix is ​​used to calculate the importance score corresponding to the word. The filtering unit is used to filter keywords from the words based on their importance scores, wherein the importance scores of the keywords meet preset score conditions.

8. A computer device, comprising a memory, a processor, and a computer program stored in the memory, characterized in that, The processor executes the computer program to implement the steps of the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the steps of the method according to any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When executed by a processor, the computer program implements the steps of the method according to any one of claims 1 to 6.

Images