Keyword extraction data set making method based on decision consultation service

Abstract

The invention discloses a keyword extraction data set making method based on decision consultation service. The method comprises the following steps: S1, capturing webpage text data by using a Selenium library in a Python language and performing data cleaning; s2, connecting a MySQL database, and storing the captured data according to dimensions such as time, title, content and the like so as to ensure structured management of the data; s3, performing Chinese word segmentation and stop word removal on the data of the title and the text content; s4, under the framework of the Transform model, introducing title information and integrating the calculation of an attention mechanism, and guiding the model to generate keywords through the combination of a title vector and attention; and S6, measuring the semantic similarity between the keywords and the original text through cosine similarity so as to screen out more accurate keywords. According to the method, the functions of data capture and data set production are realized, and meanwhile, the accuracy of keyword extraction is improved by integrating a title attention mechanism and cosine similarity.

Description

Technical Field

[0001] The present invention relates to the technical field of natural language processing, and in particular to a method for producing a data set by extracting keywords through crawling data. Background Art

[0002] Keywords are composed of a single word or multiple words, and are used to summarize relevant information about the topic of a document. In the era of big data, with the rapid development of technologies such as the Internet, cloud computing, and artificial intelligence, text information has exploded exponentially. The way people obtain information has long changed from traditional paper books and newspapers to the Internet as the main cultural medium. How to effectively obtain subject content information from massive text data has become increasingly important. Among them, how to extract keywords that reflect the topic of a document from a text document has always been a key basic problem and research hotspot that needs to be solved in the field of natural language processing. Its research results can be widely used in specific application fields such as document retrieval, text summarization, text classification, topic detection, and question-answering systems.

[0003] Traditional keyword extraction methods mostly rely on statistical features, such as genetic algorithms and support vector machines. Although these methods are effective to a certain extent, they often fail to fully consider the contextual information of the text and its potential semantic relationships. With the development of deep learning technology, models based on neural networks, especially the Transformer architecture, have gradually become a research hotspot in the field of keyword extraction. With its powerful sequence modeling capabilities and self-attention mechanism, the Transformer model can better capture long-distance dependencies in the text, thereby improving the effect of keyword extraction.

[0004] Decision-making consulting services refer to providing key information and suggestions to decision-makers to assist them in making decisions. In the context of an increasingly information-based and complex society, the important position of decision-making consulting agencies in the decision-making process of various departments and fields has become increasingly prominent, and the accuracy of keyword extraction in the field of decision-making consulting services is particularly important. However, there is currently a significant problem in the field of decision support: the lack of high-quality keyword extraction datasets. This lack limits the ability of researchers and practitioners to develop and optimize keyword extraction models. Although there are some existing datasets, they usually cannot meet the needs of specific decision-making consulting scenarios, lack sufficient diversity and richness, and cannot effectively support efficient keyword extraction and application. Summary of the invention

[0005] The present invention discloses a method for preparing a keyword extraction data set based on decision-making consulting services. The method integrates advanced text crawling technology, database management, text preprocessing and deep learning models to construct an efficient keyword extraction data set preparation method. The method aims to provide a data set preparation system that not only realizes efficient crawling and management of data, but also significantly improves the accuracy of keyword extraction by introducing a title attention mechanism and the calculation of cosine similarity.

[0006] To solve the above problems, this application adopts the following technical solutions:

[0007] A method for preparing a keyword extraction data set based on decision-making consulting services, the method comprising the following steps:

[0008] S1. Crawl website text data and perform data cleaning:

[0009] Click on the web link through the Selenium library to capture text data; unify the time format in the captured data, and remove the captured garbled symbols and link addresses.

[0010] S2, structured storage:

[0011] The cleaned data in S1 is stored in MySQL according to time, text content, and title, and the keyword dimension is added.

[0012] S3. Text preprocessing:

[0013] Perform Chinese word segmentation on the title and text content and remove stop words.

[0014] S4. Generate keyword sequence:

[0015] Based on the Transformer model, the title information is added into the multi-head attention mechanism calculation. The title vector is combined with the attention of the text content and the sequence code word embedding is added to guide the model to generate keywords.

[0016] S5. Measure the semantic similarity between keywords and text:

[0017] The keywords generated by S4 and the text content in S2 are converted into vector representations, and the cosine similarity between each keyword and the text content is calculated respectively.

[0018] S6. Determine keywords:

[0019] A cosine similarity threshold is set, and the results calculated in S5 are sorted in descending order. The first 50% are selected as the final keywords of the document and stored in the corresponding position of MySQL.

[0020] Compared with the prior art, the beneficial effects achieved by the present invention are:

[0021] Improve keyword extraction accuracy: Based on the Transformer model, the title information and text content are integrated into the attention calculation, which can more accurately identify and extract keywords related to the document topic, thereby improving the efficiency of information retrieval and analysis.

[0022] Enhance data management capabilities: Through structured storage, the captured data is ensured to be well queryable and manageable, making subsequent data analysis and processing more efficient and systematic.

[0023] Improve processing speed: This method significantly reduces the need for human intervention through automated data capture and processing processes, improves the overall processing speed, and makes real-time decision support possible.

[0024] Adaptability to diverse needs: Due to the flexibility and scalability of this method, it can be adjusted to meet the decision-making consulting needs in different fields, is applicable to a variety of text data types, and meets the information acquisition needs of different decision makers.

[0025] Support scientific decision-making: Through precise keyword extraction, decision makers can quickly obtain key information, support a more scientific and efficient decision-making process, and make more informed choices in a complex information environment.

[0026] Promote research progress: This method provides a standardized dataset production process, provides a basis for academic research and industrial applications, and helps promote the further development and application of keyword extraction technology in the field of decision support. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0028] Figure 1 It is a flowchart of a method for producing a keyword extraction data set based on decision-making consulting services.

[0029] Figure 2 This is the Transformer model architecture diagram.

[0030] Figure 3 This is the improved keyword generation network architecture diagram based on Transformer. DETAILED DESCRIPTION

[0031] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0032] like Figure 1 As shown, for step S1, Selenium is needed to capture data. Selenium is an automated testing tool for Web application testing that supports multiple browsers. It can simulate the user's behavior of interacting with the Web application in a Web browser, such as clicking, inputting, selecting, scrolling, etc., and can also verify and report the test results. The step S1 specifically includes:

[0033] S11. Use the Selenium library in Python language to crawl web page text data.

[0034] S12. Clean the captured text data, remove link addresses, remove garbled symbols, and unify the time format; convert xxxx-xx-xx-dd, xx-xx-dd-xxxx, xx-xx-xx-xxxx, etc. into the standard format %y-%m-%d.

[0035] For structured storage in step S2, MySQL is needed. MySQL is a relational database management system that uses structured query language (SQL) to store, retrieve and manage data. Step S2 specifically includes:

[0036] S21. Connect to MySQL.

[0037] S22. The captured data is stored according to dimensions such as time, title, and content to ensure structured management of the data.

[0038] S23. Add keyword dimension.

[0039] For the text preprocessing in step S3, the jieba library is needed. Jieba is a powerful Python Chinese word segmentation library and the most popular Chinese word segmentation component. Jieba supports simplified or traditional Chinese word segmentation, custom dictionaries, part-of-speech tagging, word position query and other functions.

[0040] The corresponding step S3 specifically includes:

[0041] S31. Chinese word segmentation: Use the jieba library to segment the captured text content. During the word segmentation, non-word characters (non-letters, numbers or underscores) in the sentence need to be removed because these punctuation marks will also affect the final keyword prediction.

[0042] S32, remove stop words: stop words refer to words that frequently appear in text but lack actual meaning. After removing these words, the semantic understanding of the whole sentence will not be affected. In Chinese, such as "I", "just", "ah" and function words, pronouns or verbs, nouns lacking specific meanings, etc., are not helpful for text analysis. Therefore, they will usually be filtered out as stop words, and the present invention adopts the stop word list of Harbin Institute of Technology to remove stop words to text.

[0043] Regarding the generation of the keyword sequence described in step S4, the step S4 specifically includes:

[0044] S41, using the Transformer model framework, the Transformer model is implemented based on the encoder-decoder framework, and its structure is as follows Figure 2 shown.

[0045] Figure 2 The left and right sides of the figure are the basic encoder unit and decoder unit, which together form a Transformer layer. Figure 2 Introduction to each part:

[0046] Input: The model is trained based on one-way, many-to-many information. It does not require the length of input and output information to be equal. When the lengths of the two are equal, the missing part is filled with a 0 vector.

[0047] Output (right swipe): In general tasks, the purpose of model training is to predict the probability of the next word. The output result is shifted one position to the right relative to the input sequence, that is, a right swipe.

[0048] N layers: The Transformer model has a depth of 6 layers. At the end of each layer, the encoder sends the hidden state to the encoder of the next layer. The same is true for the decoder.

[0049] Multi-head attention layer: Figure 2 There are three parallel arrows under each multi-head attention layer in the figure, representing Value (V), Key (K) and Query (Q) from left to right. This technology can be expressed as Y = MultiHead (Q, K, V). The encoder transforms the hidden state linearly at each layer to differentiate the Key and Value and transmit them to the multi-head attention layer of the decoder.

[0050] Word embedding conversion: Use pre-trained word vectors to represent text content with a dimension of 512.

[0051] Position-based word embeddings: Generate sinusoids using the relative position of words in the text.

[0052] Fully connected feedforward neural network: The word embedding at each position is transformed individually to unify the dimension of its context.

[0053] Addition and normalization layer: Add the input and output of the previous layer to form a residual structure, and normalize the result of the residual structure.

[0054] Fully connected output layer: Outputs the probability distribution of the model results, and the output dimension is the vocabulary size of the predicted target size.

[0055] The Transformer model adds position-based word embedding, that is, the position information of the word is added. The input of the encoder is "input word embedding + position-based word embedding of the corresponding position". The actual encoder is composed of multiple encoder units. The specific implementation steps are as follows:

[0056] S411. Add the converted word embedding and the position-based word embedding as the input of the encoder.

[0057] S412: Input the result of S411 into the multi-head attention layer to learn the correlation between words in the input.

[0058] S413, input the result of S412 into the "addition and normalization layer". The "addition" means that different outputs are added using residual connection; the normalization uses layer-based normalization.

[0059] S414, input the result of S413 into the fully connected neural network layer.

[0060] S415, input the result of S414 into the "addition and normalization layer" again to obtain the final output.

[0061] The decoder process steps are as follows:

[0062] S416. Pass the word decoded by the last decoder through "output word embedding + position-based word embedding of the corresponding position", and then input the result into the "mask-based multi-head attention layer".

[0063] S417: Input the result in S416 into the "addition and normalization layer", and use the output obtained as the input of this decoder. At the same time, perform a residual connection between the input of the decoder and the output of the "mask-based multi-head attention layer".

[0064] S418: All input information obtained by the encoder and the input information obtained by the first half of the decoder are sent to the subsequent "mask-based multi-head attention layer". This feature that combines input information and previous output information is more conducive to decoding.

[0065] S419, input the result of S418 into "add and normalize to one layer".

[0066] S4110, input the result of S419 into the fully connected neural network layer.

[0067] S4111. Input the result of S4110 into the "addition and normalization layer" again to obtain the final output.

[0068] S42, input the text content into the Transformer encoder for encoding processing, and incorporate the title information into the calculation of the attention mechanism, using Figure 3 The improved Transformer-based keyword generation method. The attention mechanism is a mechanism in which a neural network assigns different weights to each element in a sequence. Self-attention uses the entire sequence to calculate the weighted average of each embedding value, that is, given a sequence of word embeddings x1, x2, ..., x n , self-note

[0069] as follows:

[0070] S421. Project each word embedding into three vectors, called V, K, and Q respectively.

[0071] S422. Calculate the attention score, that is, calculate the dot product of Q and K. In a sequence of n input words, it corresponds to an n*n attention score matrix.

[0072] S423, calculate the attention weights, that is, multiply the attention scores by a scaling factor to normalize their variance, and then normalize them through softmax to ensure that the sum of all column values ​​is 1. The result is an n*n matrix that contains all the attention weights w ji .

[0073] S424, update word embeddings, after calculating the attention weights, add them to the value vectors v1,...v n Multiply.

[0074] In summary, the formula for scaling dot product attention is: Among them, QK Τ is the dot product between the query (Q) and the key vector (K), d k is the dimension of the key vector. The dot product is divided by This prevents the values ​​from becoming very large, pushing the values ​​of the softmax layer into areas where the gradients may be very small.

[0075] S43. Through the title vector and attention guidance, the final embedding with title attention is obtained.

[0076] S431, such as Figure 3 As shown in the figure, based on the Transformer model, the text-preprocessed title and document are fed into the Transformer encoder for encoding, and the title information is integrated into the attention calculation. The formula is as follows:

[0077] H = Transformer(x1,x2,...x n )={h1,h2,...h n}

[0078] H T = Transformer(t1,t2,...t n )={h t 1,h t 2,...h t n}

[0079] Among them, H T is the hidden state code, h i is the hidden state of each word in the text content, h t i is the hidden state of each word in the title.

[0080] S432. The semantic similarity between word embeddings is measured by introducing title information. The calculation method is consistent with the calculation of the cross-attention mechanism. The idea of ​​cross-attention is to enable one sequence to "pay attention" to another sequence. It is very similar to the self-attention mechanism, but the cross-attention focuses on another sequence instead of itself. The V matrix is ​​regarded as the similarity matrix for calculating the title and text content, and H Τ The matrix is ​​regarded as the K matrix in the cross attention, and the H matrix is ​​regarded as the Q matrix in the cross attention. The attention score is calculated, that is, the similarity between Q and each Key is calculated, that is, the dot product of Q and K is directly calculated. The formula is as follows:

[0081]

[0082]

[0083] in: is the similarity between the embedding of the i-th word in the title and the embedding of the j-th word in the summary; α ijis the attention weight between the i-th word in the title and the j-th word in the text; the similarity of the j-th word in the text is expressed as: A = {α1, α2, ..., α m}, α i is the attention weight between the i-th word in the title and the text content; α i ={α ij'} j'＝1,2,...n The hidden state of the text content is added to the title similarity matrix, and the title similarity matrix is ​​calculated as AH Τ .

[0084] S433, the encoder undergoes residual connection and normalization operations, where the normalization layer normalizes the embedded distribution to output between 0 and 1. The formula is: Output = LayerNorm (AH Τ +H), where AH Τ is the title similarity matrix, and H is the hidden state encoding matrix of the text content.

[0085] S434. Through the title vector and attention guidance, the final embedding with title attention is obtained, and then sent to the Transformer decoder to generate a keyword sequence.

[0086] A set of sequence code embeddings are added as additional decoder inputs. Each embedding represents the order of each phrase in the keyword sequence, which can indicate the difference between the generated words, thereby reducing the repetition rate of the generated keywords. After the sequence is introduced, the input calculation formula of the decoder is: Where: Words The embedding is the i-th sinusoidal position embedding, c n is the i-th sequence code embedding.

[0087] In order to measure the semantic similarity between the keyword and the text, step S5 needs to be calculated by cosine similarity. Cosine similarity is a commonly used text similarity calculation method, which can be used to calculate the relationship between the keyword and the text. By converting the keyword and the text into vectors respectively, calculating the cosine value of the angle between the two, the similarity score is obtained. The closer the cosine value is to 1, the closer the relationship between the keyword and the text is. The step S5 specifically includes:

[0088] S51. Text vectorization: Convert both keywords and text into vector representations.

[0089] S52, calculate cosine similarity: use the cosine similarity formula to calculate the similarity between the keyword vector and the text vector as follows:

[0090] A and B represent the vectors of keywords and texts respectively, and the cosθ value is between 0 and 1. The larger the value, the higher the similarity.

[0091] Step S6 is to determine the keywords, and the step S6 specifically includes:

[0092] S61. Threshold judgment: set a similarity threshold. If the cosine similarity is higher than the threshold, it means that the keyword and the text are highly correlated; if it is lower than the threshold, the correlation is low.

[0093] S62, sorting the parts with higher relevance, and selecting the top 50% as the keywords of the document.

[0094] S63. Store the keywords in the corresponding position of MySQL.

[0095] Finally, it should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art can still modify the technical solutions described in the aforementioned embodiments or replace some of the technical features therein by equivalents. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for preparing a keyword extraction data set based on decision-making consulting services, characterized in that: The following steps are involved: S1. Crawl website text data and perform data cleaning: Click on the web link through the Selenium library to capture the title and text content of the text data; unify the time format in the captured data, and remove the captured garbled symbols and link addresses. S2, structured storage: The cleaned data in S1 is stored in MySQL according to time, text content, and title, and the keyword dimension is added. S3. Text preprocessing: Perform Chinese word segmentation on the title and text content and remove stop words. S4. Generate keyword sequence: Based on the Transformer model, the title information is added into the multi-head attention mechanism calculation. The title vector is combined with the attention of the text content and the sequence code word embedding is added to guide the model to generate keywords. S5. Measure the semantic similarity between keywords and original text: The keywords generated by S4 and the text content in S2 are converted into vector representations, and the cosine similarity between each keyword and the text content is calculated respectively. S6. Determine keywords: A cosine similarity threshold is set, and the results calculated in S5 are sorted in descending order. The first 50% are selected as the final keywords of the document and stored in the corresponding position of MySQL.

2. The method for preparing a keyword extraction data set based on decision-making consulting services according to claim 1, characterized in that: The step S1 specifically includes: S11. Use the Selenium library in Python language to crawl web page text data; S12. Clean the captured text data, remove link addresses, remove garbled symbols, and unify the time format.

3. The method for preparing a keyword extraction data set based on decision-making consulting services according to claim 1, characterized in that: The step S2 specifically includes: S21. Connect to MySQL. S22, storing the captured data by time, title, content and other dimensions to ensure structured management of the data; S23. Add keyword dimension.

4. The method for preparing a keyword extraction data set based on decision-making consulting services according to claim 1, characterized in that: The step S3 specifically includes: S31, Chinese word segmentation: Use the jieba library to segment the captured text content; S32. Remove stop words: Remove stop words such as function words and pronouns in Chinese text that are not helpful for text analysis.

5. The method for preparing a keyword extraction data set based on decision-making consulting services according to claim 1, characterized in that: The step S4 specifically includes: S41, build the framework of Transformer model; S411, the encoder converts the text into a hidden state, the process is: H = Encoder (X), where X represents the input Chinese text sequence; H represents the hidden state output by the encoder, which is the context representation of the input text; the core calculation is: H' = MSA (LN (H)) + H, H = MLP (LN (H')) + H', where MSA is a multi-head self-attention mechanism, which is used to calculate the correlation between each word, and MLP is used to perform nonlinear transformation on the features; S412, the decoder generates Chinese keywords according to the hidden state H output by the encoder. The operation of the decoder is as follows: Y t '=MSA(LN(Y t-1 ))+Y t-1 Y t ”=ENCAttention(LN(Y' t ),H)+Y t ' AND t =MLP(LN(Y t ”))+Y t ” Among them, Y t-1 It is the intermediate representation generated by the decoder at step t-1. ENCAttention is a masked multi-head attention used to extract the contextual information required for keyword generation from the encoder output H; S413, keyword generation: through the output Y of the decoder t , the probability distribution is obtained through the softmax layer, the formula is as follows: P(Y t |X)=softmax(W0·Y t ) Among them, W0 is the weight matrix of the output layer, P(Y t |X) is the probability distribution of generating a word at time t given the input text X; S42. Improve the Transformer model, input the title information and text content into the Transformer encoder for encoding, and incorporate the title information into the calculation of the attention mechanism. The formula is as follows: H=Transformer(x1,x2,...x n )={h1,h2,...h n } H T =Transformer(t1,t2,...t n )={h t 1,h t 2,...h t n } Among them, H T is the hidden state code, h i is the hidden state of each word in the text content, h t i is the hidden state of each word in the title; S43. Through the title vector and attention guidance, the final embedding with title attention is obtained. The formula is as follows: in: is the similarity between the embedding of the i-th word in the title and the embedding of the j-th word in the summary; α ij is the attention weight between the i-th word in the title and the j-th word in the text; the similarity of the j-th word in the text is expressed as: A = {α1, α2, ..., α m }, α i is the attention weight between the i-th word in the title and the text content; α i ={α ij' } j'＝1,2,...n Add the hidden state of the text content to the title similarity matrix; S44. Generate a keyword sequence through the Transformer decoder.

6. The method for preparing a keyword extraction data set based on decision-making consulting services according to claim 1, characterized in that: The step S5 specifically includes: S51, vectorizing the generated keywords and the original text; S52. Use cosine similarity to measure the semantic similarity between each keyword and the original text. The formula is as follows: A and B represent the vectors of keywords and text content respectively, and the cosθ value is between 0 and 1. The larger the value, the higher the similarity.

7. The method for preparing a keyword extraction data set based on decision-making consulting services according to claim 1, characterized in that: The step S6 specifically includes: S61, setting a similarity threshold; S62, sorting the keyword similarities measured in S51 in descending order, and selecting the top 50% as the keywords of the document; S63: Store the final keyword.

Images