Keyword extraction method, system, equipment and medium
By combining the context modeling of the BERT model and the statistical characteristics of TF-IDF, combined with adaptive learning rate and L2 regularization technology, the keyword extraction process is optimized, and the semantic extraction efficiency under complex language and large-scale data in natural language processing is solved. The problem is to achieve more efficient and accurate keyword extraction.
Patent Information
- Application Number
- CN202411419344.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-12
- Publication Date
- 2024-11-08
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
In the field of natural language processing, it is difficult for the prior art to improve the accuracy and processing efficiency of semantic extraction when dealing with complex language phenomena and large-scale data.
Combining the context modeling ability of the BERT model and the statistical characteristics of TF-IDF, the candidate keywords are analyzed by pre-training the BERT model, and combining adaptive learning rate and L2 regularization technology, the keyword extraction process is optimized to ensure the extracted Keywords are both accurate and efficient.
It significantly improves the accuracy and processing efficiency of keyword extraction, can more accurately identify keywords in complex language environments, overcomes the limitations of a single technical method, and provides a more comprehensive and effective solution.
Smart Images

Figure CN118917310A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of natural language processing, and in particular to a keyword extraction method, system, device and medium. Background Art
[0002] In the field of natural language processing (NLP), keyword extraction is the process of automatically identifying the most representative and important words or phrases from a text. Keyword extraction is of great significance in many applications such as information retrieval, content analysis, recommendation systems, and data mining. Effective keyword extraction methods can significantly improve the quality and efficiency of text analysis.
[0003] Traditional keyword extraction methods mainly include statistical-based techniques and rule-based techniques. Statistical methods such as TF-IDF (Term Frequency-Inverse Document Frequency) evaluate the importance of words by calculating word frequency and inverse document frequency. Rule-based methods rely on specific rule sets to filter keywords. These methods usually work well, but may not be smart enough when faced with complex contexts and implicit meanings.
[0004] Therefore, how to improve the accuracy and processing efficiency of semantic extraction when dealing with complex language phenomena and large-scale data is a technical problem that needs to be solved urgently. Summary of the invention
[0005] The technical task of the present invention is to provide a keyword extraction method, system, device and medium to solve the problem of how to improve the accuracy and processing efficiency of semantic extraction when processing complex language phenomena and large-scale data.
[0006] The technical task of the present invention is achieved in the following manner, and the method is specifically as follows: Collect text data to be processed; Use the standard text preprocessing tool NLTK to perform noise removal and word segmentation preprocessing operations on the text data to be processed, and obtain the preprocessed text data; The preprocessed text data is preliminarily analyzed using the IF-TDF algorithm to calculate the TF-IDF value of each word in the text data, and TfidfVectorizer is used to calculate the TF-IDF value and inverse frequency. Filter candidate keywords from text data based on TF-IDF values: determine the candidate list of keywords using the set threshold and sorting mechanism; the threshold is set by the user-defined text similarity value, so as to filter keywords; the sorting mechanism is to sort keywords according to the text similarity value; Perform semantic analysis on candidate keywords through the pre-trained BERT model to obtain semantic analysis results; Post-process the semantic analysis results based on the pre-trained BERT model to optimize and select the final keywords; The final keywords are verified and adjusted, and the pre-trained BERT model parameters and IF-TDF algorithm settings are adjusted according to the verification results to optimize the keyword extraction effect.
[0007] As a preference, semantic analysis is performed on candidate keywords through a pre-trained BERT model, and the semantic analysis results are obtained as follows: Text embedding: Use Hugging Face's Transformers library to load the pre-trained BERT model and convert the text data into context-sensitive embedding vectors through the pre-trained BERT model; Context modeling: Context modeling is performed on each candidate keyword through the pre-trained BERT model; Optimizer: By customizing The optimizer accelerates the convergence of the pre-trained BERT model and uses L2 regularization to suppress overfitting of the pre-trained BERT model.
[0008] Preferably, the context modeling is as follows: In the encoding process of the pre-trained BERT model mentioned above, the temporal information, semantic segmentation information and text information of the text are added together to generate the final vector, and the semantic information of the vocabulary is captured according to the final vector; among them, the pre-trained BERT model will consider the context of the vocabulary in the entire text, so as to conduct in-depth analysis of the candidate keywords.
[0009] More preferably, The optimizer is an adaptive learning rate method that combines the advantages of AdaGrad and RMSProp. The Adam optimizer dynamically adjusts the learning rate of each parameter, automatically adjusting the learning rate based on the first-order moment estimate and second-order moment estimate of the gradient, so as to converge quickly in the early stage of training and avoid oscillation or divergence in the later stage. The first-order moment estimation formula of the optimizer gradient is: ; The second-order moment estimation formula of the optimizer gradient is: ; The bias correction formula of the optimizer is: ; The update parameter formula of the optimizer is: ; in: represents the gradient; and is a hyperparameter; represents the basic learning rate; is a small constant to prevent division by zero error; t represents the number of iterations; represents the hyperparameters of the first-order matrix estimation in the t-th iteration; represents the hyperparameters of the second-order matrix estimation in the t-th iteration; exist L2 regularization is introduced on the basis of the optimizer; L2 regularization penalizes weight values that exceed the set threshold by adding a term proportional to the parameter size to the update rule, helping the pre-trained BERT model to select the corresponding parameter value and improve the generalization ability of the model; The update formula after adding L2 regularization is as follows: ; in, l represents the regularization strength coefficient.
[0010] As a preference, the semantic analysis results based on the pre-trained BERT model are post-processed to optimize and select the final keywords as follows: Relevance evaluation: By calculating the semantic relevance between candidate keywords and text, the most representative and informative keywords are further screened out; Sorting and filtering: Sort candidate keywords according to semantic relevance scores, and apply set filtering criteria of frequency of occurrence or keyword importance to determine the final set of keywords.
[0011] A keyword extraction system, the system comprising: A collection module, used for collecting text data to be processed; The preprocessing module is used to perform noise removal and word segmentation preprocessing operations on the text data to be processed through the standard text preprocessing tool NLTK to obtain the preprocessed text data; The preliminary keyword screening module is used to perform preliminary analysis on the preprocessed text data through the IF-TDF algorithm, calculate the TF-IDF value of each word in the text data, and use TfidfVectorizer to calculate the TF-IDF value and inverse frequency; The candidate keyword screening module is used to screen candidate keywords from text data based on TF-IDF values, specifically: using the set threshold and sorting mechanism to determine the candidate list of keywords; wherein the threshold is set by the user-defined text similarity value, so as to screen the user keywords; the sorting mechanism is to sort the keywords according to the text similarity value; The semantic analysis module is used to perform semantic analysis on candidate keywords through the pre-trained BERT model to obtain semantic analysis results; Keyword optimization and selection module, which is used to post-process the semantic analysis results based on the pre-trained BERT model, optimize and select the final keywords; The result verification and adjustment module is used to verify and adjust the final keywords, and adjust the pre-trained BERT model parameters and IF-TDF algorithm settings according to the verification results, thereby optimizing the keyword extraction effect.
[0012] Preferably, the semantic analysis module comprises: The text embedding submodule is used to load the pre-trained BERT model using Hugging Face's Transformers library and convert text data into context-sensitive embedding vectors through the pre-trained BERT model; The context modeling submodule is used to perform context modeling on each candidate keyword using the pre-trained BERT model; Optimizer, used to customize The optimizer accelerates the convergence of the pre-trained BERT model and uses L2 regularization to suppress overfitting of the pre-trained BERT model. The optimizer is an adaptive learning rate method that combines the advantages of AdaGrad and RMSProp. The Adam optimizer dynamically adjusts the learning rate of each parameter, automatically adjusting the learning rate based on the first-order moment estimate and second-order moment estimate of the gradient, so as to converge quickly in the early stage of training and avoid oscillation or divergence in the later stage. The first-order moment estimation formula of the optimizer gradient is: ; The second-order moment estimation formula of the optimizer gradient is: ; The bias correction formula of the optimizer is: ; The update parameter formula of the optimizer is: ; in: represents the gradient; and is a hyperparameter; represents the basic learning rate; is a small constant to prevent division by zero error; t represents the number of iterations; represents the hyperparameters of the first-order matrix estimation in the t-th iteration; represents the hyperparameters of the second-order matrix estimation in the t-th iteration; exist L2 regularization is introduced on the basis of the optimizer; L2 regularization penalizes weight values that exceed the set threshold by adding a term proportional to the parameter size to the update rule, helping the pre-trained BERT model to select the corresponding parameter value and improve the generalization ability of the model; The update formula after adding L2 regularization is as follows: ; in, l represents the regularization strength coefficient.
[0013] Preferably, the keyword optimization and selection module includes: The relevance evaluation submodule is used to further select the most representative and informative keywords by calculating the semantic relevance between candidate keywords and text; The sorting and filtering submodule is used to sort the candidate keywords according to the semantic relevance scores and apply the set filtering criteria of the frequency of occurrence or the importance of the keywords to determine the final set of keywords.
[0014] An electronic device comprising: a memory and at least one processor; Wherein, the memory stores a computer program; The at least one processor executes the computer program stored in the memory, so that the at least one processor performs the keyword extraction method as described above.
[0015] A computer-readable storage medium stores a computer program, and the computer program can be executed by a processor to implement the keyword extraction method as described above.
[0016] The keyword extraction method, system, device and medium of the present invention have the following advantages: (I) The present invention combines the context modeling capability of BERT and the statistical characteristics of TF-IDF, thereby showing excellent performance in the keyword extraction task, which is reflected in the following aspects: ① BERT's excellent context modeling capabilities: BERT (Bidirectional Encoder Representations from Transformers) can effectively capture the contextual information of words through its bidirectional encoder, thereby understanding the deep structure of semantics; BERT has significant advantages when dealing with complex language phenomena, such as synonyms, polysemy, and contextual dependencies. Its bidirectional modeling capabilities enable the model to understand the specific meaning of words in different contexts, so it can more accurately identify and extract keywords in complex language environments; ❶ Synonym processing: BERT can identify synonymous relationships between different words through contextual information, thereby extracting semantically relevant keywords; ❷ Context dependency: For context-dependent words, BERT can capture the meaning of words in specific contexts through its deep semantic understanding ability, thereby improving the accuracy of keyword extraction.
[0017] ②TF-IDF (Term Frequency-Inverse Document Frequency) is a traditional keyword extraction method that evaluates the importance of words by calculating term frequency (TF) and inverse document frequency (IDF). It is known for its efficient computing power and can process large amounts of text data in a short time and quickly identify important words in the text. ❶ Efficiency: The TF-IDF method extracts keywords through simple statistical calculations, with a fast processing speed, which is suitable for the analysis of large-scale data sets; ❷Statistical accuracy: TF-IDF can effectively filter out words with high importance in the document by calculating word frequency and document frequency.
[0018] ③Combining the advantages of BERT and TF-IDF can give full play to the advantages of both: ❶ Balance between efficiency and accuracy: TF-IDF provides an efficient keyword candidate screening process, while BERT performs in-depth semantic analysis on these candidate keywords. This hierarchical processing method not only retains the computational efficiency of TF-IDF, but also uses BERT to improve the accuracy of semantic extraction; ❷ Comprehensiveness and pertinence: TF-IDF can quickly screen out potential keyword candidates, while BERT can further semantically verify and optimize these candidates to ensure that the keywords finally extracted are both comprehensive and highly relevant; In summary, by combining BERT's deep semantic understanding and TF-IDF's efficient calculation, this invention can provide excellent results in processing complex language phenomena and large-scale data, which not only improves the accuracy of keyword extraction, but also ensures the efficiency of the processing process. Compared with using either BERT or TF-IDF alone, it is superior in terms of semantic extraction accuracy and processing efficiency. (ii) The present invention combines BERT with the TF-IDF keyword extraction model to fully utilize the advantages of both to achieve more accurate and efficient keyword extraction; (III) The present invention uses TF-IDF technology to perform preliminary analysis on the text and quickly calculate the importance of words in the document, thereby screening out potential keyword candidates; the TF-IDF method can process large-scale data sets with its efficient computing power and provide an efficient vocabulary screening process; (iv) The present invention uses BERT to perform in-depth semantic understanding of these candidate keywords. BERT, through its bidirectional encoder modeling capability, can deeply capture contextual information and semantic features, handle complex language phenomena such as synonyms, polysemy, and context dependence, and thus semantically optimize the keywords initially screened, ensuring that the extracted keywords are both accurate and highly relevant. (V) By introducing an adaptive learning rate strategy and L2 regularization, the present invention not only improves the training efficiency of the Bert model, but also enhances the convergence stability of the model; combining the efficient screening ability of TF-IDF and the powerful semantic understanding ability of Bert, it can not only overcome the limitations of a single technology, but also provide a more comprehensive and effective solution in practical applications; through this hierarchical processing method, the accuracy of keyword extraction is improved while maintaining the efficiency of processing, showing excellent performance in the analysis of large-scale text data; (VI) The present invention is mainly used in data intelligence platforms to help users summarize uploaded documents and further refine keywords, thereby helping users quickly understand the content of the documents; (VII) The present invention introduces an adaptive learning rate strategy. Specifically, the Adam optimizer is adopted, which is an adaptive learning rate method that combines the advantages of AdaGrad and RMSProp. The Adam optimizer dynamically adjusts the learning rate of each parameter according to the first-order moment estimate (i.e., mean) and second-order moment estimate (i.e., variance) of the gradient to automatically adjust the learning rate, thereby achieving rapid convergence in the early stage of training and avoiding oscillation or divergence in the later stage. (VIII) The present invention improves the accuracy of keyword extraction: by combining the two methods of BERT and TFIDF, key information can be extracted from the text more accurately; the BERT model can understand the deep semantics and contextual information of the text through its deep learning ability, while TFIDF can effectively identify the frequency and importance of keywords in the text; (IX) The semantic understanding ability of the present invention is significantly improved: The bidirectional encoding characteristics of the BERT model give it a significant advantage in understanding the semantics of text, which is particularly important for extracting accurate keywords, especially when dealing with complex or ambiguous text; (10) The present invention improves text analysis and data processing efficiency: By combining BERT and TFIDF, the efficiency is greatly improved when processing large amounts of text data, especially in scenarios where key information needs to be extracted quickly and accurately; (XI) The present invention is applicable to a variety of text types and fields, not limited to specific types of text, and can be widely used in various text analysis and processing scenarios; (XII) Fusion model architecture: One of the key technical points of the present invention is to propose a hybrid model architecture that integrates BERT and TFIDF, which fully utilizes the advantages of BERT in understanding the deep semantics of text and combines the effectiveness of TF-IDF in keyword weight recognition; (XIII) Training and optimization strategy: The unique training and optimization strategy involved in the present invention enables the hybrid model to show higher accuracy and robustness on different text data sets, including but not limited to model pre-training, selection of fine-tuning parameters, and customized optimization for specific application scenarios. The present invention adds an adaptive learning rate adjustment strategy to the Bert model, which effectively accelerates the convergence of the model, thereby improving the efficiency of model keyword generation and effectively saving user time; (XIV) Keyword extraction algorithm: This invention proposes a novel keyword extraction algorithm, which combines the context embedding generated by BERT and the word frequency information calculated by TFIDF to achieve more accurate keyword identification; (XV) Adaptive weight adjustment mechanism: The invention includes an adaptive weight adjustment mechanism that can dynamically adjust the contribution ratio of BERT and TFIDF in the final keyword score according to different text content and application requirements. BRIEF DESCRIPTION OF THE DRAWINGS
[0019] The present invention is further described below in conjunction with the accompanying drawings.
[0020] Attached Figure 1 This is a flowchart of the keyword extraction method. DETAILED DESCRIPTION
[0021] The keyword extraction method, system, device and medium of the present invention are described in detail below with reference to the accompanying drawings and specific embodiments.
[0022] Embodiment 1: As attached Figure 1 As shown, this embodiment provides a keyword extraction method, which is specifically as follows: S1. Collect text data to be processed; the text data can be any form of text, such as news articles, social media posts, or scientific papers; S2. Perform noise removal and word segmentation preprocessing operations on the text data to be processed by using the standard text preprocessing tool NLTK to obtain the preprocessed text data; S3. Perform a preliminary analysis on the preprocessed text data using the IF-TDF algorithm, calculate the TF-IDF value of each word in the text data, and use TfidfVectorizer to calculate the TF-IDF value and inverse frequency. S4, screening candidate keywords from text data based on TF-IDF values: using a set threshold and sorting mechanism to determine a candidate list of keywords; wherein the threshold is set by a user-defined text similarity value, thereby screening user keywords; the sorting mechanism is to sort keywords according to the text similarity value; S5. Perform semantic analysis on candidate keywords through the pre-trained BERT model to obtain semantic analysis results; S6. Post-process the semantic analysis results based on the pre-trained BERT model to optimize and select the final keywords; S7. Verify and adjust the final keywords, and adjust the pre-trained BERT model parameters and IF-TDF algorithm settings based on the verification results to optimize the keyword extraction effect.
[0023] In step S5 of this embodiment, semantic analysis is performed on the candidate keywords by using the pre-trained BERT model to obtain the semantic analysis results as follows: S501, Text Embedding: Use Hugging Face's Transformers library to load the pre-trained BERT model and convert the text data into context-related embedding vectors through the pre-trained BERT model; S502, context modeling: context modeling is performed on each candidate keyword through a pre-trained BERT model; S503, Optimizer: By custom The optimizer accelerates the convergence of the pre-trained BERT model, and L2 regularization is used to suppress overfitting of the pre-trained BERT model.
[0024] The context modeling in step S502 of this embodiment is specifically as follows: In the encoding process of the pre-trained BERT model mentioned above, the temporal information, semantic segmentation information and text information of the text are added together to generate the final vector, and the semantic information of the vocabulary is captured according to the final vector; among them, the pre-trained BERT model will consider the context of the vocabulary in the entire text, so as to conduct in-depth analysis of the candidate keywords.
[0025] In step S503 of this embodiment, The optimizer is an adaptive learning rate method that combines the advantages of AdaGrad and RMSProp. The Adam optimizer dynamically adjusts the learning rate of each parameter, automatically adjusting the learning rate based on the first-order moment estimate and second-order moment estimate of the gradient, so as to converge quickly in the early stage of training and avoid oscillation or divergence in the later stage. The first-order moment estimation formula of the optimizer gradient is: ; The second-order moment estimation formula of the optimizer gradient is: ; The bias correction formula of the optimizer is: ; The update parameter formula of the optimizer is: ; in: represents the gradient; and is a hyperparameter, Usually set to 0.9, Usually set to 0.999; represents the basic learning rate; is a small constant to prevent division by zero errors, usually set to ; t represents the number of iteration steps; represents the hyperparameters of the first-order matrix estimation in the t-th iteration; represents the hyperparameters of the second-order matrix estimation in the t-th iteration; exist L2 regularization is introduced on the basis of the optimizer; L2 regularization penalizes weight values that exceed the set threshold by adding a term proportional to the parameter size to the update rule, helping the pre-trained BERT model to select the corresponding parameter value and improve the generalization ability of the model; The update formula after adding L2 regularization is as follows: ; in, l represents the regularization strength coefficient.
[0026] In step S6 of this embodiment, the semantic analysis results based on the pre-trained BERT model are post-processed to optimize and select the final keywords as follows: S601, relevance evaluation: by calculating the semantic relevance between candidate keywords and text, further screening out the most representative and informative keywords; S602 Sorting and filtering: Sorting candidate keywords according to semantic relevance scores, and applying set filtering criteria of occurrence frequency or keyword importance to determine the final keyword set.
[0027] Integrate this embodiment into the data intelligence platform system to ensure that the data intelligence platform system can process newly input data, and apply the keyword extraction method based on BERT+TF-IDF of this embodiment.
[0028] Embodiment 2: This embodiment provides a keyword extraction system, which includes: A collection module, used for collecting text data to be processed; The preprocessing module is used to perform noise removal and word segmentation preprocessing operations on the text data to be processed through the standard text preprocessing tool NLTK to obtain the preprocessed text data; The preliminary keyword screening module is used to perform preliminary analysis on the preprocessed text data through the IF-TDF algorithm, calculate the TF-IDF value of each word in the text data, and use TfidfVectorizer to calculate the TF-IDF value and inverse frequency; The candidate keyword screening module is used to screen candidate keywords from text data based on TF-IDF values, specifically: using the set threshold and sorting mechanism to determine the candidate list of keywords; wherein the threshold is set by the user-defined text similarity value, so as to screen the user keywords; the sorting mechanism is to sort the keywords according to the text similarity value; The semantic analysis module is used to perform semantic analysis on candidate keywords through the pre-trained BERT model to obtain semantic analysis results; Keyword optimization and selection module, which is used to post-process the semantic analysis results based on the pre-trained BERT model, optimize and select the final keywords; The result verification and adjustment module is used to verify and adjust the final keywords, and adjust the pre-trained BERT model parameters and IF-TDF algorithm settings according to the verification results, thereby optimizing the keyword extraction effect.
[0029] The semantic analysis module in this embodiment includes: The text embedding submodule is used to load the pre-trained BERT model using Hugging Face's Transformers library and convert text data into context-sensitive embedding vectors through the pre-trained BERT model; The context modeling submodule is used to perform context modeling on each candidate keyword using the pre-trained BERT model; Optimizer, used to customize The optimizer accelerates the convergence of the pre-trained BERT model and uses L2 regularization to suppress overfitting of the pre-trained BERT model. The optimizer is an adaptive learning rate method that combines the advantages of AdaGrad and RMSProp. The Adam optimizer dynamically adjusts the learning rate of each parameter, automatically adjusting the learning rate based on the first-order moment estimate and second-order moment estimate of the gradient, so as to converge quickly in the early stage of training and avoid oscillation or divergence in the later stage. The first-order moment estimation formula of the optimizer gradient is: ; The second-order moment estimation formula of the optimizer gradient is: ; The bias correction formula of the optimizer is: ; The update parameter formula of the optimizer is: ; in: represents the gradient; and is a hyperparameter, Usually set to 0.9, Usually set to 0.999; represents the basic learning rate; is a small constant to prevent division by zero errors, usually set to ; t represents the number of iteration steps; represents the hyperparameters of the first-order matrix estimation in the t-th iteration; represents the hyperparameters of the second-order matrix estimation in the t-th iteration; exist L2 regularization is introduced on the basis of the optimizer; L2 regularization penalizes weight values that exceed the set threshold by adding a term proportional to the parameter size to the update rule, helping the pre-trained BERT model to select the corresponding parameter value and improve the generalization ability of the model; The update formula after adding L2 regularization is as follows: ; in, l represents the regularization strength coefficient.
[0030] The keyword optimization and selection module in this embodiment includes: The relevance evaluation submodule is used to further select the most representative and informative keywords by calculating the semantic relevance between candidate keywords and text; The sorting and filtering submodule is used to sort the candidate keywords according to the semantic relevance scores and apply the set filtering criteria of the frequency of occurrence or the importance of the keywords to determine the final set of keywords.
[0031] Embodiment 3: This embodiment also provides an electronic device, including: a memory and a processor; Wherein, the memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory, so that the processor executes the keyword extraction method in any embodiment of the present invention.
[0032] The processor may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The processor may be a microprocessor or any conventional processor, etc.
[0033] The memory can be used to store computer programs and / or modules. The processor realizes various functions of the electronic device by running or executing the computer programs and / or modules stored in the memory, and calling the data stored in the memory. The memory can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, at least one application required for a function, etc.; the data storage area can store data created according to the use of the terminal, etc. In addition, the memory can also include a high-speed random access memory, and can also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart memory card (SMC), a secure digital (SD) card, a flash memory card, at least one disk storage period, a flash memory device, or other volatile solid-state storage devices.
[0034] Embodiment 4: This embodiment also provides a computer-readable storage medium, in which a plurality of instructions are stored, and the instructions are loaded by a processor, so that the processor executes the keyword extraction method in any embodiment of the present invention. Specifically, a system or device equipped with a storage medium can be provided, on which a software program code for implementing the functions of any of the above embodiments is stored, and a computer (or CPU or MPU) of the system or device reads out and executes the program code stored in the storage medium.
[0035] In this case, the program code itself read from the storage medium can realize the function of any one of the above-mentioned embodiments, and thus the program code and the storage medium storing the program code constitute a part of the present invention.
[0036] Examples of storage media for providing program code include floppy disks, hard disks, magneto-optical disks, optical disks (such as CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-RYM, DVD-RW, DVD+RW), magnetic tapes, non-volatile memory cards, and ROMs. Alternatively, program code may be downloaded from a server computer via a communication network.
[0037] In addition, it should be clear that the functions of any of the above embodiments can be implemented not only by executing the program code read by the computer, but also by enabling an operating system operating on the computer to complete part or all of the actual operations based on instructions from the program code.
[0038] In addition, it can be understood that the program code read from the storage medium is written to a memory provided in an expansion board inserted into the computer or written to a memory provided in an expansion unit connected to the computer, and then based on the instructions of the program code, a CPU installed on the expansion board or the expansion unit is enabled to perform part or all of the actual operations, thereby realizing the functions of any of the above-mentioned embodiments.
[0039] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein by equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.
Claims
1. A keyword extraction method, characterized in that: The method is as follows: Collect text data to be processed; Use the standard text preprocessing tool NLTK to perform noise removal and word segmentation preprocessing operations on the text data to be processed, and obtain the preprocessed text data; The preprocessed text data is preliminarily analyzed using the IF-TDF algorithm to calculate the TF-IDF value of each word in the text data, and TfidfVectorizer is used to calculate the TF-IDF value and inverse frequency. Filter candidate keywords from text data based on TF-IDF values: use the set threshold and sorting mechanism to determine the candidate list of keywords; the threshold is set by the user-defined text similarity value, so as to filter the user keywords; the sorting mechanism is to sort the keywords according to the text similarity value; Perform semantic analysis on candidate keywords through the pre-trained BERT model to obtain semantic analysis results; Post-process the semantic analysis results based on the pre-trained BERT model to optimize and select the final keywords; The final keywords are verified and adjusted, and the pre-trained BERT model parameters and IF-TDF algorithm settings are adjusted according to the verification results to optimize the keyword extraction effect.
2. The keyword extraction method according to claim 1, characterized in that: The semantic analysis of candidate keywords is performed through the pre-trained BERT model, and the semantic analysis results are obtained as follows: Text embedding: Use Hugging Face's Transformers library to load the pre-trained BERT model and convert the text data into context-sensitive embedding vectors through the pre-trained BERT model; Context modeling: Context modeling is performed on each candidate keyword through the pre-trained BERT model; Optimizer: By customizing The optimizer accelerates the convergence of the pre-trained BERT model and uses L2 regularization to suppress overfitting of the pre-trained BERT model.
3. The keyword extraction method according to claim 2, characterized in that: The context modeling is as follows: In the encoding process of the pre-trained BERT model mentioned above, the temporal information, semantic segmentation information and text information of the text are added together to generate the final vector, and the semantic information of the vocabulary is captured according to the final vector; among them, the pre-trained BERT model will consider the context of the vocabulary in the entire text, so as to conduct in-depth analysis of the candidate keywords.
4. The keyword extraction method according to claim 2, characterized in that: The optimizer is an adaptive learning rate method that combines the advantages of AdaGrad and RMSProp; The Adam optimizer dynamically adjusts the learning rate of each parameter based on the first-order moment estimate and the second-order moment estimate of the gradient to automatically adjust the learning rate, thereby achieving rapid convergence in the early stages of training and avoiding oscillation or divergence in the later stages. The first-order moment estimation formula of the optimizer gradient is: ; The second-order moment estimation formula of the optimizer gradient is: ; The bias correction formula of the optimizer is: ; The update parameter formula of the optimizer is: ; in: represents the gradient; and is a hyperparameter; represents the basic learning rate; is a small constant to prevent division by zero error; t represents the number of iterations; represents the hyperparameters of the first-order matrix estimation in the t-th iteration; represents the hyperparameters of the second-order matrix estimation in the t-th iteration; exist L2 regularization is introduced on the basis of the optimizer; L2 regularization penalizes weight values that exceed the set threshold by adding a term proportional to the parameter size to the update rule, helping the pre-trained BERT model to select the corresponding parameter value and improve the generalization ability of the model; The update formula after adding L2 regularization is as follows: ; in, λ represents the regularization strength coefficient.
5. The keyword extraction method according to claim 1, characterized in that: The semantic analysis results based on the pre-trained BERT model are post-processed to optimize and select the final keywords as follows: Relevance evaluation: By calculating the semantic relevance between candidate keywords and text, the most representative and informative keywords are further screened out; Sorting and filtering: Sort candidate keywords according to semantic relevance scores, and apply set filtering criteria of frequency of occurrence or keyword importance to determine the final set of keywords.
6. A keyword extraction system, characterized in that: The system includes: A collection module, used for collecting text data to be processed; The preprocessing module is used to perform noise removal and word segmentation preprocessing operations on the text data to be processed through the standard text preprocessing tool NLTK to obtain the preprocessed text data; The preliminary keyword screening module is used to perform preliminary analysis on the preprocessed text data through the IF-TDF algorithm, calculate the TF-IDF value of each word in the text data, and use TfidfVectorizer to calculate the TF-IDF value and inverse frequency; The candidate keyword screening module is used to screen candidate keywords from text data based on TF-IDF values, specifically: using the set threshold and sorting mechanism to determine the candidate list of keywords; wherein the threshold is set by the user-defined text similarity value, so as to screen the user keywords; the sorting mechanism is to sort the keywords according to the text similarity value; The semantic analysis module is used to perform semantic analysis on candidate keywords through the pre-trained BERT model to obtain semantic analysis results; Keyword optimization and selection module, which is used to post-process the semantic analysis results based on the pre-trained BERT model, optimize and select the final keywords; The result verification and adjustment module is used to verify and adjust the final keywords, and adjust the pre-trained BERT model parameters and IF-TDF algorithm settings according to the verification results, thereby optimizing the keyword extraction effect.
7. The keyword extraction system according to claim 6, characterized in that: The semantic analysis module includes: The text embedding submodule is used to load the pre-trained BERT model using Hugging Face's Transformers library and convert text data into context-sensitive embedding vectors through the pre-trained BERT model; The context modeling submodule is used to perform context modeling on each candidate keyword using the pre-trained BERT model; Optimizer, used to customize The optimizer accelerates the convergence of the pre-trained BERT model and uses L2 regularization to suppress overfitting of the pre-trained BERT model. The optimizer is an adaptive learning rate method that combines the advantages of AdaGrad and RMSProp. The Adam optimizer dynamically adjusts the learning rate of each parameter, automatically adjusting the learning rate based on the first-order moment estimate and second-order moment estimate of the gradient, so as to converge quickly in the early stage of training and avoid oscillation or divergence in the later stage. The first-order moment estimation formula of the optimizer gradient is: ; The second-order moment estimation formula of the optimizer gradient is: ; The bias correction formula of the optimizer is: ; The update parameter formula of the optimizer is: ; in: represents the gradient; and is a hyperparameter; represents the basic learning rate; is a small constant to prevent division by zero error; t represents the number of iterations; represents the hyperparameters of the first-order matrix estimation in the t-th iteration; represents the hyperparameters of the second-order matrix estimation in the t-th iteration; exist L2 regularization is introduced on the basis of the optimizer; L2 regularization penalizes weight values that exceed the set threshold by adding a term proportional to the parameter size to the update rule, helping the pre-trained BERT model to select the corresponding parameter value and improve the generalization ability of the model; The update formula after adding L2 regularization is as follows: ; in, λ represents the regularization strength coefficient.
8. The keyword extraction system according to claim 6 or 7, characterized in that: The keyword optimization and selection module includes: The relevance evaluation submodule is used to further select the most representative and informative keywords by calculating the semantic relevance between candidate keywords and text; The sorting and filtering submodule is used to sort the candidate keywords according to the semantic relevance scores and apply the set filtering criteria of the frequency of occurrence or the importance of the keywords to determine the final set of keywords.
9. An electronic device, characterized in that: include: memory and at least one processor; Wherein, the memory stores a computer program; The at least one processor executes the computer program stored in the memory, so that the at least one processor performs the keyword extraction method according to any one of claims 1 to 5.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which can be executed by a processor to implement the keyword extraction method according to any one of claims 1 to 5.
Citation Information
Patent Citations
Semantic-based unsupervised academic keyword extraction method and equipment
CN115186665A
Fine-grained sentiment analysis method based on BERT model and attention concentration network
CN115730606A
AI-based medical auxiliary discrimination method and discrimination system
CN117476215A
Word and sentence vector-based online annotation insurance policy term viewing method, device and system
CN117993393A
Activity scheme generation method and system based on artificial intelligence
CN118537044A
