Intelligent invalid keyword filtering method and system for large-scale text mining
By conducting risk assessments and dynamic monitoring of invalid keywords, the problem of insufficient fine-grained classification in invalid keyword filtering is solved, achieving efficient and automated keyword management and improving the accuracy and efficiency of the data mining system.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-01-13
- Publication Date
- 2026-03-24
AI Technical Summary
Existing technologies lack fine-grained classification in filtering invalid keywords, resulting in a lack of clear guidance for system optimization and strategy adjustment. Relying on manual intervention is costly and cannot adapt to the rapid iteration needs of massive amounts of data, affecting search accuracy and user experience.
By classifying invalid keywords based on initial and risk assessments, calculating keyword risk indicators and levels, and combining relevance and resource usage, the status of keywords is dynamically monitored and adjusted to achieve automated management.
It improves the accuracy of filtering invalid keywords, reduces storage costs, reduces waste of computing resources, enhances data governance efficiency and intelligence, adapts to changes in the language environment, and prevents the permanent loss of valid information.
Smart Images
Figure CN121722896A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The application belongs to the technical field of text mining, and particularly relates to an invalid keyword intelligent filtering method and system for large-scale text mining. BACKGROUND
[0002] As a highly condensed and core representation of text information, the extraction quality of keywords directly determines the final effectiveness of downstream tasks such as information retrieval, topic modeling, content recommendation, etc. Therefore, obtaining a high-quality keyword set from complex unstructured text plays a fundamental role in improving data utilization value and optimizing information service experience.
[0003] However, the existing keyword processing technology has insufficient filtering ability for invalid information. The current method only labels keywords as "valid" or "invalid", but cannot deeply and multi-dimensionally classify the causes of invalid keywords. This lack of fine classification labeling system makes subsequent system optimization and strategy adjustment lack clear guidance. There are architectural limitations and real-time bottlenecks in the processing flow. The existing technology usually uses offline or batch processing to identify invalid keywords, resulting in a time delay between problem discovery and system rule updating. During this delay window, invalid keywords continue to pollute data analysis results, affecting the accuracy of retrieval and user experience. The system lacks the ability of dynamic self-adaptation and real-time optimization, relies on manual intervention, and has high operation and maintenance costs. Due to the lack of intelligent classification and feedback mechanism, system managers invest a lot of manpower in secondary screening, attribution, and manual updating of the filter dictionary for the general "invalid" label, which is not only inefficient and costly, but also prone to human errors, making it difficult to adapt to the rapid iteration needs of massive data.
[0004] Therefore, the application provides an invalid keyword intelligent filtering method and system for large-scale text mining. SUMMARY
[0005] The application aims to provide an invalid keyword intelligent filtering method for large-scale text mining, which can reasonably classify and process invalid keywords, reduce the waste of computing resources caused by matching invalid keywords without affecting user experience, and improve the calling efficiency of keywords in the database.
[0006] To achieve the above application purpose, the technical solution adopted by the application is as follows: an invalid keyword intelligent filtering method for large-scale text mining, comprising:
[0007] Based on the initial evaluation, the keywords are judged as first-level invalid keywords, and risk assessment and reclassification processing are performed on the first-level invalid keywords, including calculating the risk indicators of the first-level invalid keywords.
[0008] based on the risk level, the first-level invalid keyword is classified and identified;
[0009] The initial evaluation includes: obtaining sentence information and keyword information, counting the number of times the keyword appears in the sentence, and taking the number of times as the correlation degree parameter; based on the correlation degree parameter, the keyword whose correlation degree does not meet the preset condition is judged as a first-level invalid keyword.
[0010] Preferably, the step of calculating the risk indicator of the first-level invalid keyword includes: obtaining the total storage duration and the total byte amount of the first-level invalid keyword in the database; substituting the total storage duration and the total byte amount into a preset to-be-evaluated indicator calculation function to calculate a to-be-evaluated indicator, and substituting the to-be-evaluated indicator into a preset risk indicator calculation function to calculate the risk indicator.
[0011] Preferably, the step of classifying and identifying the first-level invalid keyword according to the risk level includes: when the risk level is a low risk level, the first-level invalid keyword is identified as a second-level invalid keyword.
[0012] Preferably, the method further includes: when the number of occurrences of the second-level invalid keyword in the database is greater than or equal to a preset number threshold, the second-level invalid keyword is identified as a valid keyword.
[0013] An invalid keyword intelligent filtering system for large-scale text mining includes:
[0014] A keyword initial judgment module is configured to judge a keyword as a first-level invalid keyword based on initial evaluation;
[0015] A risk indicator calculation module is configured to perform risk evaluation and reclassification processing on the first-level invalid keyword, and the risk evaluation and reclassification processing includes: calculating a risk indicator of the first-level invalid keyword; and classifying and identifying the first-level invalid keyword according to a risk level;
[0016] A risk level evaluation module is configured to determine a risk level of the first-level invalid keyword based on the risk indicator, and identify the first-level invalid keyword as a second-level invalid keyword when the risk level is a low risk level;
[0017] A keyword revalidation judgment module is configured to monitor the number of occurrences of the second-level invalid keyword in the database, and identify the second-level invalid keyword as a valid keyword when the number of occurrences is greater than or equal to a preset number threshold.
[0018] Preferably, the initial evaluation includes: obtaining sentence information and keyword information, counting the number of times the keyword appears in the sentence, and taking the number of times as the correlation degree parameter; based on the correlation degree parameter, the keyword whose correlation degree does not meet the preset condition is judged as a first-level invalid keyword.
[0019] Preferably, the calculating the risk indicator of the primary invalid keyword comprises: obtaining a total storage duration and a total byte amount of the primary invalid keyword in the database.
[0020] Preferably, the total storage duration and the total byte amount are substituted into a preset evaluation indicator calculation function to calculate an evaluation indicator, and the evaluation indicator is substituted into a preset risk indicator calculation function to calculate the risk indicator.
[0021] An electronic device comprises at least one processor and a memory connected with the at least one processor in communication.
[0022] The memory stores computer program instructions, and the computer program instructions are executed by the at least one processor to implement an intelligent invalid keyword filtering method for large-scale text mining.
[0023] A computer readable storage medium stores a computer program, and the computer program is executed by a processor to implement an intelligent invalid keyword filtering method for large-scale text mining.
[0024] Advantages
[0025] 1. The intelligent invalid keyword filtering method provided by the present application can accurately distinguish long-term existing and high storage cost invalid keywords by calculating the risk indicator to determine the risk level according to the total storage duration and the total byte amount of the primary invalid keyword, instead of only judging according to the frequency of occurrence, thereby improving the filtering accuracy of the invalid keyword, and the mechanism based on risk assessment can accurately identify and process the invalid keyword with high storage cost, thereby reducing the storage cost of the database and the waste of computing resources caused by invalid retrieval.
[0026] 2. The present application identifies the keyword with a low risk level as a secondary invalid keyword, and continuously counts the number of occurrences of the keyword in the database. When the number of occurrences is greater than or equal to a preset number threshold, the keyword is identified as a valid keyword. The dynamic feedback and correction mechanism is established, the invalid keyword intelligent filtering system is given the self-adaptive ability to language environment changes, and the keyword that is misjudged initially or has a frequency of use that rises later is automatically identified and recovered. The defect of permanent loss of valid information caused by static filtering rules is overcome, and the long-term effectiveness and integrity of the keyword set are ensured.
[0027] 3、The application divides the keywords into effective keywords, first-level invalid keywords and second-level invalid keywords, adopts a differentiated processing strategy combining risk assessment and dynamic monitoring for invalid keywords of different levels, generates notification information containing risk levels, and realizes an automatic keyword life cycle management process based on the hierarchical processing and dynamic adjustment mechanism, which converts keyword screening into a structured and intelligent management process, reduces the dependence on manual review, and improves the overall efficiency and intelligent level of data governance in large-scale text mining applications. BRIEF DESCRIPTION OF DRAWINGS
[0028] Figure 1 is a method flowchart for obtaining keywords and preliminarily identifying keywords of the application;
[0029] Figure 2 is a method flowchart for determining the risk of first-level invalid keywords of the application;
[0030] Figure 3 is a method flowchart for monitoring second-level invalid keywords to distinguish effective keywords of the application;
[0031] Figure 4 is a system module diagram provided by Embodiment 2 of the application. DETAILED DESCRIPTION
[0032] In the following description, certain specific details are set forth in order to provide a thorough understanding of the different embodiments of the application. However, persons skilled in the art will understand that the application can be practiced without many of these details after reading the present disclosure.
[0033] Embodiment I
[0034] Referring to Figure 1 , the embodiment provides an invalid keyword intelligent filtering method for large-scale text mining, which is specifically applied to automatically identify and classify keywords with low relevance to text content and occupy system resources, thereby improving data quality and storage efficiency. The complete processing flow is integrated into a coherent and automatically executed narrative paragraph, which specifically includes the following steps:
[0035] The sentence information to be processed is obtained, such as batch documents, log records or user-generated content obtained from a large-scale text corpus, and keyword information extracted therefrom. For each keyword, the number of occurrences in the sentence information is counted, and the number is directly used as a correlation degree parameter measuring the close degree of the keyword to the sentence content, which is a numerical indicator quantifying the close degree of the keyword to the text content, and the value is directly determined by the total number of occurrences of the keyword in the specified sentence information. The correlation degree parameter of each keyword is compared with a preset correlation degree threshold value to serve as the basis for preliminary division of effective keywords and first-level invalid keywords. If the correlation degree parameter of a keyword is greater than or equal to the correlation degree threshold value, it is determined to be an effective keyword closely related to the current sentence content, and is marked as an effective keyword. Otherwise, if the correlation degree parameter is less than the correlation degree threshold value, it is initially evaluated as a keyword with a low correlation degree to the current sentence content, and is marked as a first-level invalid keyword for subsequent in-depth resource occupation analysis.
[0036] Further, all entries identified as first-level invalid keywords are collected to form an initial set. To improve processing efficiency, the set is subjected to deduplication processing based on the text content of the keywords to generate a unique list of invalid keywords, and optionally a unique index number is established for each keyword in the list. The keyword string is converted to an integer index through a hash mapping rule to enable high-speed retrieval in subsequent steps. Based on the index number or the keyword itself, the first occurrence time point and the last occurrence time point of each unique invalid keyword are queried and recorded in the historical database, and the total storage time is calculated based on the time difference between the two time points to serve as an indicator measuring the resource occupation in the time dimension of the keyword. The total storage space occupied by all related entries of the keyword in the database is counted and recorded as the total byte amount to serve as an indicator measuring the resource occupation in the space dimension of the keyword.
[0037] Further, according to the total byte amount and the total storage duration obtained in the previous step, the resource occupation cost of each first-level invalid keyword is quantified by a preset calculation rule, and the value of the invalid keyword resource occupation cost is quantified to obtain a comprehensive evaluation index. The calculation rule can be a weighted summation process. The composite index is obtained by weighted combination calculation of the total byte amount and the total storage duration. In the calculation of the evaluation index, the preset weight value is assigned to the total byte amount, and the total byte amount is multiplied by the preset space cost weight coefficient to adjust the importance of the storage space occupation in the total cost evaluation. In the calculation of the evaluation index, the preset weight value is assigned to the total storage duration, and the total storage duration is multiplied by the preset time cost weight coefficient to adjust the importance of the time occupation in the total cost evaluation. The two products are added to obtain the evaluation index. The weight coefficient can be pre-configured by the administrator according to the sensitivity of the business system to the storage space or time occupation. The obtained evaluation index is input into a preset conversion rule to calculate a standardized risk index representing the final resource occupation risk degree of the invalid keyword, which is convenient for unified comparison and decision-making. The design purpose of the preset conversion rule is to map the evaluation index with different dimensions and value ranges into a bounded standardized interval, such as 0 to 1, so as to facilitate unified threshold comparison. The conversion rule can be set as: when the value of the input evaluation index is low, the growth of the output risk index is relatively flat; when the evaluation index is in the middle range, the growth of the risk index is relatively rapid; and when the value of the evaluation index is very high, the growth of the risk index again tends to be flat and gradually approaches the upper limit value of the interval.
[0038] The preset conversion rule is used to map the evaluation index to the calculation model of the standardized risk index interval, and the specific mathematical formula is as follows:
[0039]
[0040] In the formula, represents the risk index, which means that the standardized resource occupation risk of the keyword is normalized to 0 and 1; represents the evaluation index, which means that the comprehensive score of the original resource occupation cost of the keyword is represented; represents the steepness parameter, which means that it is a positive constant for controlling the growth rate of the function curve near the inflection point, the greater the value, the steeper the curve, and the faster the transition of the risk index from low to high; represents the inflection point parameter, which means that it is a constant representing the value of the evaluation index corresponding to the risk index value of 0.5, that is, the midpoint or sensitive point of risk identification.
[0041] Further, the risk index calculated in the previous step is compared with a preset risk threshold, which is a value between 0 and 1 configurable by the administrator, for comparison with the risk index to determine whether the risk level of the keyword is high or low, and the risk level is determined based on the comparison result, if the risk index is greater than the risk threshold, the risk level of the keyword is determined as a high risk level, indicating that the keyword has a high risk priority, and it is finally identified as a first-level high-risk invalid keyword, such keywords represent those entries with low relevance to business content and long-term occupation of a large amount of system resources, and notification information containing such keywords and their risk levels will be generated for the administrator to perform priority review, confirmation or cleaning operation, thereby quickly releasing valuable storage resources, otherwise, if the risk index is not greater than the risk threshold, the risk level is determined as a low risk level, indicating that the resource occupation cost of the keyword is within an acceptable range, and it is identified as a second-level invalid keyword, such keywords will be included in continuous monitoring rather than being immediately marked as cleaning objects, this classification operation aims to include those keywords with resource occupation costs within an acceptable range in the continuous observation process, as they may be new terms, abbreviations for specific scenarios or misspelled words, i.e. keywords with low direct relevance, thereby achieving a technical balance between efficient cleaning of system noise and avoiding accidental deletion of potential useful information.
[0042] Further, for all entries identified as secondary invalid keywords, a continuous monitoring and reevaluation process is performed to determine whether to maintain its current classification or reevaluate it by continuously analyzing the keyword's occurrence interval and recent activity, specifically in this process, the continuity of its activity is analyzed, all historical occurrence time points of a certain secondary invalid keyword in the database are obtained, arranged in chronological order, and the time interval between adjacent two occurrence time points is calculated, if the calculated maximum time interval or average time interval is greater than a preset time interval threshold, it indicates that the occurrence mode of the keyword is sporadic and lacks continuity, in this case, its invalid attribute is strengthened, even if its subsequent occurrence frequency increases in a short period, the identification of its secondary invalid keyword is maintained to prevent those already inactive keywords from being mistakenly reactivated, under the premise of meeting the continuity requirement, the occurrence frequency of each secondary invalid keyword in a specified recent query time period is further counted to evaluate its recent activity, the counted occurrence frequency is compared with a preset frequency threshold, preferably 5 times, which is used to measure whether the keyword's activity has significantly improved in the recent query time period, and is a key basis for deciding whether to upgrade or reevaluate the secondary invalid keyword, when the occurrence frequency of a certain secondary invalid keyword is greater than or equal to the preset frequency threshold, it indicates that its recent importance or relevance has significantly improved, and it is identified as an effective keyword, in an optional and more rigorous implementation manner, when the above conditions are met, instead of directly upgrading it, the secondary invalid keyword is first sent to a word form normalization processing unit, which is responsible for converting different forms of words, such as plurals and different tenses of verbs, into uniform and standard word roots or basic forms, after processing, the standardized keyword is readded to the initial set of primary invalid keywords at the beginning of the above-mentioned "collect all entries identified as primary invalid keywords to form an initial set" step, so that it starts from the resource occupation analysis and goes through a complete risk assessment and classification process, this built-in feedback loop design ensures that keywords reevaluated in subsequent steps, such as a secondary invalid keyword with improved activity, are sent back to the early stage of the resource analysis phase of the primary invalid keyword, to ensure that the classification of all keywords follows the same, complete evaluation logic, ensuring that the final classification of all keywords follows uniform and rigorous evaluation standards, and ensuring the logical closed loop and consistency of the entire filtering method.
[0043] Embodiment Two
[0044] See Figure 2As shown, the embodiment provides an invalid keyword intelligent filtering system for large-scale text mining, which realizes accurate identification and dynamic management of invalid keywords in massive texts through a multi-stage evaluation and classification mechanism, thereby improving the efficiency and accuracy of text mining. In specific implementation, the system can be deployed on electronic devices such as servers, personal computers or cloud platforms, and implemented through computer programs. The system can be logically divided into the following modules that work together:
[0045] The keyword initial judgment module is configured to perform initial evaluation of keywords, and identify keywords with an association degree parameter less than a preset association degree threshold as first-level invalid keywords. In a specific execution process, the sentence information to be processed and the keyword information extracted therefrom are obtained, the number of occurrences of a specific keyword in the corresponding sentence is counted, and the number of occurrences is taken as the association degree parameter. The association degree parameter is compared with the preset association degree threshold. If the association degree parameter of the keyword is less than the association degree threshold, the keyword is identified as a first-level invalid keyword, and is output to the subsequent module for further risk assessment.
[0046] The risk indicator calculation module is configured to receive the first-level invalid keyword output by the keyword initial judgment module and calculate the risk indicator thereof. The module accesses a database storing historical keyword information to obtain two key attributes of the first-level invalid keyword: total storage time and total byte amount. The total storage time refers to the time span between the last occurrence time point and the first occurrence time point of the keyword recorded in the database, and the total byte amount refers to the total size of the storage space occupied by all entries related to the keyword in the database. The obtained total storage time and total byte amount are substituted into a preset evaluation indicator calculation function to generate an evaluation indicator. The function can be a weighted summation or comprehensive scoring algorithm to preliminarily quantify the resource occupation cost of the keyword. The calculated evaluation indicator is substituted into a preset risk indicator calculation function, for example, through normalization or nonlinear mapping, to convert it into a risk indicator for subsequent grade division.
[0047] A risk level evaluation module is configured to perform risk evaluation and reclassification on the primary invalid keywords based on the risk indicators, receive the risk indicators generated by the risk indicator calculation module, and determine the risk level of the primary invalid keywords according to the risk indicators, specifically, different risk thresholds are set in the module, and the risk indicators are compared with the thresholds to determine whether the risk level is high, medium or low, and the primary invalid keywords are classified and identified according to the risk level, in particular, when the risk level is determined to be low, the primary invalid keyword is identified as a secondary invalid keyword, which means that the keyword is currently invalid, but the resource occupation cost is low, and it has the possibility of being observed and reactivated, so it should not be immediately and permanently excluded.
[0048] A keyword reactivation judgment module is configured to continuously monitor the secondary invalid keywords and identify them as valid keywords when certain conditions are met, periodically or in real time, monitor the occurrence of the keywords identified as secondary invalid keywords in the entire database or text corpus, specifically, the total number of occurrences of each secondary invalid keyword in the database is counted, and the counted number of occurrences is compared with a preset number threshold, when the number of occurrences of a secondary invalid keyword is greater than or equal to the preset number threshold, it indicates that the actual use frequency of the keyword has risen to a statistically significant level, its relevance and importance have been restored, the secondary invalid keyword is reidentified as a valid keyword, so that it can participate in the normal text mining process again.
[0049] The embodiment of the application also provides an electronic device, which can be a physical carrier for implementing the above-mentioned system, such as a server, an industrial computer or an embedded device, which includes at least one processor and a memory in communication connection with the at least one processor in the hardware structure, and the memory stores computer program instructions.
[0050] The embodiment of the application also provides a computer readable storage medium, which stores computer programs, and when the at least one processor executes the instructions, the above-mentioned method is executed, specifically, the above-mentioned modules and all or part of the functions of the correctness determination module are implemented.
[0051] Through the cooperative work of the modules, the electronic device and the storage medium, the system of the embodiment can construct a dynamic and hierarchical invalid keyword filtering and management process. Not only can the system filter out the keywords initially judged as invalid, but also can finely classify the keywords through a risk assessment mechanism, and provide continuous monitoring and reevaluation process for the secondary invalid keywords, thereby avoiding the misjudgment of the keywords with potential value, improving the robustness and intelligent level of the large-scale text mining system, and being suitable for application scenarios such as knowledge management and search engines which need to maintain and optimize the keyword library for a long time.
Claims
1. A method for intelligent filtering of invalid keywords for large-scale text mining, characterized in that, include: Based on the initial assessment, the keywords were identified as Level 1 invalid keywords. Risk assessment and reclassification were then performed on these Level 1 invalid keywords. The risk assessment and reclassification included calculating the risk indicators for Level 1 invalid keywords. Based on risk indicators, the risk level of primary invalid keywords is determined, and primary invalid keywords are classified and labeled according to their risk level. The initial assessment includes: obtaining sentence information and keyword information, counting the number of times keywords appear in sentences, using the number of times as a relevance parameter, and judging keywords whose relevance does not meet the preset conditions as first-level invalid keywords based on the relevance parameter.
2. The method for intelligent filtering of invalid keywords for large-scale text mining according to claim 1, characterized in that, The steps for calculating the risk index of primary invalid keywords include: obtaining the total storage time and total number of bytes of primary invalid keywords in the database; substituting the total storage time and total number of bytes into the preset evaluation index calculation function to calculate the evaluation index; and substituting the evaluation index into the preset risk index calculation function to calculate the risk index.
3. The method for intelligent filtering of invalid keywords for large-scale text mining according to claim 1, characterized in that, The steps for classifying and labeling primary invalid keywords based on risk level include: when the risk level is low, primary invalid keywords are labeled as secondary invalid keywords.
4. The method for intelligent filtering of invalid keywords for large-scale text mining according to claim 3, characterized in that, The method also includes: when the number of occurrences of a secondary invalid keyword in the database is greater than or equal to a preset threshold, the secondary invalid keyword is marked as a valid keyword.
5. An intelligent filtering system for invalid keywords in large-scale text mining, characterized in that, include: The keyword initial judgment module is used to determine keywords as level one invalid keywords based on the initial evaluation; The risk indicator calculation module is used to perform risk assessment and reclassification of primary invalid keywords. The risk assessment and reclassification process includes: calculating the risk indicators of primary invalid keywords; and classifying and labeling primary invalid keywords according to their risk levels. The risk level assessment module is used to determine the risk level of primary invalid keywords based on risk indicators, and to mark primary invalid keywords as secondary invalid keywords when the risk level is low. The system also includes a keyword validity determination module, which monitors the number of times secondary invalid keywords appear in the database and identifies them as valid keywords when the number of appearances is greater than or equal to a preset threshold.
6. The intelligent filtering system for invalid keywords for large-scale text mining according to claim 5, characterized in that, The initial assessment includes: obtaining sentence information and keyword information, counting the number of times keywords appear in sentences, and using the number of times as a relevance parameter; based on the relevance parameter, keywords whose relevance does not meet the preset conditions are judged as first-level invalid keywords.
7. The intelligent filtering system for invalid keywords for large-scale text mining according to claim 5, characterized in that, include: The risk indicators for calculating first-level invalid keywords include: obtaining the total storage time and total number of bytes of first-level invalid keywords in the database.
8. The intelligent filtering system for invalid keywords for large-scale text mining according to claim 7, characterized in that, The total storage duration and total number of bytes are substituted into the preset evaluation indicator calculation function to calculate the evaluation indicator; the evaluation indicator is substituted into the preset risk indicator calculation function to calculate the risk indicator.
9. An electronic device, characterized in that, include: At least one processor; and memory that is communicatively connected to at least one processor; The memory stores computer program instructions, which, when executed by at least one processor, implement an intelligent filtering method for invalid keywords for large-scale text mining as claimed in any one of claims 1 to 4.
10. A computer-readable storage medium, characterized in that, It stores a computer program, which, when executed by a processor, implements an intelligent filtering method for invalid keywords for large-scale text mining as described in any one of claims 1 to 4.