A knowledge base management system based on NLP and machine learning

Abstract

This invention relates to the field of knowledge base management technology, and more particularly to a knowledge base management system based on NLP and machine learning. The system includes: a knowledge acquisition module for receiving raw knowledge data and performing text preprocessing to generate preprocessed knowledge data; a knowledge processing and classification module for receiving preprocessed knowledge data, performing text vectorization and classification to generate categorized knowledge data; a knowledge storage and index building module for storing categorized knowledge data and building a knowledge index based on the categorized knowledge data; and a knowledge update and maintenance module for filtering raw knowledge data stored within a historical time period based on user-input search terms and their historical search counts, determining the optimal raw knowledge data, and dynamically adjusting the knowledge index and recommendation strategy.

Description

Technical Field

[0001] This invention relates to the field of knowledge base management technology, and in particular to a knowledge base management system based on NLP and machine learning. Background Technology

[0002] Traditional knowledge base management methods face problems such as low automation, poor data processing efficiency, and excessive human intervention. Specifically, the collection, classification, and storage of knowledge usually rely on manual input and traditional rules, making it impossible to efficiently handle large amounts of data and constantly changing information needs.

[0003] While existing knowledge base management systems can perform basic data storage and retrieval, they have not yet fully met the needs for dynamic information updates, intelligent recommendations, and sentiment analysis. Summary of the Invention

[0004] In view of the above-mentioned shortcomings and deficiencies of the prior art, the present invention provides a knowledge base management system based on NLP and machine learning.

[0005] To achieve the above objectives, the main technical solutions adopted by the present invention include:

[0006] This invention provides a knowledge base management system based on NLP and machine learning, the system comprising:

[0007] The knowledge acquisition module is used to receive raw knowledge data and perform text preprocessing to generate preprocessed knowledge data.

[0008] The knowledge processing and classification module is used to receive preprocessed knowledge data, perform text vectorization and classification, and generate classified knowledge data.

[0009] The knowledge storage and indexing module is used to store categorized knowledge data and build knowledge indexes based on the categorized knowledge data.

[0010] The knowledge update and maintenance module is used to filter the original knowledge data stored in the historical time period based on the search terms entered by the user and their historical search counts, and to determine the optimal original knowledge data.

[0011] Preferably, the knowledge acquisition module includes:

[0012] A data input unit is used to receive raw knowledge data, including documents uploaded by users;

[0013] A data preprocessing unit is used to perform preprocessing on raw knowledge data, the preprocessing including:

[0014] Remove HTML tags, advertisements, garbled characters, and duplicate text to obtain cleaned text data;

[0015] For the cleaned text data, the BERT attention mechanism combined with the TF-IDF algorithm is used to calculate word weights in order to extract the keyword set;

[0016] The LSTM-CRF model was used to extract sentiment features from the cleaned text data and generate sentiment score data.

[0017] The cleaned text data is processed based on the BART pre-trained model to generate a text summary and output the summary data.

[0018] The data output unit is used to output preprocessed knowledge data and provide it to the knowledge processing and classification module.

[0019] Preferably, for the cleaned text data, the BERT attention mechanism combined with the TF-IDF algorithm is used to calculate word weights in order to extract a keyword set, specifically including:

[0020] Formula (1) is used to calculate the value of each term t in the cleaned text data. i The TF-IDF weights; the formula (1) is:

[0021]

[0022] Among them, TF·IDF(t) i ) is the term t i TF-IDF weights;

[0023] f i For the term t i The number of times it appears in the cleaned text data;

[0024] N represents the total number of cleaned text data in the corpus; wherein the corpus includes all original knowledge data received within a historical time period.

[0025] n i For the term t i The amount of cleaned text data;

[0026] TTF represents the total number of occurrences of all terms in the cleaned text data;

[0027] Based on the pre-trained BERT model, the cleaned text data in the pre-processed text data is processed using a Transformer structure to obtain the attention matrix, and the attention matrix of each term t is calculated using formula (2). i The average attention score across all layers of the BERT model;

[0028] The formula (2) is:

[0029]

[0030] Among them, S i For the term t i The average attention score across all layers of the BERT model;

[0031] A h，i For the h-th layer of the BERT model, the term t i Attention score;

[0032] H represents the total number of layers in the BERT model;

[0033] The term t is calculated using formula (3). i Location information P in the cleaned text data i Normalization process yields term t i Position weight;

[0034] The formula (3) is:

[0035]

[0036] Among them, R i For the term t i Position weight; P i For the term t i The sum of the order of arrangement in the cleaned text data; L is the total number of terms in the cleaned text data;

[0037] Based on term t i TF-IDF weights, term t i Average attention score and term t across all layers of the BERT model i Position weights are calculated for each term t using formula (4). i The final keyword weight W i ;

[0038] The formula (4) is:

[0039] W i =γ1×TF·IDF(t) i )+γ2×S i +γ3×R i ;

[0040] Where γ1 is the first weight adjustment parameter; γ2 is the second weight adjustment parameter; γ3 is the third weight adjustment parameter; γ1+γ2+γ3=1;

[0041] Based on the calculated comprehensive keyword weight W i Select the top K keywords with the highest keyword weights to form the final keyword set, and output the keyword set.

[0042] Preferably, the knowledge processing and classification module includes:

[0043] The text vectorization unit is used to receive preprocessed knowledge data and convert the cleaned text data into text vector data;

[0044] The classification unit is used to perform text classification based on text vector data and generate classification knowledge data.

[0045] The data output unit is used to output categorized knowledge data and transmit it to the knowledge storage and index building module.

[0046] Preferably, the classification unit is used to receive text vector data and perform text classification to obtain classification knowledge data, specifically including:

[0047] The classification unit uses a preset support vector machine model to classify text vector data. After classification, each piece of text data is assigned corresponding classification knowledge data and the keyword set.

[0048] The preset support vector machine model is obtained by training text vector data of historical stages labeled with classification knowledge data tags.

[0049] The classification knowledge data includes classification labels and semantic vectors.

[0050] Preferably, the knowledge storage and indexing construction module includes:

[0051] Storage unit, used to store classified knowledge data using a specified storage method;

[0052] The specified storage methods include: MongoDB and PostgreSQL;

[0053] Index building unit, used to build knowledge indexes based on categorized knowledge data;

[0054] The knowledge index includes an inverted index and an HNSW index;

[0055] The data output unit is used to output the knowledge index and transmit it to the knowledge update and maintenance module.

[0056] Preferably, the knowledge update and maintenance module includes: an event-driven update unit, used to calculate the relevance score S between any original knowledge data stored in a historical time period and the search term q based on the search term q input by the user and the historical search count of the search term q;

[0057] Based on the relevance score S, the optimal raw knowledge data is selected from the raw knowledge data received within the historical time period.

[0058] The optimal raw knowledge data is the raw knowledge data corresponding to the highest relevance score S.

[0059] Preferably, the relevance score S between any original knowledge data stored in the historical time period and the search term q is calculated using formula (5) based on the search term q entered by the user and the historical search count of the search term q.

[0060] The formula (5) is:

[0061]

[0062] S is the relevance score between the search term q and any original knowledge data;

[0063] f q The number of times the search term q appears in any original knowledge data;

[0064] M represents the total amount of original knowledge data stored within the historical time period;

[0065] n q The amount of raw knowledge data containing the search term q stored within a historical time period;

[0066] T q This represents the total number of searches for the term "q" within the historical time period.

[0067] T represents the total number of searches for all keywords;

[0068] α is a pre-set balance factor.

[0069] Preferably, the original knowledge data is in any one of the following formats: Word, PDF, TXT, and HTML.

[0070] Preferably, 0 < α < 1.

[0071] The beneficial effects of this invention are as follows: This knowledge base management system based on NLP and machine learning employs a knowledge acquisition module to preprocess raw knowledge data, including steps such as removing irrelevant information, keyword extraction, sentiment analysis, and summary generation. Combined with machine learning for text vectorization and classification, it ensures efficient data processing and classification. Compared to existing technologies, it can automatically process massive amounts of raw knowledge data, improving data processing efficiency and accuracy. Through a knowledge storage and indexing module, it can quickly store categorized data and generate accurate knowledge indexes, significantly improving storage and index retrieval efficiency compared to traditional manual processing methods. Furthermore, the knowledge update and maintenance module, by filtering and dynamically adjusting based on user search behavior and historical data, can update knowledge base content in real time and optimize recommendation strategies, ensuring the timeliness and personalized recommendations of the knowledge base. This achieves more accurate and personalized knowledge management, improving user query efficiency and the accuracy of knowledge acquisition. Attached Figure Description

[0072] Figure 1 This is a schematic diagram of the structure of a knowledge base management system based on NLP and machine learning according to the present invention;

[0073] Figure 2 This is a flowchart of a knowledge base management method based on NLP and machine learning provided in Embodiment 4 of the present invention. Detailed Implementation

[0074] To better explain and facilitate understanding of the present invention, the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0075] To better understand the above technical solutions, exemplary embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present invention are shown in the drawings, it should be understood that the present invention can be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that the present invention can be understood more clearly and thoroughly, and that the scope of the present invention can be fully conveyed to those skilled in the art.

[0076] Example 1

[0077] Figure 1 As shown in the diagram, this embodiment provides a structural block diagram of a knowledge base management system based on NLP and machine learning. The system includes a knowledge acquisition module, a knowledge processing and classification module, a knowledge storage and index construction module, and a knowledge update and maintenance module. Each module plays a different role in the system, working together to achieve efficient and intelligent knowledge base management.

[0078] The knowledge acquisition module includes:

[0079] Data Input Unit: This unit receives raw knowledge data from various sources, including user-uploaded documents (such as Word, PDF, and TXT formats), data from external databases, text data from social media, and real-time streaming data. The raw data is input directly into the data input unit without any processing before entering the system.

[0080] Data Preprocessing Unit: After receiving the raw knowledge data from the data input unit, the data preprocessing unit further processes the raw data. First, irrelevant information such as HTML tags, advertisements, garbled characters, and duplicate text is removed to obtain clean and usable cleaned text data. Then, for the cleaned text data, the BERT attention mechanism combined with the TF-IDF algorithm is used to calculate the weights of words, and the keyword set in the text is extracted using these weights. Next, the LSTM-CRF model is used to extract sentiment features from the text, and sentiment score data is generated for each document. In addition, based on the BART pre-trained model, the cleaned text data is processed to generate text summaries, and these summaries are used as part of the output data for subsequent modules.

[0081] Data Output Unit: The processed data, including keyword sets, sentiment score data, and summary data, will be output through the data output unit and provided to the knowledge processing and classification module for further processing.

[0082] The knowledge processing and classification module includes:

[0083] After receiving preprocessed knowledge data from the knowledge acquisition module, the knowledge processing and classification module performs text vectorization on the data. Specifically, firstly, a BERT-based pre-trained model is used to vectorize the cleaned text data, converting the text into a high-dimensional vector representation. Next, machine learning algorithms (such as Support Vector Machines (SVM) or XGBoost classifiers) are used to classify the vectorized text data, generating categorized knowledge data. This categorized knowledge data includes different categories and labels based on text content and features. The categorized knowledge data is then further transferred to the knowledge storage and indexing module.

[0084] The knowledge storage and indexing module is primarily used to store categorized knowledge data and build corresponding knowledge indexes. It utilizes distributed databases (such as MongoDB and Cassandra) to store the categorized knowledge data. To improve subsequent knowledge retrieval efficiency, inverted indexes and hierarchical navigation small-world graphs (HNSW) are used for index optimization, ensuring fast and accurate retrieval and querying even in large-scale data environments. The constructed knowledge indexes will further provide data support for the knowledge update and maintenance module.

[0085] The knowledge update and maintenance module filters the raw knowledge data stored in the system based on user-input search terms and their historical search frequency, selecting the optimal raw knowledge data within a historical timeframe. The filtering process calculates user interests and search relevance to ensure users obtain the most relevant and high-quality knowledge content during their searches. The filtered data is then transferred to the knowledge index for updates, and the recommendation strategy is optimized based on user feedback, thereby ensuring the accuracy and timeliness of the system's recommendations.

[0086] The systems and methods described in the embodiments of the present invention can efficiently and intelligently process and manage knowledge data, enabling real-time updates, dynamic adjustments, and personalized recommendations. This significantly improves the efficiency and accuracy of knowledge base management, and provides strong support for data management and retrieval in large-scale knowledge base environments.

[0087] Example 2

[0088] Suppose a company wants to build a customer service knowledge base based on natural language processing, capable of automatically pushing relevant solutions based on customer questions. The system will execute the following steps:

[0089] The company collected customer service-related document data from multiple departments, including customer issue records, technical support documents, and training materials. The data input unit used these documents as raw knowledge data input into the system. The data preprocessing unit removed HTML tags and duplicate content from the documents, extracted keywords using the BERT attention mechanism combined with the TF-IDF algorithm, and extracted sentiment features using the LSTM-CRF model. The final sentiment score data and summary information were then used as output data and entered into the knowledge processing and classification module.

[0090] In the knowledge processing and classification module, the system vectorizes these text data and classifies them using an SVM classifier.

[0091] The categorized knowledge data is stored in a distributed database and optimized using inverted indexes and HNSW indexing techniques. Through index optimization, the system can quickly match user queries and return relevant solutions. The constructed knowledge index also supports efficient personalized recommendations in future searches.

[0092] As customer questions and feedback continue to emerge, the system dynamically updates its knowledge base based on users' search history data (such as click counts) to ensure customers receive the latest and most relevant solutions. Simultaneously, the system adjusts its recommendation strategy based on user feedback, enabling more personalized recommendations and further enhancing the user experience.

[0093] Through these operations, the company is able to significantly improve the efficiency and accuracy of customer service by leveraging NLP and machine learning technologies, while ensuring that the knowledge base is always up-to-date and that personalized knowledge recommendations are made based on actual needs.

[0094] Example 3

[0095] See Figure 1 This embodiment provides a knowledge base management system based on NLP and machine learning, the system comprising:

[0096] The knowledge acquisition module is used to receive raw knowledge data and perform text preprocessing to generate preprocessed knowledge data.

[0097] The knowledge acquisition module described in this embodiment includes:

[0098] A data input unit is used to receive raw knowledge data, including documents uploaded by users;

[0099] A data preprocessing unit is used to perform preprocessing on raw knowledge data, the preprocessing including:

[0100] Remove HTML tags, advertisements, garbled characters, and duplicate text to obtain cleaned text data;

[0101] For the cleaned text data, the BERT attention mechanism combined with the TF-IDF algorithm is used to calculate word weights in order to extract the keyword set;

[0102] The LSTM-CRF model was used to extract sentiment features from the cleaned text data and generate sentiment score data.

[0103] The cleaned text data is processed based on the BART pre-trained model to generate a text summary and output the summary data.

[0104] The data output unit is used to output preprocessed knowledge data and provide it to the knowledge processing and classification module.

[0105] The knowledge processing and classification module is used to receive preprocessed knowledge data, perform text vectorization and classification, and generate classified knowledge data.

[0106] The knowledge storage and indexing module is used to store categorized knowledge data and build knowledge indexes based on the categorized knowledge data.

[0107] The knowledge update and maintenance module is used to filter the original knowledge data stored in the historical time period based on the search terms entered by the user and their historical search counts, and to determine the optimal original knowledge data.

[0108] In this embodiment, for the cleaned text data, the BERT attention mechanism combined with the TF-IDF algorithm is used to calculate word weights in order to extract the keyword set, specifically including:

[0109] Formula (1) is used to calculate the value of each term t in the cleaned text data. i The TF-IDF weights; the formula (1) is:

[0110]

[0111] Among them, TF·IDF(t) i ) is the term t i TF-IDF weights;

[0112] f i For the term t i The frequency of occurrence of terms in the cleaned text data directly affects the calculation of TF value, reflects the importance of terms in the document, provides the necessary data for calculating TF-IDF, and ensures the rationality of subsequent steps.

[0113] N represents the total number of cleaned text data in the corpus; wherein the corpus includes all raw knowledge data received within a historical time period; by considering the number of all documents in the corpus, the rarity of terms in the entire dataset can be effectively assessed, thereby improving the ability to select keywords.

[0114] n i For the term t i The amount of cleaned text data; in this embodiment, n i The definition reflects the term t i The number of documents in which a term appears can measure its prevalence, thus affecting the IDF value.

[0115] TTF represents the total number of occurrences of all terms in the cleaned text data. TFF can reflect the prevalence of a term in a document to a certain extent and is one of the key parameters when calculating TF-IDF, ensuring the rationality of term weight calculation.

[0116] This embodiment clearly demonstrates how to calculate the weight of each term using TF-IDF. This helps measure the importance of words in text, especially reducing the influence of common words and highlighting key content, which helps optimize subsequent text analysis and information retrieval. This embodiment clarifies the calculation results of TF-IDF to quantify the importance of each term in text, providing reliable data support for subsequent steps (such as classification, indexing, etc.).

[0117] Based on the pre-trained BERT model, the cleaned text data in the pre-processed text data is processed using a Transformer structure to obtain the attention matrix, and the attention matrix of each term t is calculated using formula (2). i The average attention score across all layers of the BERT model;

[0118] The formula (2) is:

[0119]

[0120] Among them, S i For the term t i The average attention score across all layers of the BERT model;

[0121] A h，i For the h-th layer of the BERT model, the term t i Attention score;

[0122] H represents the total number of layers in the BERT model;

[0123] This embodiment utilizes the attention mechanism of the BERT model to further enhance the extraction capability of keywords and semantic features from text, and uses average attention scores to evaluate the importance of terms. This operation provides a valid basis for further keyword screening and classification, optimizing the accuracy of keyword recognition. These steps, by introducing a combination of technologies such as BERT, TF-IDF, and LSTM-CRF, achieve efficient extraction of keywords and features from text data, contributing to the improvement of the intelligence and accuracy of the knowledge base management system, especially in terms of innovative effects in automatic classification, indexing, and recommendation.

[0124] The term t is calculated using formula (3). i Location information P in the cleaned text data i Normalization process yields term t i Position weight;

[0125] The formula (3) is:

[0126]

[0127] Among them, R i For the term t i Position weight; P i For the term t i The sum of the order of arrangement in the cleaned text data; L is the total number of terms in the cleaned text data;

[0128] The positional information of terms in the cleaned text data is calculated using formula (3) and then standardized to obtain the positional information weights of the terms. By considering the position of terms in the text, the accuracy of keyword extraction can be improved. Positional information, as an additional feature, helps to enhance the depth of text understanding, thus achieving higher efficiency in text analysis and information retrieval. Formula (3) clarifies how to calculate the positional information weights of terms. The position of a term is standardized relative to the total number of words L in the text using the formula for calculating positional weights. Standardization helps to balance the positional effect of each term in the text, thus ensuring that the contribution of position to text analysis is not biased due to different document lengths.

[0129] Based on term t i TF-IDF weights, term t i Average attention score and term t across all layers of the BERT model i Position weights are calculated for each term t using formula (4). i The final keyword weight W i ;

[0130] The formula (4) is:

[0131] W i =γ1×TF·IDF(t) i )+γ2×S i +γ3×R i ;

[0132] Where γ1 is the first weight adjustment parameter; γ2 is the second weight adjustment parameter; γ3 is the third weight adjustment parameter; γ1+γ2+γ3=1;

[0133] Based on the calculated comprehensive keyword weight W i Select the top K keywords with the highest keyword weights to form the final keyword set, and output the keyword set.

[0134] In this embodiment, TF-IDF is used to calculate the basic weight of each term, and the attention weight and position weight of each term in the BERT model are combined to obtain the final keyword weight. Through weighted fusion, this method can more accurately capture the most relevant keywords in the text. By using the attention and position information from both TF-IDF and BERT simultaneously, the importance of each term can be more comprehensively measured, enhancing the model's performance. By combining the contextual information from the BERT model and the statistical information from TF-IDF, the truly important keywords in the text can be better identified, rather than relying solely on a single method. By adjusting the weighting parameters (γ1, γ2, γ3), the sensitivity of the model can be adjusted according to different needs, optimizing the keyword extraction effect.

[0135] In this embodiment, the knowledge processing and classification module includes:

[0136] The text vectorization unit is used to receive preprocessed knowledge data and convert the cleaned text data into text vector data;

[0137] The classification unit is used to perform text classification based on text vector data and generate classification knowledge data.

[0138] The data output unit is used to output categorized knowledge data and transmit it to the knowledge storage and index building module.

[0139] In this embodiment, the input is assumed to be an article about artificial intelligence, such as "Application of Transformer in NLP". The system will perform preprocessing steps such as word segmentation, stop word removal, and stemming, and then convert it into numerical vectors (such as TF-IDF, Word2Vec, BERT embedding). This can be used for tasks such as news classification and automatic paper summarization. Based on the vectorized text data, machine learning or deep learning (such as SVM, CNN, LSTM, Transformer) is used to classify the text. For example, the article is automatically classified into the "Artificial Intelligence" category, rather than "Biotechnology" or "Blockchain". The classified knowledge data can be stored in a database for user queries. For example, when a user searches for "Natural Language Processing", the system can return all articles classified as "NLP" and make intelligent recommendations.

[0140] This embodiment utilizes text vectorization, enabling the system to quickly process large numbers of documents and improve classification efficiency. Combining NLP and machine learning reduces misjudgments in traditional keyword-matching-based methods, enhancing the intelligence level of text classification.

[0141] Specifically, the classification unit receives text vector data and performs text classification to obtain classification knowledge data, which includes:

[0142] The classification unit uses a preset support vector machine model to classify text vector data. After classification, each piece of text data is assigned corresponding classification knowledge data and the keyword set.

[0143] The preset support vector machine model is obtained by training text vector data of historical stages labeled with classification knowledge data tags.

[0144] The classification knowledge data includes classification labels and semantic vectors.

[0145] In this embodiment, by training models such as Support Vector Machines, text categories can be identified more accurately than traditional keyword matching. Combined with NLP technology, the system can automatically classify and store text, reducing manual processing workload. Through semantic vector representations (such as BERT and Word2Vec), the system can understand the deeper meaning of the text, rather than simply matching keywords.

[0146] In this embodiment, the knowledge storage and indexing construction module includes:

[0147] Storage unit, used to store classified knowledge data using a specified storage method;

[0148] The specified storage methods include: MongoDB and PostgreSQL;

[0149] Index building unit, used to build knowledge indexes based on categorized knowledge data;

[0150] The knowledge index includes an inverted index and an HNSW index;

[0151] The data output unit is used to output the knowledge index and transmit it to the knowledge update and maintenance module.

[0152] The following is a detailed description: The storage unit is used to store the categorized knowledge data, ensuring efficient data management and retrieval.

[0153] Specify storage method:

[0154] MongoDB: Characterized by its NoSQL-based document database, it is suitable for storing unstructured or semi-structured data, such as text and JSON data. Its advantages include fast query speed, high scalability, and support for complex queries and full-text search.

[0155] PostgreSQL: Characterized by being a SQL-based relational database, suitable for storing structured data and supporting ACID transaction processing. Its advantages include high data consistency, support for complex queries, full-text search, and extended features (such as JSONB storage).

[0156] The index building unit builds indexes based on categorized knowledge data, improving query efficiency.

[0157] Knowledge index type:

[0158] Inverted Index: The principle is to map keywords in a document to a list of documents containing those keywords, which is used to quickly query text data.

[0159] HNSW (Hierarchical Navigable Small World) index: The principle is an indexing method based on approximate nearest neighbor (ANN) search, which is suitable for high-dimensional data (such as vectors).

[0160] Applications: Semantic search, recommendation systems, knowledge graph queries.

[0161] The data output unit is used to output the constructed knowledge index and transmit it to the knowledge update and maintenance module to ensure the accessibility and maintainability of the data.

[0162] In this embodiment, MongoDB and PostgreSQL are used, which can simultaneously meet the needs of unstructured text storage and structured data management, balancing query efficiency and data consistency. Inverted indexes accelerate text keyword retrieval and are suitable for full-text search and knowledge retrieval. HNSW indexes support high-dimensional vector search, improving the accuracy and speed of semantic search and similarity matching.

[0163] In this embodiment, the knowledge update and maintenance module includes an event-driven update unit, which is used to calculate the relevance score S between any original knowledge data stored in the historical time period and the search term q based on the search term q input by the user and the historical search count of the search term q.

[0164] Based on the relevance score S, the optimal raw knowledge data is selected from the raw knowledge data received within the historical time period.

[0165] The optimal raw knowledge data is the raw knowledge data corresponding to the highest relevance score S.

[0166] In this embodiment, the matching degree between the original knowledge data and the search term q is calculated using a relevance score S. This ensures that the most relevant knowledge is returned first when a user queries, avoiding interference from irrelevant content. Using historical search frequency as a weight makes popular search results easier to retrieve, allowing users to find valuable information faster and reducing unnecessary search costs.

[0167] Specifically, by using the search term q entered by the user and the historical search count of that search term q, the relevance score S between any original knowledge data stored in the historical time period and the search term q is calculated using formula (5);

[0168] The formula (5) is:

[0169]

[0170] S is the relevance score between the search term q and any original knowledge data; f qThe number of times the search term q appears in any original knowledge data; M is the total number of original knowledge data stored within the historical time period; n q T represents the amount of raw knowledge data containing the search term q stored within a historical time period; q α represents the total number of searches for the search term q within the historical time period; T represents the total number of searches for all keywords; and α represents a pre-set balancing factor.

[0171] In this embodiment, the relevance score S is calculated using formula (5), which quantifies the degree of matching between the search term q and the knowledge data. This allows the system to prioritize returning the most relevant knowledge data, thereby improving the accuracy of the search results. Formula (5) uses: f q (Number of occurrences of search term q in the original knowledge data), ensuring that high-frequency matching data is given priority. q (The amount of original knowledge data containing search term q) avoids bias caused by a certain knowledge data containing q multiple times, enhancing generalization ability. α can adjust the impact of search popularity on relevance scores, preventing certain accidental high-frequency searches from affecting knowledge ranking and enhancing system stability.

[0172] The original knowledge data can be in any of the following formats: Word, PDF, TXT, or HTML. This embodiment supports multiple formats such as Word, PDF, TXT, and HTML, enabling the knowledge base to process text data from different sources and improving the system's scalability and applicability. This compatibility ensures that the knowledge base can automatically parse and learn knowledge data in different formats, enhancing the system's intelligence.

[0173] Where 0 < α < 1. The dynamic parameter α is designed to ensure that the knowledge base can maintain the stability of knowledge while adapting to the needs of knowledge updates, thereby improving intelligence and controllability.

[0174] Example 4

[0175] See Figure 2 This embodiment also provides a knowledge base management method based on NLP and machine learning, including:

[0176] S1. Receive raw knowledge data and perform text preprocessing to generate preprocessed knowledge data;

[0177] S2. Perform text vectorization and classification on the preprocessed knowledge data to generate categorized knowledge data;

[0178] S3. Store categorized knowledge data and build a knowledge index based on the categorized knowledge data;

[0179] S4. Based on the user's input search terms and their historical search counts, filter the raw knowledge data stored within the historical time period to determine the optimal raw knowledge data.

[0180] In this embodiment, S1 specifically includes:

[0181] S11. Data Input: Receive raw knowledge data, including user-uploaded documents, database records, or other text data;

[0182] S12. Preprocessing: Perform preprocessing on the original knowledge data, the preprocessing including:

[0183] Remove HTML tags, advertisements, garbled characters, and duplicate text to obtain cleaned text data;

[0184] An NLP engine is used to parse the content and extract the set of keywords from the document;

[0185] Generate document summaries for quick understanding of document content;

[0186] Using machine learning models for sentiment analysis to identify sentiment features in text;

[0187] In this embodiment, removing HTML tags, advertisements, garbled text, and duplicate text reduces noise, improves data quality, and makes the knowledge base more refined. The NLP engine performs content parsing, extracting keywords to make knowledge more searchable and improve the accuracy of subsequent classification and indexing. Automatically generating document summaries helps users quickly understand documents and improves information utilization efficiency. Using machine learning for sentiment analysis analyzes the text's emotional tendency, providing users with more intuitive knowledge classifications (such as positive, negative, and neutral). Data preprocessing is a key step in NLP tasks, improving data clarity and consistency, laying a solid foundation for subsequent classification, indexing, and retrieval. NLP processing techniques such as keyword extraction, summary generation, and sentiment analysis effectively reduce redundant information, making knowledge more structured and facilitating classification and retrieval.

[0188] S13. Data Output: Output preprocessed knowledge data and provide it to the knowledge processing and classification steps.

[0189] In this embodiment, S2 specifically includes:

[0190] S21. Machine Learning Classification: Applying supervised learning, unsupervised learning, or semi-supervised learning algorithms to automatically classify preprocessed knowledge data;

[0191] S22. Topic Recognition: Based on the content and characteristics of a document, it is categorized into a predefined category or a new category is generated;

[0192] S23. Index Building: Create an index for the categorized documents to support fast retrieval and output the categorization and index results.

[0193] In this embodiment, supervised, unsupervised, or semi-supervised learning algorithms are employed to automatically identify and classify knowledge data, reducing manual intervention and improving classification efficiency and accuracy. Topic recognition can automatically categorize knowledge data and even discover new knowledge categories, making the knowledge base structure more dynamic and intelligent. Index building improves retrieval efficiency, supports rapid querying of massive amounts of data, and enhances user experience. Machine learning's classification capabilities far surpass traditional rule-based classification methods, adapting to constantly changing knowledge systems and achieving adaptive classification. Topic recognition, combined with NLP models such as LDA, BERT, or Transformer, can automatically identify knowledge domains and enhance semantic understanding of knowledge. Establishing efficient indexes, such as inverted indexes or vector-based ANN (Approximate Nearest Neighbors) retrieval, makes queries faster and more accurate.

[0194] In this embodiment, S3 specifically includes:

[0195] S31. Distributed database storage: Stores data from NLP engines and machine learning classifiers, supporting structured, semi-structured, and unstructured data management;

[0196] S32. Intelligent Index: Based on the content of documents and user behavior data, a multi-dimensional index structure is built to support fast retrieval and improve the efficiency of users finding information.

[0197] S33. Personalized Recommendations: Push relevant content based on users' historical behavior and preferences.

[0198] In this embodiment, the distributed database storage supports structured, semi-structured, and unstructured data management, ensuring the scalability of the knowledge base and adapting to large-scale data growth. The intelligent indexing system combines NLP with user behavior data to build multi-dimensional indexes, improving query efficiency and reducing knowledge retrieval latency. Personalized recommendations proactively push relevant knowledge based on user interests and behavioral history, improving knowledge utilization.

[0199] Utilizing distributed databases (such as Elasticsearch, MongoDB, and HBase) supports petabyte-scale data storage, ensuring system scalability and high availability. Intelligent indexing, combined with TF-IDF, Word2Vec, and BERT semantic vectors, optimizes search results and improves the accuracy of user queries. Personalized recommendations, leveraging collaborative filtering and deep learning (such as Transformer), dynamically adjust recommended content to meet individual user needs.

[0200] In this embodiment, S4 specifically includes:

[0201] S41. Event-driven update: Connects to business systems and external data sources in real time via API interface to monitor data change events and automatically trigger the knowledge base update process;

[0202] S42. User Feedback Processing: Provides functions such as user evaluation, annotation, and error correction, collects and analyzes user feedback to optimize the content and structure of the knowledge base;

[0203] S43 Continuous Monitoring and Optimization: Set knowledge base health indicators, including access volume, update frequency, and user satisfaction, regularly evaluate the performance of the knowledge base, and adjust management strategies based on the evaluation results.

[0204] In this embodiment, event-driven updates can monitor external data changes in real time, ensuring the knowledge base remains up-to-date. The user feedback system allows users to evaluate, annotate, and correct knowledge content, improving knowledge quality and making the knowledge base more aligned with user needs. Continuous monitoring and optimization, through metrics such as access volume, update frequency, and user satisfaction, periodically optimize the knowledge base to improve overall system performance. Event-driven models (such as Kafka and RabbitMQ) enable real-time data updates, preventing information lag in the knowledge base. The user feedback system, combined with NLP's active learning mechanism, allows the model to continuously learn from user feedback, thereby improving the accuracy of knowledge classification and retrieval.

[0205] In the description of this invention, it should be understood that the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0206] In this invention, unless otherwise explicitly specified and limited, the terms "installation," "connection," "linking," and "fixing," etc., should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral part; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; they can refer to the internal communication of two components or the interaction between two components. Those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances.

[0207] In this invention, unless otherwise explicitly specified and limited, "above" or "below" the second feature can mean that the first and second features are in direct contact, or that they are in indirect contact through an intermediate medium. Furthermore, "above," "over," or "on top" the second feature can mean that the first feature is directly above or diagonally above the second feature, or simply indicates that the first feature is at a higher horizontal level than the second feature. "Below," "below," or "beneath" the second feature can mean that the first feature is directly below or diagonally below the second feature, or simply indicates that the first feature is at a lower horizontal level than the second feature.

[0208] In the description of this specification, the terms "one embodiment," "some embodiments," "embodiment," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0209] Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make modifications, alterations, substitutions and variations to the above embodiments within the scope of the present invention.

Claims

1. A knowledge base management system based on NLP and machine learning, characterized in that, The system includes: The knowledge acquisition module is used to receive raw knowledge data and perform text preprocessing to generate preprocessed knowledge data. The knowledge processing and classification module is used to receive preprocessed knowledge data, perform text vectorization and classification, and generate classified knowledge data. The knowledge storage and indexing module is used to store categorized knowledge data and build knowledge indexes based on the categorized knowledge data. The knowledge update and maintenance module is used to filter the original knowledge data stored in the historical time period based on the search terms input by the user and their historical search counts, and to determine the optimal original knowledge data. The knowledge update and maintenance module includes an event-driven update unit, which is used to calculate the relevance score S between any original knowledge data stored in the historical time period and the search term q using formula (5) based on the search term q input by the user and the historical search count of the search term q. The formula (5) is: S is the relevance score between the search term q and any original knowledge data; f q The number of times the search term q appears in any original knowledge data; M is the total number of original knowledge data stored within the historical time period; n q T represents the amount of raw knowledge data containing the search term q stored within a historical time period; q α represents the total number of searches for the search term q within the historical time period; T represents the total number of searches for all keywords; α is a pre-set balancing factor. Based on the relevance score S, the optimal raw knowledge data is selected from the raw knowledge data received within the historical time period; the optimal raw knowledge data is the raw knowledge data corresponding to the highest relevance score S.

2. The knowledge base management system based on NLP and machine learning according to claim 1, characterized in that, The knowledge acquisition module includes: A data input unit is used to receive raw knowledge data, including documents uploaded by users; A data preprocessing unit is used to perform preprocessing on raw knowledge data, the preprocessing including: Remove HTML tags, advertisements, garbled characters, and duplicate text to obtain cleaned text data; For the cleaned text data, the BERT attention mechanism combined with the TF-IDF algorithm is used to calculate word weights in order to extract the keyword set; The LSTM-CRF model was used to extract sentiment features from the cleaned text data and generate sentiment score data. The cleaned text data is processed based on the BART pre-trained model to generate a text summary and output the summary data. The data output unit is used to output preprocessed knowledge data and provide it to the knowledge processing and classification module.

3. The knowledge base management system based on NLP and machine learning according to claim 2, characterized in that, For the cleaned text data, the BERT attention mechanism combined with the TF-IDF algorithm is used to calculate word weights in order to extract the keyword set, specifically including: Formula (1) is used to calculate the value of each term t in the cleaned text data. i The TF-IDF weights; the formula (1) is: Among them, TF·IDF(t) i ) is the term t i TF-IDF weights; f i For term t i The number of times it appears in the cleaned text data; N represents the total number of cleaned text data in the corpus; wherein the corpus includes all original knowledge data received within a historical time period. n i For the term t i The amount of cleaned text data; TTF represents the total number of occurrences of all terms in the cleaned text data; Based on the pre-trained BERT model, the cleaned text data in the pre-processed text data is processed using a Transformer structure to obtain the attention matrix, and the attention matrix of each term t is calculated using formula (2). i The average attention score across all layers of the BERT model; The formula (2) is: Among them, S i For term t i The average attention score across all layers of the BERT model; A h,i For the h-th layer of the BERT model, the term t i Attention score; H represents the total number of layers in the BERT model; The term t is calculated using formula (3). i Location information P in the cleaned text data i Normalization process yields term t i Position weight; The formula (3) is: Among them, R i For term t i Position weight; P i For term t i The sum of the order of arrangement in the cleaned text data; L is the total number of terms in the cleaned text data; Based on term t i TF-IDF weights, term t i Average attention score and term t across all layers of the BERT model i Position weights are calculated for each term t using formula (4). i The final keyword weight W i ; The formula (4) is: W i =γ1×TF·IDF(t i )+γ2×S i +γ3×R i ; Where γ1 is the first weight adjustment parameter; γ2 is the second weight adjustment parameter; γ3 is the third weight adjustment parameter; γ1+γ2+γ3=1; Based on the calculated comprehensive keyword weight W i Select the top K keywords with the highest keyword weights to form the final keyword set, and output the keyword set.

4. The knowledge base management system based on NLP and machine learning according to claim 3, characterized in that, The knowledge processing and classification module includes: The text vectorization unit is used to receive preprocessed knowledge data and convert the cleaned text data into text vector data; The classification unit is used to perform text classification based on text vector data and generate classification knowledge data. The data output unit is used to output categorized knowledge data and transmit it to the knowledge storage and index building module.

5. The knowledge base management system based on NLP and machine learning according to claim 4, characterized in that, The classification unit receives text vector data and performs text classification to obtain classification knowledge data, specifically including: The classification unit uses a preset support vector machine model to classify text vector data. After classification, each piece of text data is assigned corresponding classification knowledge data and the keyword set. The preset support vector machine model is obtained by training text vector data of historical stages labeled with classification knowledge data tags. The classification knowledge data includes classification labels and semantic vectors.

6. The knowledge base management system based on NLP and machine learning according to claim 5, characterized in that, The knowledge storage and indexing construction module includes: Storage unit, used to store classified knowledge data using a specified storage method; The specified storage methods include: MongoDB and PostgreSQL; Index building unit, used to build knowledge indexes based on categorized knowledge data; The knowledge index includes an inverted index and an HNSW index; The data output unit is used to output the knowledge index and transmit it to the knowledge update and maintenance module.

7. The knowledge base management system based on NLP and machine learning according to claim 6, characterized in that, The original knowledge data can be in any of the following formats: Word, PDF, TXT, or HTML.

8. The knowledge base management system based on NLP and machine learning according to claim 7, characterized in that, in, 0＜α＜1。

Images