Knowledge base intelligent analysis method and system

Abstract

The invention relates to the technical field of artificial intelligence, in particular to a knowledge base intelligent analysis method and system.The method comprises the steps that firstly, data in a service system is regularly extracted to form an internal knowledge base on the basis of a Streammable Http protocol, the data are preprocessed, then model calling, model fine tuning, vectorization processing and adoption of a self-adaptive fine tuning algorithm are provided, and then the knowledge base intelligent analysis result is obtained; and carrying out knowledge base retrieval, carrying out unified management on a model algorithm, and finally, recommending a recommendation result to a final user in a visual mode for viewing and selecting. Compared with the prior art, the method has the advantages that excellent expandability can be shown in practical application, sharing of domain knowledge is powerfully promoted, and digital transformation of enterprises is promoted.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and specifically provides a knowledge base intelligent analysis method and system. Background Art

[0002] With the advancement of global economic integration, AI knowledge bases built using LLMs have become a key strategy for intelligent decision-making analysis. Traditional data analysis models (such as regression analysis and clustering) struggle with unstructured data (such as policy documents and corporate annual reports). They rely on manual feature engineering, resulting in inefficiencies and incomplete information. In recent years, the development of large-scale model technology has provided new solutions to these problems. However, effectively expanding the capabilities of large models and integrating them with business data for intelligent analysis remains a challenge.

[0003] General-purpose large-scale models have knowledge blind spots in vertical fields, and direct application can lead to misjudgment of industry terminology. Existing information systems often rely on rule engines for decision analysis, which cannot dynamically adapt to regional policy changes and fluctuations in the industrial chain. By incorporating a large amount of business data into the pre-training phase, such as policy documents from different regions, annual reports from various companies, and a rich industry knowledge graph, we address the problem that collaborative filtering algorithms applied to the model cannot fully consider the total amount of upstream and downstream connections in the industrial chain. This enables the model to deeply understand the specific concepts and logical relationships in domain knowledge, such as investment promotion, and reduce misjudgments caused by knowledge blind spots.

[0004] How to combine large model capabilities with new external tools to improve the efficiency and accuracy of data analysis is an urgent problem to be solved by technicians in this field. Summary of the Invention

[0005] The present invention aims to address the above-mentioned deficiencies in the prior art and provides a knowledge base intelligent analysis method with strong practicality.

[0006] A further technical task of the present invention is to provide a knowledge base intelligent analysis system that is rationally designed, safe and applicable.

[0007] The technical solution adopted by the present invention to solve its technical problem is:

[0008] A knowledge base intelligent analysis method based on the Streamable Http protocol. First, it regularly extracts data from the business system to form an internal knowledge base and preprocesses the data. Then, it provides model calling, model fine-tuning, vectorization processing, and adopts an adaptive fine-tuning algorithm. After that, it performs knowledge base retrieval and unified management of model algorithms. Finally, the recommendation results are presented to the end user in a visual manner for viewing and selection.

[0009] Furthermore, in the data processing stage, according to the definition in the data quality rule base, data processing technology is applied to extract, transform and load the data in sequence to achieve missing value processing, duplicate data processing, outlier processing, format standardization, consistency checking, logical error correction and incompleteness checking operations on the data set;

[0010] The format standardization uses the BiLSTM-CRF model to identify industrial entities, and calls the Embedding model to format the cleaned data into a text format suitable for LLM processing, and then constructs a multi-dimensional weighted industry correlation degree of 0-1, policy intensity of 0-5 and corporate credit rating of AAA-D.

[0011] Furthermore, after fine-tuning the model, a model call service is provided to perform vectorized data processing, and the model is lightly fine-tuned based on LoRA. The model parameters are automatically adjusted according to the characteristics of the business data and the actual operation effect of the model.

[0012] In terms of vectorization processing, multi-vector fusion technology is introduced to inject and fuse domain vector knowledge from different sources and dimensions into the model, so that the data contains richer semantic information when entering the LLM knowledge base.

[0013] Furthermore, when performing knowledge base retrieval, domain knowledge is first extracted from the data, the extracted domain knowledge is represented in a format understandable by LLM, and stored in the knowledge base, and the knowledge base is updated regularly.

[0014] Furthermore, when unified management of model algorithms is performed, LLM is used to perform text classification and clustering of business data, discover potential patterns and associations in the data, perform sentiment analysis and opinion mining on related texts, understand external trends in the field, and combine domain knowledge and intelligent analysis algorithms to predict risks and identify opportunities in the data.

[0015] Furthermore, when the user feedback matching error is greater than the threshold, model retraining is triggered and knowledge versioning management is implemented. Through interaction with users, their domain knowledge needs and preferences are understood. Combined with user demand retrieval and intelligent analysis results, personalized recommendation algorithms are designed and the recommendation results are recommended to end users in a visual manner.

[0016] A knowledge base intelligent analysis system, based on the Streamable Http protocol, first regularly extracts data from the business system to form an internal knowledge base and preprocesses the data. It then provides model calling, model fine-tuning, vectorization processing, and an adaptive fine-tuning algorithm. After that, it performs knowledge base retrieval and uniformly manages the model algorithm. Finally, when the user feedback matching error exceeds the threshold, it triggers model retraining and implements knowledge versioning management. Through interaction with users, it understands their domain knowledge needs and preferences. Based on user demand retrieval and intelligent analysis results, it designs a personalized recommendation algorithm and presents the recommendation results to end users in a visual manner.

[0017] Furthermore, in the data processing stage, according to the definition in the data quality rule base, data processing technology is applied to extract, transform and load the data in sequence to achieve missing value processing, duplicate data processing, outlier processing, format standardization, consistency checking, logical error correction and incompleteness checking operations on the data set;

[0018] The format standardization uses the BiLSTM-CRF model to identify industry entities, and calls the Embedding model to format the cleaned data into a text format suitable for LLM processing, and then constructs a multi-dimensional weighted industry correlation degree of 0-1, a strategy strength of 0-5, and a corporate credit rating of AAA-D.

[0019] Furthermore, after fine-tuning the model, a model call service is provided to perform vectorized data processing, and the model is lightly fine-tuned based on LoRA. The model parameters are automatically adjusted according to the characteristics of the business data and the actual operation effect of the model.

[0020] In terms of vectorization processing, multi-vector fusion technology is introduced to inject and fuse domain vector knowledge from different sources and dimensions into the model, so that the data contains richer semantic information when entering the LLM knowledge base.

[0021] Furthermore, when unified management of model algorithms is performed, LLM is used to perform text classification and clustering of business data, discover potential patterns and associations in the data, perform sentiment analysis and opinion mining on related texts, understand external trends in the field, and combine domain knowledge and intelligent analysis algorithms to predict risks and identify opportunities in the data.

[0022] Compared with the prior art, the knowledge base intelligent analysis method and system of the present invention have the following outstanding beneficial effects:

[0023] The knowledge base intelligent analysis model based on the Streamable Http protocol in this invention has demonstrated excellent scalability in practical applications, which will effectively promote the sharing of domain knowledge and facilitate the digital transformation of enterprises. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0025] Attachment Figure 1 It is a flowchart of a knowledge base intelligent analysis method. DETAILED DESCRIPTION

[0026] In order to enable those skilled in the art to better understand the solutions of the present invention, the present invention will be further described in detail below in conjunction with specific embodiments. Obviously, the embodiments described are only some embodiments of the present invention, rather than all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0027] A best embodiment is given below:

[0028] like Figure 1 As shown, this embodiment of a knowledge base intelligent analysis method first regularly extracts data from business systems to form an internal knowledge base. For information from channels such as industry websites, news websites, market research institutions, and investment promotion policies, due to their high real-time nature and low data security, the MCP client calls server-side services in real time during retrieval, and implements crawler technology (FireCrawl) to capture the latest industry information in real time.

[0029] Data preprocessing: In the data processing stage, according to the definition in the data quality rule base, data processing technology is applied to extract (Extract), transform (Transform) and load (Load) the data in sequence to implement missing value processing, duplicate data processing, outlier processing, format standardization, consistency checking, logical error correction and incompleteness checking of the data set.

[0030] Among them, format standardization uses the BiLSTM-CRF model to identify industrial entities, and calls the Embedding model to format the cleaned data into a text format suitable for LLM processing, and then constructs multi-dimensional weighted industry correlation (0-1), strategy strength (0-5 levels), corporate credit rating (AAA-D), etc.

[0031] Provides model calling, model fine-tuning, vectorized processing, and adopts adaptive fine-tuning algorithm.

[0032] After fine-tuning the model, a model call service is provided for data vectorization. Based on LoRA, the model is lightweight and fine-tuned, automatically adjusting model parameters based on the characteristics of the business data and the actual performance of the model. Regarding vectorization, in addition to the traditional embedding method, multi-vector fusion technology is introduced to inject and fuse domain vector knowledge from different sources and dimensions into the model, ensuring that the data contains richer semantic information when entering the LLM knowledge base.

[0033] Specific approaches include clause embedding vectorization (Policy2Vec) and providing an industry chain relationship constraint loss function.

[0034] Extract domain knowledge from business domain databases, expert knowledge bases, authoritative literature, industry reports, and other data. Express the extracted domain knowledge in a format understandable to LLM and store it in the knowledge base. Update the knowledge base regularly to ensure the timeliness and accuracy of the knowledge.

[0035] Unified management of model algorithms, through the application of hybrid model architecture, to achieve semantic feature extraction, numerical feature extraction and knowledge graph reasoning. The process includes:

[0036] LLM is used to classify and cluster business data to identify potential patterns and relationships within the data. Sentiment analysis and opinion mining are also conducted on relevant approval and review texts to understand external trends in the field.

[0037] Combine domain knowledge and intelligent analysis algorithms to predict risks and identify opportunities in data.

[0038] When the user feedback matching error is greater than the threshold (initial experience value 15%), model retraining is triggered, and knowledge version management is implemented, automatically triggering the associated knowledge base to update and index automatically based on the changes.

[0039] Through interaction with users, we understand their domain knowledge needs and preferences. We combine user needs retrieval and intelligent analysis results to design personalized recommendation algorithms. Recommendation results are presented to end users in a visual format for easy viewing and selection.

[0040] Based on the above method, a knowledge base intelligent analysis system in this embodiment is based on the Streamable Http protocol. First, it regularly extracts data from the business system to form an internal knowledge base and preprocesses the data. Then, it provides model calling, model fine-tuning, vectorization processing and adopts an adaptive fine-tuning algorithm. After that, it performs knowledge base retrieval and uniformly manages the model algorithm. Finally, when the user feedback matching error is greater than the threshold, it triggers model retraining and implements knowledge versioning management. Through interaction with users, it understands their domain knowledge needs and preferences, combines user demand retrieval and intelligent analysis results, designs personalized recommendation algorithms, and recommends the recommendation results to end users in a visual manner.

[0041] In the data processing stage, according to the definition in the data quality rule base, data processing technology is applied to extract, transform and load the data in sequence to achieve missing value processing, duplicate data processing, outlier processing, format standardization, consistency checking, logical error correction and incompleteness checking operations on the data set;

[0042] The format standardization uses the BiLSTM-CRF model to identify industry entities, and calls the Embedding model to format the cleaned data into a text format suitable for LLM processing, and then constructs a multi-dimensional weighted industry correlation degree of 0-1, a strategy strength of 0-5, and a corporate credit rating of AAA-D.

[0043] After fine-tuning the model, we provide a model call service to perform vectorized data processing, perform lightweight fine-tuning on the model based on LoRA, and automatically adjust the model parameters according to the characteristics of the business data and the actual operation effect of the model.

[0044] In terms of vectorization processing, multi-vector fusion technology is introduced to inject and fuse domain vector knowledge from different sources and dimensions into the model, so that the data contains richer semantic information when entering the LLM knowledge base.

[0045] When managing model algorithms in a unified manner, LLM is used to perform text classification and clustering on business data, discover potential patterns and associations in the data, perform sentiment analysis and opinion mining on related texts, understand external trends in the field, and combine domain knowledge and intelligent analysis algorithms to predict risks and identify opportunities in the data.

[0046] The above-mentioned specific implementation methods are only specific cases of the present invention. The patent protection scope of the present invention includes but is not limited to the above-mentioned specific implementation methods. Any technical solutions that conform to the above-mentioned specific implementation methods of the present invention and any appropriate changes or substitutions made thereto by ordinary technicians in the relevant technical field shall fall within the patent protection scope of the present invention.

[0047] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to these embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the appended claims and their equivalents.

Claims

1. A knowledge base intelligent analysis method, characterized in that: Based on the Streamable Http protocol, first, data from the business system is regularly extracted to form an internal knowledge base and preprocessed. Then, model calling, model fine-tuning, vectorization processing, and adaptive fine-tuning algorithms are provided. After that, knowledge base retrieval is performed and model algorithms are uniformly managed. Finally, the recommendation results are visually recommended to end users for viewing and selection.

2. A knowledge base intelligent analysis method according to claim 1, characterized in that: In the data processing stage, according to the definition in the data quality rule base, data processing technology is applied to extract, transform and load the data in sequence to achieve missing value processing, duplicate data processing, outlier processing, format standardization, consistency checking, logical error correction and incompleteness checking operations on the data set; The format standardization uses the BiLSTM-CRF model to identify industry entities, and calls the Embedding model to format the cleaned data into a text format suitable for LLM processing, and then constructs a multi-dimensional weighted industry correlation degree of 0-1, a strategy strength of 0-5, and a corporate credit rating of AAA-D.

3. A knowledge base intelligent analysis method according to claim 2, characterized in that: After fine-tuning the model, we provide a model call service to perform vectorized data processing, perform lightweight fine-tuning on the model based on LoRA, and automatically adjust the model parameters according to the characteristics of the business data and the actual operation effect of the model. In terms of vectorization processing, multi-vector fusion technology is introduced to inject and fuse domain vector knowledge from different sources and dimensions into the model, so that the data contains richer semantic information when entering the LLM knowledge base.

4. A knowledge base intelligent analysis method according to claim 3, characterized in that: When performing knowledge base retrieval, domain knowledge is first extracted from the data, the extracted domain knowledge is represented in a format understandable to LLM, and stored in the knowledge base, and the knowledge base is updated regularly.

5. A knowledge base intelligent analysis method according to claim 4, characterized in that: When managing model algorithms in a unified manner, LLM is used to perform text classification and clustering on business data, discover potential patterns and associations in the data, perform sentiment analysis and opinion mining on related texts, understand external trends in the field, and combine domain knowledge and intelligent analysis algorithms to predict risks and identify opportunities in the data.

6. A knowledge base intelligent analysis method according to claim 5, characterized in that: When the user feedback matching error is greater than the threshold, model retraining is triggered and knowledge versioning management is implemented. Through interaction with users, their domain knowledge needs and preferences are understood. Combined with user demand retrieval and intelligent analysis results, personalized recommendation algorithms are designed and the recommendation results are recommended to end users in a visual manner.

7. A knowledge base intelligent analysis system, characterized in that: Based on the Streamable Http protocol, first, data from the business system is regularly extracted to form an internal knowledge base and preprocessed. Then, model calling, model fine-tuning, vectorization processing, and adaptive fine-tuning algorithms are provided. After that, knowledge base retrieval is performed and model algorithms are uniformly managed. Finally, when user feedback matching error exceeds the threshold, model retraining is triggered and knowledge versioning management is implemented. Through interaction with users, their domain knowledge needs and preferences are understood. Combined with user demand retrieval and intelligent analysis results, personalized recommendation algorithms are designed and recommendation results are presented to end users in a visual manner.

8. A knowledge base intelligent analysis system according to claim 7, characterized in that: In the data processing stage, according to the definition in the data quality rule base, data processing technology is applied to extract, transform and load the data in sequence to achieve missing value processing, duplicate data processing, outlier processing, format standardization, consistency checking, logical error correction and incompleteness checking operations on the data set; The format standardization uses the BiLSTM-CRF model to identify industry entities, and calls the Embedding model to format the cleaned data into a text format suitable for LLM processing, and then constructs a multi-dimensional weighted industry correlation degree of 0-1, a strategy strength of 0-5, and a corporate credit rating of AAA-D.

9. A knowledge base intelligent analysis system according to claim 8, characterized in that: After fine-tuning the model, we provide a model call service to perform vectorized data processing, perform lightweight fine-tuning on the model based on LoRA, and automatically adjust the model parameters according to the characteristics of the business data and the actual operation effect of the model. In terms of vectorization processing, multi-vector fusion technology is introduced to inject and fuse domain vector knowledge from different sources and dimensions into the model, so that the data contains richer semantic information when entering the LLM knowledge base.

10. A knowledge base intelligent analysis system according to claim 9, characterized in that: When managing model algorithms in a unified manner, LLM is used to perform text classification and clustering on business data, discover potential patterns and associations in the data, perform sentiment analysis and opinion mining on related texts, understand external trends in the field, and combine domain knowledge and intelligent analysis algorithms to predict risks and identify opportunities in the data.

Images