Knowledge base information processing method and device, storage medium and computer equipment

Abstract

The invention discloses a knowledge base information processing method and device, a storage medium and computer equipment, relates to the technical field of information processing, and mainly aims at solving the problems that an existing literature retrieval method is low in efficiency, poor in retrieval result correlation and low in accuracy. Obtaining standard text data; building a journal paper knowledge base based on the standard text data; performing vector representation processing on the standard text data, and constructing a mixed index corresponding to the standard text data based on the obtained knowledge vector; receiving a query text of a user, and rewriting the query text based on a preset hybrid retrieval strategy to obtain a target query text; and determining target journal paper data corresponding to the target query text from the journal paper knowledge base based on a pre-trained retrieval enhancement generation model and the mixed index, and generating a reply text based on the pre-trained retrieval enhancement generation model and the target journal paper data.

Description

Technical Field

[0001] The present invention relates to the field of information processing technology, and in particular to a method and device for knowledge base information processing, a storage medium, and a computer device. Background Art

[0002] In the current information age, academic research has become an important driving force for scientific and technological progress and innovation. As the main carrier of academic research results, the number and quality of journal articles have increased year by year, covering the latest discoveries and theories in various fields. With the explosive growth of information, journal article data can be used to serve academic research through literature retrieval.

[0003] Traditional literature retrieval methods, such as keyword-based searches, are often limited by the query conditions provided by users and may not be able to fully capture relevant knowledge. Moreover, keyword-based retrieval methods usually only perform exact matches and often ignore the semantic associations and contextual relationships between papers. Since academic papers usually contain rich concepts and arguments, relying solely on keywords often fails to capture these deep-level associations. Therefore, the returned search results may only be superficially related to the query keywords, but lack substantive relevance, causing users to browse a large number of potentially irrelevant or incompletely relevant documents, greatly reducing retrieval efficiency. How to quickly and accurately obtain relevant information from a large amount of journal article data has become an urgent problem to be solved. Summary of the invention

[0004] In view of this, the present invention provides a method and apparatus for knowledge base information processing, a storage medium, and a computer device, the main purpose of which is to solve the problems of low efficiency, poor relevance and low accuracy of existing document retrieval methods.

[0005] According to one aspect of the present invention, a method for processing knowledge base information is provided, comprising:

[0006] Collecting journal article data, standardizing the journal article data to obtain standard text data; and constructing a journal article knowledge base based on the standard text data;

[0007] Performing vector representation processing on the standard text data in the journal article knowledge base, and constructing a mixed index corresponding to the standard text data based on the obtained knowledge vector;

[0008] Receiving a user's query text, and rewriting the query text based on a preset hybrid search strategy to obtain a target query text;

[0009] Based on the pre-trained retrieval enhancement generation model and the hybrid index, target journal article data corresponding to the target query text is determined from the journal article knowledge base, and a reply text is generated based on the pre-trained retrieval enhancement generation model and the target journal article data.

[0010] Furthermore, the standardization of the journal article data includes:

[0011] Performing text standardization on the journal articles, including unified character encoding, formatted text and standardized symbols;

[0012] The metadata of the journal articles are standardized, including unified field processing and unified format processing.

[0013] Furthermore, the vector representation processing is performed on the standard text data in the journal article knowledge base, and a mixed index corresponding to the standard text data is constructed based on the obtained knowledge vector, including:

[0014] Using a pre-trained dense vector model to extract semantic information from the standard text data to obtain a corresponding dense vector representation;

[0015] Using a pre-trained sparse vector model to extract information expressions of keyword analysis and word frequency statistics in the standard text data, and obtain corresponding sparse vector representations;

[0016] The dense vector representation and the sparse vector representation are combined to obtain the knowledge vector; and a mixed index is constructed between the knowledge vector and the standard text data.

[0017] Furthermore, the query text is rewritten based on a preset hybrid search strategy to obtain a target query text, including:

[0018] Performing query analysis on the query text to obtain keywords and query intent corresponding to the query text;

[0019] At least one rewriting method is selected from synonym expansion processing, query completion processing, query enhancement processing and context-aware processing, and the query text is rewritten based on the keywords and the query intent to obtain the target query text.

[0020] Furthermore, the retrieval enhancement generation model includes a retrieval module and a generation module; and the process of pre-training the retrieval enhancement generation model includes:

[0021] Extracting part of the standard text data from the journal article knowledge base as a training data set; and using the training data set to train the retrieval module to obtain a trained retrieval module;

[0022] The generation module is trained based on the search result of the search module to obtain a trained generation module.

[0023] Further, the method of determining target journal article data corresponding to the target query text from the journal article knowledge base based on the pre-trained retrieval enhancement generation model and the hybrid index, and generating a reply text based on the pre-trained retrieval enhancement generation model and the target journal article data, comprises:

[0024] Converting the target query text into a query vector, and determining the target knowledge vector with the highest similarity from the journal article knowledge base based on the trained retrieval module;

[0025] Retrieving the target journal article data most relevant to the target query text from the mixed index based on the target knowledge vector;

[0026] A reply text corresponding to the target query text is generated based on the trained generation module and the target journal article data.

[0027] Furthermore, the method further comprises:

[0028] Collect user feedback information, and classify and process the user feedback information to obtain user feedback problem types;

[0029] Based on the user feedback question type, the pre-trained retrieval enhancement generation model is subjected to model optimization processing, including at least one of data enhancement processing, algorithm adjustment processing, model fine-tuning processing, and feature adjustment processing.

[0030] According to another aspect of the present invention, there is provided a device for processing knowledge base information, comprising:

[0031] A knowledge base construction module is used to collect journal article data, standardize the journal article data to obtain standard text data, and construct a journal article knowledge base based on the standard text data;

[0032] An index building module, used for performing vector representation processing on the standard text data in the journal article knowledge base, and building a mixed index corresponding to the standard text data based on the obtained knowledge vector;

[0033] A receiving and rewriting module, used for receiving a user's query text, and rewriting the query text based on a preset hybrid search strategy to obtain a target query text;

[0034] A retrieval and generation module is used to determine the target journal paper data corresponding to the target query text from the journal paper knowledge base based on the pre-trained retrieval enhancement generation model and the hybrid index, and to generate a reply text based on the pre-trained retrieval enhancement generation model and the target journal paper data.

[0035] Furthermore, the knowledge base construction module also includes:

[0036] A text standardization unit, used for performing text standardization processing on the journal article, including unified character encoding, formatted text and standardized symbols;

[0037] The metadata standardization unit is used to perform metadata standardization processing on the journal articles, including unified field processing and unified format processing.

[0038] Furthermore, the index building module includes:

[0039] A first vector representation unit, used to extract semantic information from the standard text data using a pre-trained dense vector model to obtain a corresponding dense vector representation;

[0040] A second vector representation unit is used to extract the information expression of keyword analysis and word frequency statistics in the standard text data by using a pre-trained sparse vector model to obtain a corresponding sparse vector representation;

[0041] An index building unit is used to combine the dense vector representation and the sparse vector representation to obtain the knowledge vector; and to build a mixed index between the knowledge vector and the standard text data.

[0042] Furthermore, the receiving and rewriting module includes:

[0043] A query analysis unit, used to perform query analysis on the query text to obtain keywords and query intent corresponding to the query text;

[0044] The rewriting unit is used to select at least one rewriting method from synonym expansion processing, query completion processing, query enhancement processing and context-aware processing, and rewrite the query text based on the keywords and the query intent to obtain the target query text.

[0045] Furthermore, the retrieval enhancement generation model includes a retrieval module and a generation module; the device also includes a model training module for:

[0046] Extracting part of the standard text data from the journal article knowledge base as a training data set; and using the training data set to train the retrieval module to obtain a trained retrieval module;

[0047] The generation module is trained based on the search result of the search module to obtain a trained generation module.

[0048] Furthermore, the retrieval and generation module includes:

[0049] A vector retrieval unit, used to convert the target query text into a query vector, and determine the target knowledge vector with the highest similarity from the journal article knowledge base based on the trained retrieval module;

[0050] A hybrid retrieval unit, configured to retrieve the target journal article data most relevant to the target query text from the hybrid index based on the target knowledge vector;

[0051] A generation unit is used to generate a response text corresponding to the target query text based on the trained generation module and the target journal article data.

[0052] Furthermore, the device also includes a model optimization module, which is used to:

[0053] Collect user feedback information, and classify and process the user feedback information to obtain user feedback problem types;

[0054] Based on the user feedback question type, the pre-trained retrieval enhancement generation model is subjected to model optimization processing, including at least one of data enhancement processing, algorithm adjustment processing, model fine-tuning processing, and feature adjustment processing.

[0055] According to another aspect of the present invention, a storage medium is provided, wherein the storage medium stores at least one executable instruction, wherein the executable instruction enables a processor to execute operations corresponding to the above-mentioned method for processing knowledge base information.

[0056] According to another aspect of the present invention, there is provided a computer device, comprising a processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface communicate with each other via the communication bus;

[0057] The memory is used to store at least one executable instruction, and the executable instruction enables the processor to execute operations corresponding to the above-mentioned knowledge base information processing method.

[0058] By means of the above technical solution, the technical solution provided by the embodiment of the present invention has at least the following advantages:

[0059] The present invention provides a method and device for processing knowledge base information, a storage medium, and a computer device. Compared with the prior art, the present invention collects journal paper data, performs standardization processing on the journal paper data, and obtains standard text data; and constructs a journal paper knowledge base based on the standard text data; performs vector representation processing on the standard text data in the journal paper knowledge base, and constructs a mixed index corresponding to the standard text data based on the obtained knowledge vector; receives a user's query text, rewrites the query text based on a preset mixed retrieval strategy, and obtains a target query text; determines the target journal paper data corresponding to the target query text from the journal paper knowledge base based on a pre-trained retrieval enhancement generation model and the mixed index, and generates a reply text based on the pre-trained retrieval enhancement generation model and the target journal paper data, which not only realizes the deep mining and application of knowledge, but also generates more comprehensive and in-depth content, and can better meet the application needs of various professional fields; and also solves the problems of poor relevance and low accuracy of retrieval results while improving the efficiency of document retrieval.

[0060] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, it can be implemented according to the contents of the specification. In order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present invention. Moreover, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings:

[0062] Figure 1 A schematic diagram showing a flow chart of a method for processing knowledge base information provided by an embodiment of the present invention;

[0063] Figure 2 A schematic diagram of retrieval and generation results of a retrieval enhancement generation model provided by an embodiment of the present invention is shown;

[0064] Figure 3 A schematic diagram showing a flow chart of another method for processing knowledge base information provided by an embodiment of the present invention;

[0065] Figure 4 A schematic diagram showing a flow chart of another method for processing knowledge base information provided by an embodiment of the present invention;

[0066] Figure 5A schematic flow chart of another method for processing knowledge base information provided by an embodiment of the present invention is shown;

[0067] Figure 6 A schematic diagram showing the structure of a device for processing knowledge base information provided by an embodiment of the present invention;

[0068] Figure 7 A schematic structural diagram of a computer device provided by an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0069] The exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0070] The embodiment of the present invention provides a method for processing knowledge base information. Figure 1 As shown, the method includes:

[0071] 101. Collect journal article data, perform standardization processing on the journal article data to obtain standard text data; and construct a journal article knowledge base based on the standard text data;

[0072] In the embodiment of the present invention, the current execution end collects journal article data from multiple data sources to ensure the extensiveness and authority of the data. After the data is collected, the collected journal article data is standardized to obtain standard text data. The purpose of the standardization process is to make the standard text data have a unified expression form and a unified format. The current execution end integrates the standard text data obtained through standardization to construct a journal article knowledge base for users to query.

[0073] It should be noted that, before the current execution end performs standardization processing on the collected journal paper data, it may also perform data cleaning processing on the collected journal paper data to eliminate irrelevant data, such as author, publication year, etc., which is not specifically limited in the embodiment of the present invention; in addition, the collected journal paper data may also be deduplicated to eliminate redundant data, etc., which is not specifically limited in the embodiment of the present invention.

[0074] 102. Performing vector representation processing on the standard text data in the journal article knowledge base, and constructing a mixed index corresponding to the standard text data based on the obtained knowledge vector;

[0075] In an embodiment of the present invention, the current execution end performs vector representation on the standard text data in the journal article knowledge base, such as using a pre-trained deep learning model (such as BERT, RoBERTa, etc.) to convert the journal article content into vectors to capture semantic information, etc., which is not specifically limited in the embodiment of the present invention. The current execution end constructs a hybrid index corresponding to the standard text data based on the knowledge vector obtained by the vector representation. Among them, the hybrid index can be constructed by an index structure (such as Annoy, Faiss, HNSW, etc.), which is not specifically limited in the embodiment of the present invention.

[0076] 103. Receive a user's query text, and rewrite the query text based on a preset hybrid search strategy to obtain a target query text;

[0077] In the embodiment of the present invention, the current execution end receives the user's query text, such as Figure 2 As shown, the query text received from the user input is "the impact of artificial intelligence on the future publishing industry and future development trends", etc., which is not specifically limited in the embodiment of the present invention. The current execution end needs to obtain a preset hybrid retrieval strategy, and then rewrite the query text based on the obtained hybrid retrieval strategy to obtain the target query text. Among them, the purpose of the rewriting operation is to improve the recall rate and accuracy of the retrieval. This process mainly revolves around two aspects: one is to add some useful content to the original query text to improve the recall rate; the other is to remove some irrelevant content from the original query text to optimize the retrieval results. In the embodiment of the present invention, the current execution end mainly adds some useful content to the original query text according to the application scenario to improve the recall rate.

[0078] 104. Determine the target journal article data corresponding to the target query text from the journal article knowledge base based on the pre-trained retrieval enhancement generation model and the hybrid index, and generate a reply text based on the pre-trained retrieval enhancement generation model and the target journal article data.

[0079] In the embodiment of the present invention, the current execution end determines the target journal article data corresponding to the target query text from the journal article knowledge base based on the pre-trained retrieval augmented generation model (RAG) and the hybrid index constructed for each standard text data in step 102. Then, the current execution end generates a reply text based on the pre-trained retrieval augmented generation model and the obtained target journal article data. Figure 2As shown, for the query text "the impact of artificial intelligence on the future publishing industry and future development trends", the corresponding target journal paper data is "A Brief Analysis of How Artificial Intelligence Affects the Future of the Publishing Industry", and the generated reply text is "The impact of artificial intelligence on the publishing industry and future development trends are as follows: 1. Reshape the publishing industry and promote the establishment of a new publishing order. ......2. Accelerate the speed of knowledge dissemination. ......3. Establish more extensive and profound connections with other industries. ......4. Reshape the new publishing format through intelligent marketing, intelligent platforms and intelligent creation. ......5. Realize the diversification, personalization and intelligence of publishing content, as well as the efficiency and precision of publishing services. ......10. Artificial intelligence will become a new 'new' subject in the publishing industry, forming a complementary relationship with the traditional publishing industry, and jointly promoting the development of the publishing industry." The embodiments of the present invention are not specifically limited.

[0080] It should be noted that the retrieval enhancement generation model RAG in the embodiment of the present invention includes a retrieval module and a generation module. The process of pre-training the above-mentioned retrieval enhancement generation model RAG in the embodiment of the present invention includes: extracting part of the standard text data from the journal article knowledge base constructed in step 101 as a training data set, which can ensure the quality and representativeness of the training data set; and using the extracted high-quality and representative training data set to train the retrieval module in the retrieval enhancement generation model RAG to obtain a trained retrieval module. It should be noted that the use of journal article data in the training data set to train the retrieval module can also optimize the representation of the knowledge vector. Then, based on the retrieval results of the retrieval module, that is, the retrieved knowledge vector, the generation module in the retrieval enhancement generation model RAG is trained to obtain a trained generation module. The whole process is an iterative process, which needs to be adjusted and optimized according to the actual situation to obtain the best feature set. Through the process of data preprocessing, feature selection, feature extraction and feature verification, the accuracy and interpretability of the model can be improved, and valuable information and patterns hidden in the data can be discovered. Among them, the generation module can use GPT, Transformer, etc., which is not specifically limited in the embodiment of the present invention.

[0081] Furthermore, as a refinement and extension of the specific implementation of the above embodiment, in order to facilitate the retrieval of journal articles and improve the efficiency and accuracy of the retrieval, another method for processing knowledge base information is provided, such as Figure 3 As shown, the steps of standardizing the journal article data include:

[0082] 201. Performing text standardization on the journal articles, including unified character encoding, formatted text and standardized symbols;

[0083] In an embodiment of the present invention, the current execution end performs text standardization processing on journal articles, including unified character encoding, formatted text, and standardized symbols. Among them, the unified character encoding can ensure that all texts use the same character encoding (such as UTF-8) to avoid garbled characters. Formatted text is used to unify the text format, such as removing unnecessary spaces, line breaks, etc., to ensure the neatness of the text. Standardized symbols are used to unify the use of standard punctuation marks, such as replacing full-width punctuation marks with half-width punctuation marks, etc., which are not specifically limited in the embodiment of the present invention.

[0084] 202. Perform metadata standardization on the journal article, including unified field processing and unified format processing.

[0085] In the embodiment of the present invention, the current execution end performs metadata standardization processing on the journal article, including unified field processing and unified format processing. Among them, the unified field processing can ensure that all metadata fields (such as title, author, abstract, keywords, etc.) are named consistently. The unified format processing is used to normalize the metadata fields, such as unifying the date format, unifying the writing method of the author name, etc., which is not specifically limited in the embodiment of the present invention.

[0086] Further, as a refinement and extension of the specific implementation of the above embodiment, in order to meet the hybrid retrieval requirements, another method for processing knowledge base information is provided, such as Figure 4 As shown, the step performs vector representation processing on the standard text data in the journal article knowledge base, and constructs a mixed index corresponding to the standard text data based on the obtained knowledge vector, including:

[0087] 301. Using a pre-trained dense vector model to extract semantic information from the standard text data to obtain a corresponding dense vector representation;

[0088] In the embodiment of the present invention, the current execution end uses a pre-trained dense vector model to extract semantic information from the standard text data in step 101 to obtain a corresponding dense vector representation Dense Vector. The pre-trained dense vector model can be a deep learning model (such as BERT, RoBERTa, etc.), which is used to obtain high-dimensional feature expressions after semantic analysis of the standard text data.

[0089] 302. Use a pre-trained sparse vector model to extract information expressions of keyword analysis and word frequency statistics in the standard text data to obtain corresponding sparse vector representations;

[0090] In an embodiment of the present invention, the current execution end uses a pre-trained sparse vector model to extract the information expression of keyword analysis and word frequency statistics in the standard text data to obtain a corresponding sparse vector representation Sparse Vector. Among them, keyword analysis and word frequency statistics are to extract all keywords in the standard text data, and count the frequency of occurrence of each keyword, sort and analyze the importance of the keywords, and after obtaining the target keyword, the target keyword is represented by a vector to obtain the corresponding sparse vector representation, which is not specifically limited in the embodiment of the present invention.

[0091] 303. Combine the dense vector representation and the sparse vector representation to obtain the knowledge vector; and construct a mixed index between the knowledge vector and the standard text data.

[0092] In an embodiment of the present invention, the current execution end combines the dense vector representation and the sparse vector representation to obtain a knowledge vector. In addition, the current execution end also constructs a hybrid index between the knowledge vector and the standard text data. The hybrid index can be processed using index structures such as Annoy, Faiss, HNSW, etc., which are not specifically limited in the embodiment of the present invention.

[0093] Further, as a refinement and extension of the specific implementation of the above embodiment, in order to facilitate mixed retrieval and improve the efficiency and accuracy of retrieval, another method for processing knowledge base information is provided, such as Figure 5 As shown, the step rewrites the query text based on a preset hybrid search strategy to obtain a target query text, including:

[0094] 401. Perform query analysis on the query text to obtain keywords and query intent corresponding to the query text;

[0095] In the embodiment of the present invention, the current execution end performs query analysis processing on the query text input by the user, such as analyzing the structure of the query text, extracting keywords of the query text using the TF-IDF algorithm after word segmentation of the query text; and analyzing the semantic structure of the query text using natural language processing technology (such as a semantic parser) to identify the query intent. After query analysis processing, keywords and query intent corresponding to the query text are obtained.

[0096] 402. Select at least one rewriting method from synonym expansion processing, query completion processing, query enhancement processing, and context-aware processing, and rewrite the query text based on the keyword and the query intent to obtain the target query text.

[0097] In an embodiment of the present invention, the current execution end selects at least one rewriting method from synonym expansion processing, query completion processing, query enhancement processing and context-aware processing, and rewrites the query text input by the user based on the keywords and query intent obtained in step 401 to obtain the target query text. Among them, the synonym expansion processing can use a built or existing synonym table to replace the keywords in the query text with their synonyms to expand the search scope; it can also select synonyms related to the query intent according to the context, etc., which are not specifically limited in the embodiment of the present invention. Among them, the query completion processing can find complete queries similar to the user's query text by analyzing historical query logs, and provide completion suggestions to the user; it can also use a language model (such as BERT) to predict possible complete queries based on the user's query text, etc., which are not specifically limited in the embodiment of the present invention. Among them, the query enhancement processing can use named entity recognition technology to identify entities in the query, such as names of people, organization names, etc., and then link the entities to corresponding entities in the knowledge graph to use additional information in the knowledge graph for query enhancement, etc., which are not specifically limited in the embodiment of the present invention. Among them, context-aware processing can utilize the user's query history to understand the context of the query and perform personalized query rewriting; it can also understand the user's intention based on the previous conversation history in multiple rounds of conversations, dynamically adjust the query text, etc., which is not specifically limited in the embodiments of the present invention.

[0098] Further, as a refinement and extension of the specific implementation of the above embodiment, in order to facilitate hybrid retrieval, speed up retrieval efficiency, and improve retrieval accuracy, another method for knowledge base information processing is provided, the steps of determining target journal paper data corresponding to the target query text from the journal paper knowledge base based on the pre-trained retrieval enhancement generation model and the hybrid index, and generating a reply text based on the pre-trained retrieval enhancement generation model and the target journal paper data, including:

[0099] Converting the target query text into a query vector, and determining the target knowledge vector with the highest similarity from the journal article knowledge base based on the trained retrieval module;

[0100] Retrieving the target journal article data most relevant to the target query text from the mixed index based on the target knowledge vector;

[0101] A reply text corresponding to the target query text is generated based on the trained generation module and the target journal article data.

[0102] In an embodiment of the present invention, the current execution end converts the target query text obtained in step 103 or step 402 into a query vector, that is, converts the target query text into a Dense vector and a Sparse vector at the same time, and determines the target knowledge vector with the highest similarity from the journal article knowledge base based on the retrieval module trained in the retrieval enhancement generation model RAG, that is, provides a hybrid retrieval combining Dense index + Sparse index; then, based on the target knowledge vector, retrieve the target journal article data most relevant to the target query text from the hybrid index constructed in step 102 or step 303. The above retrieval process includes hybrid matching methods such as semantic matching, keyword matching, and word frequency matching, which are not specifically limited in the embodiment of the present invention. After obtaining the target journal article data, the current execution end also generates a reply text corresponding to the target query text based on the generation module trained in the retrieval enhancement generation model RAG and the target journal article data.

[0103] Furthermore, as a refinement and extension of the specific implementation of the above embodiment, in order to further improve the accuracy of retrieval, another method for processing knowledge base information is provided, and the method further includes:

[0104] Collect user feedback information, and classify and process the user feedback information to obtain user feedback problem types;

[0105] Based on the user feedback question type, the pre-trained retrieval enhancement generation model is subjected to model optimization processing, including at least one of data enhancement processing, algorithm adjustment processing, model fine-tuning processing, and feature adjustment processing.

[0106] In an embodiment of the present invention, the current execution end can understand the user's satisfaction with the search results through user surveys, user interviews, user behavior analysis, etc. Then, the collected user feedback information is classified and summarized, for example, grouped according to the theme of the feedback, and common problems and patterns in user feedback are identified to find out where improvements are needed. Or, based on user feedback, various performance indicators of the model are evaluated, such as recall rate, precision rate, etc. Specific user feedback cases are selected, and the performance of the model in these cases is analyzed to find out where the problems lie.

[0107] Next, the current execution end performs model optimization processing on the pre-trained retrieval enhancement generation model based on the user feedback question type, including at least one of data enhancement processing, algorithm adjustment processing, model fine-tuning processing, and feature adjustment processing. Among them, the data enhancement processing can collect more relevant data for model training based on the problems mentioned in the user feedback. The algorithm adjustment processing can adjust the algorithm parameters of the RAG model, such as learning rate, loss function, etc., based on user feedback. The model fine-tuning processing can use the newly collected data and adjusted parameters to fine-tune the model to improve the performance of the model. The feature adjustment processing can add or adjust features based on the problems mentioned in the user feedback to improve the performance of the model.

[0108] The embodiment of the present invention provides a method for processing knowledge base information. Compared with the prior art, the present invention collects journal article data, standardizes the journal article data, and obtains standard text data; and constructs a journal article knowledge base based on the standard text data; performs vector representation processing on the standard text data in the journal article knowledge base, and constructs a hybrid index corresponding to the standard text data based on the obtained knowledge vector; receives a user's query text, rewrites the query text based on a preset hybrid retrieval strategy, and obtains a target query text; determines the target journal article data corresponding to the target query text from the journal article knowledge base based on a pre-trained retrieval enhancement generation model and the hybrid index, and generates a reply text based on the pre-trained retrieval enhancement generation model and the target journal article data, which not only realizes the deep mining and application of knowledge, but also generates more comprehensive and in-depth content, which can better meet the application needs of various professional fields; and while improving the efficiency of document retrieval, solves the problems of poor relevance and low accuracy of retrieval results.

[0109] As the above Figure 1 The embodiment of the present invention provides a device for processing knowledge base information, such as Figure 6 As shown, the device comprises:

[0110] The knowledge base construction module 51 is used to collect journal article data, standardize the journal article data to obtain standard text data, and construct a journal article knowledge base based on the standard text data;

[0111] An index building module 52, used for performing vector representation processing on the standard text data in the journal article knowledge base, and building a mixed index corresponding to the standard text data based on the obtained knowledge vector;

[0112] The receiving and rewriting module 53 is used to receive the user's query text and rewrite the query text based on a preset hybrid search strategy to obtain a target query text;

[0113] The retrieval and generation module 54 is used to determine the target journal article data corresponding to the target query text from the journal article knowledge base based on the pre-trained retrieval enhancement generation model and the hybrid index, and to generate a reply text based on the pre-trained retrieval enhancement generation model and the target journal article data.

[0114] Furthermore, the knowledge base construction module 51 also includes:

[0115] A text standardization unit, used for performing text standardization processing on the journal article, including unified character encoding, formatted text and standardized symbols;

[0116] The metadata standardization unit is used to perform metadata standardization processing on the journal articles, including unified field processing and unified format processing.

[0117] Furthermore, the index building module 52 includes:

[0118] A first vector representation unit, used to extract semantic information from the standard text data using a pre-trained dense vector model to obtain a corresponding dense vector representation;

[0119] A second vector representation unit is used to extract the information expression of keyword analysis and word frequency statistics in the standard text data by using a pre-trained sparse vector model to obtain a corresponding sparse vector representation;

[0120] An index building unit is used to combine the dense vector representation and the sparse vector representation to obtain the knowledge vector; and to build a mixed index between the knowledge vector and the standard text data.

[0121] Furthermore, the receiving and rewriting module 53 includes:

[0122] A query analysis unit, used to perform query analysis on the query text to obtain keywords and query intent corresponding to the query text;

[0123] The rewriting unit is used to select at least one rewriting method from synonym expansion processing, query completion processing, query enhancement processing and context-aware processing, and rewrite the query text based on the keywords and the query intent to obtain the target query text.

[0124] Furthermore, the retrieval enhancement generation model includes a retrieval module and a generation module; the device also includes a model training module for:

[0125] Extracting part of the standard text data from the journal article knowledge base as a training data set; and using the training data set to train the retrieval module to obtain a trained retrieval module;

[0126] The generation module is trained based on the search result of the search module to obtain a trained generation module.

[0127] Furthermore, the retrieval and generation module 54 includes:

[0128] A vector retrieval unit, used to convert the target query text into a query vector, and determine the target knowledge vector with the highest similarity from the journal article knowledge base based on the trained retrieval module;

[0129] A hybrid retrieval unit, configured to retrieve the target journal article data most relevant to the target query text from the hybrid index based on the target knowledge vector;

[0130] A generation unit is used to generate a response text corresponding to the target query text based on the trained generation module and the target journal article data.

[0131] Furthermore, the device also includes a model optimization module, which is used to:

[0132] Collect user feedback information, and classify and process the user feedback information to obtain user feedback problem types;

[0133] Based on the user feedback question type, the pre-trained retrieval enhancement generation model is subjected to model optimization processing, including at least one of data enhancement processing, algorithm adjustment processing, model fine-tuning processing, and feature adjustment processing.

[0134] The embodiment of the present invention provides a device for processing knowledge base information. Compared with the prior art, the present invention collects journal article data, standardizes the journal article data, and obtains standard text data; and constructs a journal article knowledge base based on the standard text data; performs vector representation processing on the standard text data in the journal article knowledge base, and constructs a mixed index corresponding to the standard text data based on the obtained knowledge vector; receives a user's query text, rewrites the query text based on a preset mixed retrieval strategy, and obtains a target query text; determines the target journal article data corresponding to the target query text from the journal article knowledge base based on a pre-trained retrieval enhancement generation model and the mixed index, and generates a reply text based on the pre-trained retrieval enhancement generation model and the target journal article data, which not only realizes the deep mining and application of knowledge, but also generates more comprehensive and in-depth content, which can better meet the application needs of various professional fields; and solves the problems of poor relevance and low accuracy of retrieval results while improving the efficiency of document retrieval.

[0135] According to an embodiment of the present invention, a storage medium is provided, wherein the storage medium stores at least one executable instruction, and the computer executable instruction can execute the method for processing knowledge base information in any of the above method embodiments.

[0136] Figure 7 A schematic diagram of the structure of a computer device provided according to an embodiment of the present invention is shown. The specific embodiment of the present invention does not limit the specific implementation of the computer device.

[0137] like Figure 7 As shown, the computer device may include: a processor (processor) 602 , a communication interface (Communications Interface) 604 , a memory (memory) 606 , and a communication bus 608 .

[0138] The processor 602 , the communication interface 604 , and the memory 606 communicate with each other via a communication bus 608 .

[0139] The communication interface 604 is used to communicate with other devices such as clients or other servers.

[0140] The processor 602 is used to execute the program 610, and specifically can execute the relevant steps of the above-mentioned knowledge base information processing method.

[0141] Specifically, the program 610 may include program codes, which include computer operation instructions.

[0142] The processor 602 may be a central processing unit (CPU), or an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present invention. The one or more processors included in the computer device may be processors of the same type, such as one or more CPUs; or processors of different types, such as one or more CPUs and one or more ASICs.

[0143] The memory 606 is used to store the program 610. The memory 606 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0144] The program 610 may be specifically configured to enable the processor 602 to perform the following operations:

[0145] Collecting journal article data, standardizing the journal article data to obtain standard text data; and constructing a journal article knowledge base based on the standard text data;

[0146] Performing vector representation processing on the standard text data in the journal article knowledge base, and constructing a mixed index corresponding to the standard text data based on the obtained knowledge vector;

[0147] Receiving a user's query text, and rewriting the query text based on a preset hybrid search strategy to obtain a target query text;

[0148] Based on the pre-trained retrieval enhancement generation model and the hybrid index, target journal article data corresponding to the target query text is determined from the journal article knowledge base, and a reply text is generated based on the pre-trained retrieval enhancement generation model and the target journal article data.

[0149] Obviously, those skilled in the art should understand that the above modules or steps of the present invention can be implemented by a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, and optionally, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be executed in a different order than here, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. Thus, the present invention is not limited to any specific combination of hardware and software.

[0150] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for knowledge base information processing, characterized in that: include: Collecting journal article data, standardizing the journal article data to obtain standard text data; and constructing a journal article knowledge base based on the standard text data; Performing vector representation processing on the standard text data in the journal article knowledge base, and constructing a mixed index corresponding to the standard text data based on the obtained knowledge vector; Receiving a user's query text, and rewriting the query text based on a preset hybrid search strategy to obtain a target query text; Based on the pre-trained retrieval enhancement generation model and the hybrid index, target journal article data corresponding to the target query text is determined from the journal article knowledge base, and a reply text is generated based on the pre-trained retrieval enhancement generation model and the target journal article data.

2. The method according to claim 1, characterized in that The standardization of the journal article data includes: Performing text standardization on the journal articles, including unified character encoding, formatted text and standardized symbols; The metadata of the journal articles are standardized, including unified field processing and unified format processing.

3. The method according to claim 1, characterized in that The step of performing vector representation processing on the standard text data in the journal article knowledge base and constructing a mixed index corresponding to the standard text data based on the obtained knowledge vector includes: Using a pre-trained dense vector model to extract semantic information from the standard text data to obtain a corresponding dense vector representation; Using a pre-trained sparse vector model to extract information expressions of keyword analysis and word frequency statistics in the standard text data, and obtain corresponding sparse vector representations; The dense vector representation and the sparse vector representation are combined to obtain the knowledge vector; and a mixed index is constructed between the knowledge vector and the standard text data.

4. The method according to claim 1, characterized in that: The rewriting operation on the query text based on the preset hybrid search strategy to obtain the target query text includes: Performing query analysis on the query text to obtain keywords and query intent corresponding to the query text; At least one rewriting method is selected from synonym expansion processing, query completion processing, query enhancement processing and context-aware processing, and the query text is rewritten based on the keywords and the query intent to obtain the target query text.

5. The method according to claim 1, characterized in that: The retrieval enhancement generation model includes a retrieval module and a generation module; The process of pre-training the retrieval-enhanced generative model includes: Extracting part of the standard text data from the journal article knowledge base as a training data set; and using the training data set to train the retrieval module to obtain a trained retrieval module; The generation module is trained based on the search result of the search module to obtain a trained generation module.

6. The method according to claim 5, characterized in that The method comprises: determining target journal article data corresponding to the target query text from the journal article knowledge base based on the pre-trained retrieval enhancement generation model and the hybrid index, and generating a reply text based on the pre-trained retrieval enhancement generation model and the target journal article data, including: Converting the target query text into a query vector, and determining the target knowledge vector with the highest similarity from the journal article knowledge base based on the trained retrieval module; Retrieving the target journal article data most relevant to the target query text from the mixed index based on the target knowledge vector; A reply text corresponding to the target query text is generated based on the trained generation module and the target journal article data.

7. The method according to any one of claims 1 to 6, characterized in that: The method further comprises: Collect user feedback information, and classify and process the user feedback information to obtain user feedback problem types; Based on the user feedback question type, the pre-trained retrieval enhancement generation model is subjected to model optimization processing, including at least one of data enhancement processing, algorithm adjustment processing, model fine-tuning processing, and feature adjustment processing.

8. A device for processing knowledge base information, characterized in that: include: A knowledge base construction module is used to collect journal article data, standardize the journal article data to obtain standard text data, and construct a journal article knowledge base based on the standard text data; An index building module, used for performing vector representation processing on the standard text data in the journal article knowledge base, and building a mixed index corresponding to the standard text data based on the obtained knowledge vector; A receiving and rewriting module, used for receiving a user's query text, and rewriting the query text based on a preset hybrid search strategy to obtain a target query text; A retrieval and generation module is used to determine the target journal paper data corresponding to the target query text from the journal paper knowledge base based on the pre-trained retrieval enhancement generation model and the hybrid index, and to generate a reply text based on the pre-trained retrieval enhancement generation model and the target journal paper data.

9. A storage medium, wherein at least one executable instruction is stored in the storage medium, and the executable instruction executes an operation corresponding to the method for processing knowledge base information as described in any one of claims 1 to 7.

10. A computer device, comprising a processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface communicate with each other via the communication bus; The memory is used to store at least one executable instruction, and the executable instruction enables the processor to perform operations corresponding to the method for processing knowledge base information as described in any one of claims 1-7.

Images