Knowledge base query method and device, electronic equipment and storage medium

Abstract

The invention provides a knowledge base query method and device, electronic equipment and a storage medium, relates to the technical field of data processing, and aims to construct a multi-source heterogeneous data knowledge base by analyzing and structuring documents in various formats. Semantic understanding and vector representation generation of a user query instruction are realized by means of a large language model deployed by a local server, semantic retrieval is carried out in a multi-source heterogeneous data knowledge base based on the vector representation, and meanwhile, condition inspection is carried out on a semantic retrieval result to dynamically determine whether a structure query statement generation process is triggered or not. The problems that in the prior art, due to dependence on external large model service, data privacy risks exist, multi-source heterogeneous data unified analysis and retrieval capacity is lacked, and structured query statements cannot be generated according to needs can be solved.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of data processing, and in particular, to a knowledge base query method and device, electronic equipment and storage medium. BACKGROUND

[0002] Retrieval-Augmented Generation (RAG) system, as an important technical support for enterprise knowledge management, is widely used in industries such as finance and government that have strict requirements for data compliance. In related technologies, a knowledge-enhanced question and answer system is constructed through the collaborative work of pre-training large models and vector retrieval. Specifically, this technical system covers the whole process from natural language analysis to semantic vector generation, including document vectorization storage, similarity retrieval, context fusion and other key links. With the exponential growth of enterprise data assets, the existing technology has developed from the primary stage based on keyword matching to an intelligent retrieval system that integrates semantic understanding, but its technical implementation still mainly relies on cloud API services, making it difficult to meet the needs of local processing of multi-source heterogeneous data.

[0003] However, in the existing RAG implementation scheme, third-party API services such as OpenAI are directly used, and a complete local technical closed loop is not constructed, which may lead to data cross-border transmission risks or delay in API calls (200-500ms) that cannot meet the needs of real-time business scenarios. Specifically, related technologies usually implement knowledge retrieval through a single text vectorization, but lack special parsing capabilities for code libraries and database queries, resulting in the need for manual maintenance of the semantic mapping of code logic and database structure, which is costly and prone to errors. Based on this, the existing system cannot establish dynamically updated knowledge associations when processing function definitions and annotations in Git repositories, resulting in lagging knowledge base updates. In addition, although traditional natural language to SQL conversion methods use template matching, they do not integrate syntax tree analysis and entity recognition technology, making it difficult to meet the field mapping needs in complex query scenarios, thereby affecting the intelligence level of data statistical tools. SUMMARY

[0004] The present disclosure provides a knowledge base query method and device, electronic equipment and storage medium. Its main purpose is to at least partially solve one of the technical problems in the related art. According to a first aspect of the present disclosure, a knowledge base query method is provided, comprising: parsing and structuring a plurality of format documents to construct a multi-source heterogeneous data knowledge base; in response to a user query instruction, a large language model deployed on a local server performs semantic understanding on the query text in the user query instruction and generates a vector representation thereof; Based on the vector representation, semantic retrieval is performed in the multi-source heterogeneous data knowledge base; The result of semantic retrieval is conditionally tested to determine whether to perform retrieval enhancement processing, and when the result of semantic retrieval does not meet the confidence condition, a structured query statement generation process is triggered to convert the query text into an executable structured query statement.

[0005] Optionally, a plurality of format documents are parsed and structured to build a multi-source heterogeneous data knowledge base, including: A plurality of format documents are processed by a text parsing engine to extract text content and its logical structure; The code parser accesses the version control system to extract function definitions, associated comments and embedded query statements in the code file, and establishes a semantic association index between them and natural language descriptions.

[0006] Optionally, the text parsing engine supports at least parsing of portable document format and word processing document format, and can recognize and convert table structures in the documents; The code parser obtains the target code repository by performing a cloning operation, and parses the structured query language files and script files therein to build a mapping table between function names and function descriptions.

[0007] Optionally, it also includes: The large language model is quantized and compressed, and the quantized and compressed large language model is deployed on a local server through a containerization algorithm, wherein the large language model enables a parallel computing engine during inference and sets tensor parallel parameters to speed up the computing process.

[0008] Optionally, based on the vector representation, semantic retrieval is performed in the multi-source heterogeneous data knowledge base, including: Based on a preset search algorithm, the vector index of the multi-source heterogeneous data knowledge base is retrieved; The preset confidence condition is a set similarity threshold, and when the highest similarity score of the retrieved content is lower than the threshold, it is determined that the confidence condition is not met.

[0009] Optionally, the structured query statement generation process is triggered to convert the query text into an executable structured query statement, including: The query text is parsed by a syntax parser and mapped to a standard query template; The named entity recognition algorithm is used to identify entity information from the query text, and the entity information is dynamically filled into the corresponding parameter position in the query template in combination with the metadata of the target database to form a structured query statement.

[0010] Optionally, after converting the query text into an executable structured query statement, it further includes: The database is connected through a query executor to execute structured query statements and obtain returned data results; The data results are converted into a standardized data format required by a front-end visualization chart component for graphical display.

[0011] According to a second aspect of the present disclosure, a knowledge base query device is provided, comprising: A construction unit for parsing and structuring multiple format documents to construct a multi-source heterogeneous data knowledge base; A generation unit for responding to a user query instruction, performing semantic understanding on query text in the user query instruction through a large language model deployed on a local server and generating a vector representation thereof; A retrieval unit for performing semantic retrieval in the multi-source heterogeneous data knowledge base based on the vector representation; An optimization unit for conditionally checking the results of semantic retrieval to determine whether to perform retrieval enhancement processing, and triggering a structured query statement generation process to convert the query text into an executable structured query statement when the results of semantic retrieval do not meet a confidence condition.

[0012] Optionally, the construction unit comprises: A first extraction module for processing multiple format documents through a text parsing engine to extract text content and its logical structure; A second extraction module for accessing a version control system through a code parser to extract function definitions, associated comments and embedded query statements in code files and establish a semantic association index thereof with natural language descriptions.

[0013] Optionally, the text parsing engine supports at least parsing of portable document format and word processing document format and can recognize and convert table structures in the documents; The code parser acquires a target code repository by performing a cloning operation and parses structured query language files and script files therein to construct a mapping relationship table between function names and function descriptions.

[0014] Optionally, further comprising: A compression unit for quantitatively compressing the large language model and deploying the quantitatively compressed large language model on the local server through a containerization algorithm, wherein the large language model enables a parallel computing engine during inference and sets tensor parallel parameters to accelerate the computing process.

[0015] Optionally, the retrieval unit comprises: A retrieval module for retrieving vector indexes of the multi-source heterogeneous data knowledge base based on a preset search algorithm; The determination module is configured to determine that the confidence condition is not satisfied when a highest similarity score of the returned content is lower than a preset similarity threshold.

[0016] Optionally, the optimization unit comprises: The mapping module is configured to parse the query text by using a syntax parser to perform natural language analysis and map the query text into a standard query template. The forming module is configured to identify entity information from the query text by using a named entity recognition algorithm, and dynamically fill the entity information into corresponding parameter positions in the query template in combination with metadata of the target database to form a structured query statement.

[0017] Optionally, after the query text is converted into an executable structured query statement, the method further comprises: The execution unit is configured to connect to the database by using a query executor, execute the structured query statement, and obtain returned data results; and convert the data results into a standardized data format required by a front-end visual chart component for graphical display.

[0018] According to a third aspect of the present disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the method of the first aspect.

[0019] According to a fourth aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute the method of the first aspect.

[0020] According to a fifth aspect of the present disclosure, a computer program product is provided, comprising a computer program which, when executed by a processor, implements the method of the first aspect.

[0021] The knowledge base query method and device, the electronic equipment and the storage medium provided by the present disclosure can solve the problems in the prior art, such as data privacy risks due to dependence on external large model services, lack of unified analysis and retrieval capabilities for multi-source heterogeneous data, and inability to generate structured query statements on demand, thereby achieving technical effects of guaranteeing enterprise data security compliance, improving multi-source data retrieval accuracy and efficiency, and meeting the demand for structured data acquisition in complex query scenarios.

[0022] It should be understood that the content described in this part is not intended to identify key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become apparent from the following description. BRIEF DESCRIPTION OF DRAWINGS

[0023] The accompanying drawings are used to better understand the present scheme and do not constitute a limitation on the present disclosure. Among them: Figure 1 A flowchart of a knowledge base query method provided by an embodiment of the present disclosure; Figure 2 A structural schematic diagram of a knowledge base query device provided by an embodiment of the present disclosure; Figure 3 A structural schematic diagram of a knowledge base query device provided by an embodiment of the present disclosure; Figure 4 A schematic block diagram of an example electronic device provided by an embodiment of the present disclosure. DETAILED DESCRIPTION

[0024] Exemplary embodiments of the present disclosure are described below with reference to the accompanying drawings, including various details in order to facilitate understanding. They should be considered as merely exemplary. Therefore, those of ordinary skill in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Also, in order to be clear and concise, descriptions of well-known functions and structures are omitted in the following description.

[0025] The knowledge base query method and device, the electronic equipment and the storage medium of the embodiments of the present disclosure are described below with reference to the accompanying drawings.

[0026] Figure 1 A flowchart of a knowledge base query method provided by an embodiment of the present disclosure;

[0027] As shown in the method comprises the following steps: Figure 1 Step 101, parsing and structuring a plurality of format documents to build a multi-source heterogeneous data knowledge base.

[0028] In the embodiments of the present application, in order to effectively integrate and subsequently retrieve and call the multi-source data, the technical solution first performs parsing and structuring operation on a plurality of format documents to build a multi-source heterogeneous data knowledge base. Among them, the core of the parsing process of the plurality of format documents is to extract the effective information (including text content, code logic, annotation information, etc.) contained in the different types of documents according to the storage characteristics and data structure of the different types of documents. The structured processing is to convert the unstructured or semi-structured data obtained by parsing into structured data form that conforms to the unified data specification and can be recognized and called by the subsequent retrieval module. Finally, through the integration and storage of these structured data, a multi-source heterogeneous data knowledge base covering multiple data sources and containing different types of information is formed. As an implementation manner, the text content of the PDF format document can be extracted by using the corresponding document parsing tool, the paragraph and table data of the Word format document can be obtained by using the office document processing tool, the encoding of the TXT format document is detected to ensure the accurate extraction of the text information, and the code function definition, submission history and annotation information of the Git code repository document can be extracted by using the code version control tool, thereby completing the construction of the multi-source heterogeneous data knowledge base.

[0029] Through the parsing and structuring of the plurality of format documents, the problem that the multi-source heterogeneous data is difficult to be uniformly integrated in the prior art is effectively solved, the data coverage range and information type of the knowledge base are enriched, a comprehensive and accurate data basis is provided for the subsequent semantic retrieval based on user query, and the integrity and effectiveness of the retrieval result are guaranteed.

[0030] Step 102, in response to the user query instruction, the query text in the user query instruction is understood by the large language model deployed in the local server and a vector representation thereof is generated.

[0031] ​In the embodiments of the present application, when the user query instruction is received, the technical solution responds to the instruction, relies on the large language model deployed in the local server, performs semantic understanding operation on the query text contained in the user query instruction, and further generates the vector representation corresponding to the query text. Wherein, the deployment of large language model in local server can ensure that the query text processing process is completed in the enterprise internal environment, avoiding data transmission to the outside; the semantic understanding operation aims to accurately identify the user demand intention implied in the query text, including the analysis of query theme, core appeal and potential data demand; the generation of vector representation is to convert the query text after semantic understanding into numerical vector form for subsequent similarity matching of retrieval module, laying a foundation for subsequent accurate retrieval in knowledge base. As an implementation, DeepSeek series of large language models can be deployed on local NVIDIA server through Ollama, and when the user initiates the query instruction, the query text semantics is parsed by the locally deployed large language model, and a 768-dimensional semantic vector that meets the subsequent retrieval requirements is generated.

[0032] By deploying large language model in local server to process query text, the data privacy leakage risk caused by relying on external API in the prior art is effectively solved, and the delay problem of external call is avoided; and based on the semantic understanding and vector generation of large language model, the user query intention can be accurately captured, providing accurate vector basis for subsequent semantic retrieval and improving the accuracy of retrieval matching.

[0033] Step 103, based on the vector representation, performing semantic retrieval in the multi-source heterogeneous data knowledge base.

[0034] In the embodiments of the present application, after obtaining the vector representation of the user query text, the technical solution performs semantic retrieval operation in the multi-source heterogeneous data knowledge base based on the vector representation. The core of the semantic retrieval is to compare the query vector with the vectors corresponding to various data (including parsed document data, code related structured data, etc.) stored in the multi-source heterogeneous data knowledge base through vector similarity calculation, and filter out the knowledge base data associated with the user query intention; since the knowledge base covers multi-source heterogeneous data, the retrieval process needs to adapt to the vector storage form of different types of data to ensure that relevant content can be matched from various data such as document text and code information, providing basic data support for subsequent result processing. As an implementation, the FAISS index library can be used to construct the vector index of multi-source heterogeneous data knowledge base (such as configuring HNSW algorithm to optimize retrieval efficiency), through the VectorStoreRetriever component in Spring AI, the similarity between query vector and knowledge base data vector is matched, and the similarity threshold (such as 0.65) can be set to filter out Top N related data, thereby completing the semantic retrieval.

[0035] By semantic retrieval based on vector representation, the problems that traditional keyword retrieval is difficult to understand context association and poor adaptability to multi-source heterogeneous data are effectively solved, knowledge base data related to user query intention can be accurately matched, and the relevance and comprehensiveness of retrieval results are improved, providing high-quality data basis for subsequent retrieval enhancement processing or structured query statement generation.

[0036] In step 104, the result of semantic retrieval is conditionally tested to determine whether retrieval enhancement processing is performed, and when the result of semantic retrieval does not meet the confidence condition, a structured query statement generation process is triggered to convert the query text into an executable structured query statement.

[0037] In the embodiment of the present application, after completing the semantic retrieval of the multi-source heterogeneous data knowledge base, the technical solution performs a conditional test operation on the obtained semantic retrieval result, the core of which is to determine whether the result meets the preset confidence condition, and then determine whether to carry out retrieval enhancement processing according to the test result. If the semantic retrieval result meets the confidence condition, the retrieval enhancement processing can be promoted based on the result to generate an adapted answer; if the semantic retrieval result does not meet the confidence condition, a structured query statement generation process is automatically triggered to convert the user query text into an executable structured query statement that meets the data interaction standard and can be directly connected to the business data system, ensuring that even if the retrieval result does not meet the expectation, accurate data can still be obtained through the structured query path. As an implementation, the preset confidence condition can be set as a vector similarity threshold (such as 0.65), when the retrieval result similarity is lower than the threshold, the triggered structured query statement generation process can use a grammar parser (such as ANTLR4) to build the mapping relationship between natural language and structured query template, and combine entity recognition technology to complete field matching and filling, finally generate an executable SQL statement, and the generated statement can be verified through syntax tree analysis to ensure its execution effectiveness.

[0038] By conditionally testing the semantic retrieval result, the problem of answer deviation caused by directly using low-quality retrieval results is effectively avoided, and the reliability of retrieval enhancement processing is ensured; at the same time, when the result does not meet the confidence condition, the structured query statement generation process is triggered, which solves the problem of lack of adaptability to complex structured data query requirements in the prior art, which not only expands the path of data acquisition, but also improves the accuracy and efficiency of enterprise business data query, and further improves the integrity of knowledge service.

[0039] The present disclosure provides a knowledge base query method, which constructs a multi-source heterogeneous data knowledge base by parsing and structuring various format documents, realizes semantic understanding and vector representation generation of user query instructions relying on a large language model deployed on a local server, carries out semantic retrieval in the multi-source heterogeneous data knowledge base based on the vector representation, simultaneously carries out conditional inspection on the semantic retrieval result to dynamically determine whether to trigger a structured query statement generation process, and converts the query text into an executable structured query statement when the confidence condition is not met, thereby solving the problems in the prior art of data privacy risk due to dependence on external large model services, lack of unified parsing and retrieval capability of multi-source heterogeneous data, and inability to generate structured query statements on demand, achieving the technical effects of guaranteeing enterprise data security compliance, improving multi-source data retrieval accuracy and efficiency, and meeting the demand for structured data acquisition in complex query scenarios.

[0040] Under the technical solution framework disclosed in step 101, the multi-source heterogeneous data knowledge base is constructed by parsing and structuring various format documents, specifically including: processing various format documents through a text parsing engine to extract text content and its logical structure; accessing a version control system through a code parser to extract function definitions, associated comments and embedded query statements in code files, and establishing semantic association indexes thereof with natural language descriptions.

[0041] Specifically, in the process of constructing the multi-source heterogeneous data knowledge base, the parsing and structuring of multiple formats of documents are specifically implemented by a text parsing engine and a code parser. The text parsing engine uses adaptive professional tools for different formats of documents: when processing PDF format documents, the tool reads the document byte stream to not only extract the text content, but also capture the logical structure such as chapter title hierarchy, paragraph separation, and chart associated description; when processing Word format documents, the Apache POI tool is used to parse the XML structure of the document to obtain paragraph text, table data, and formula annotations, and to retain the logical relationships such as table row / column association and paragraph order; when processing TXT format documents, the UTF-8 encoding automatic detection process is first performed to exclude text loss caused by encoding disorder, and the original logical arrangement of the text is retained based on the line sequence. The code parser uses the JGit tool as an interactive carrier with the Git version control system (a specific form of version control system), clones the target code repository by executing the “git clone” command, obtains the complete code file set, and then performs syntax scanning and semantic analysis on the code files to extract function definitions (such as the calc_activity_user() function related to data statistics), associated comments in the function header and internal (such as function functionality and parameter description), and code-embedded query statements (such as SQL query fragments in Python scripts or complete query statements in independent.sql files), and based on the above extracted contents, constructs a semantic association index of “code element-natural language description”, such as binding and storing the calc_activity_user() function with the natural language expression “calculate the number of active participants” to form a structured code knowledge unit. Finally, the text parsing results and code structured data are integrated to construct a multi-source heterogeneous data knowledge base.

[0042] By adapting professional parsing tools for different formats of documents, the accurate extraction of PDF, Word, TXT, and other text information and their logical structures is ensured, avoiding data loss caused by format differences; at the same time, relying on the JGit tool and the construction of semantic association index, the association mapping between code knowledge and natural language is realized, which not only enriches the data source types of the knowledge base, but also provides a precise matching basis for subsequent natural language-based code-related retrieval, significantly improving the completeness and practical value of the multi-source heterogeneous data knowledge base.

[0043] In the embodiments of the present application, the text parsing engine at least supports parsing of portable document format and word processing document format, and can recognize and convert table structures in the documents; The code parser obtains the target code repository by performing a cloning operation, and parses the structured query language files and script files therein to construct a mapping relationship table between function names and function descriptions.

[0044] Specifically, the text analysis engine realizes accurate processing of various format documents through professional tools: for portable document format (i.e. PDF format) documents, Apache PDFBox tool is used to read the document data stream, while extracting the text content of the main text, the document page layout and graphic elements are analyzed to identify the boundaries of the table, row / column separation lines and cell positions, and the table data is separated from the document and converted into a structured row-column data format; for word processing document format (i.e. Word format) documents, Apache POI tool is used to analyze the underlying XML structure of the document, not only to extract paragraph text, but also to identify table objects in the document, read the row index, column index and corresponding cell text content of the table, and ensure complete extraction and structured conversion of table data. The code parser uses JGit tool as the carrier for interaction with the version control system, first executes the "git clone" command to clone the remote target code repository, obtains all code files in the repository, and then performs syntax scanning on the structured query language files (i.e..sql files) in the files, extracts the query statement templates and field association information in the files, and at the same time performs semantic analysis on the script files (such as Python script files), identifies the function definitions (including function names and parameter lists) in the scripts and the function description annotations in the function headers, and then associates the extracted function names with the corresponding function descriptions one by one to build a structured "function name-function description" mapping table and store it in the knowledge base.

[0045] By specifying professional tools to realize the processing of specific format documents and table structures by the text analysis engine, the accurate extraction and structured conversion of PDF and Word document text and table data are ensured, avoiding the loss or confusion of table data due to complex formats; the code parser clones the repository and analyzes specific files to build a mapping table, so that the function of the code can be quickly associated and identified by name, providing a clear matching basis for subsequent natural language-based code-related retrieval, effectively improving the utilization efficiency of code data in the multi-source heterogeneous data knowledge base.

[0046] Under the technical solution framework disclosed in the foregoing embodiments, in order to adapt to the computing power limitations of the local server, the present application embodiment further provides a model deployment method, specifically including: quantizing and compressing a large language model, and deploying the quantized and compressed large language model to the local server through a containerization algorithm, wherein the large language model enables a parallel computing engine during inference and sets tensor parallel parameters to speed up the computing process.

[0047] Specifically, to adapt to the computing power limitations of the local server, the specific implementation of the model deployment manner in the embodiment of the application is as follows: first, the large language model used for semantic understanding and vector generation is subjected to quantization compression processing, a 4-bit quantization manner is adopted to reduce model parameter storage occupation and computing resource consumption, and on the premise that the model semantic understanding ability is basically not damaged, the hardware computing power level of the local server is adapted; then the large language model after quantization compression is deployed on the local server by using a containerization algorithm, specifically, the Docker image of Ollama is taken as a base image, the pre-set command is executed in the container construction process to pull the quantized DeepSeek model file, and the container is configured to expose the default port 11434 to realize external interface calling, and finally the model service in the container is started through the command. In addition, the vLLM parallel computing engine is enabled in the inference process of the large language model, and combined with the GPU hardware configuration of the local server, the tensor parallel parameters are set, the model inference task is distributed to multiple GPUs for parallel processing, so as to accelerate the semantic understanding and vector generation calculation process of the model on the query text.

[0048] The computing power and storage requirement of the large language model is greatly reduced by 4-bit quantization compression, effectively solving the problem that the local server with limited computing power cannot carry large model deployment; containerization deployment simplifies the installation and operation and maintenance process of the model, ensuring the stability and portability of the model service; and the configuration of the vLLM parallel computing engine and the tensor parallel parameters further improves the model inference speed, ensuring that the locally deployed large model can quickly respond to user query instructions, and the model performance and local computing power adaptability are considered.

[0049] Under the technical scheme framework disclosed in step 103, based on vector representation, semantic retrieval is performed in the multi-source heterogeneous data knowledge base, specifically including: based on a pre-set search algorithm, the vector index of the multi-source heterogeneous data knowledge base is retrieved; the pre-set confidence condition is a set similarity threshold, when the highest similarity score of the returned content is lower than the threshold, it is determined that the confidence condition is not met.

[0050] Specifically, for the constructed multi-source heterogeneous data knowledge base, a vector index is constructed in advance using the FAISS index library. The preset search algorithm used is the HNSW (Hierarchical Navigable Small Worlds) algorithm, and the configuration parameters are specifically 768-dimensional vector dimension (consistent with the query vector dimension generated by the large language model), 32-edge graph structure connection mode. The semantic vectors corresponding to the parsed text data, code structured data, etc. in the knowledge base are uniformly added to the FAISS vector index to form a vector index library that can be quickly searched. When the vector representation of the user query text is obtained, a search request is initiated to the FAISS vector index library based on the above-mentioned preset HNSW algorithm. By calculating the cosine similarity between the query vector and each data vector in the index library, a search result set containing similarity scores is obtained, and the highest similarity score in the result set is extracted. At the same time, the preset confidence condition is specifically set to a similarity threshold of 0.65. The highest similarity score obtained by searching is compared with the threshold. If the highest similarity score is lower than 0.65, it is directly determined that the semantic search result does not meet the confidence condition, providing a basis for judging the subsequent structured query statement generation process.

[0051] By selecting the HNSW algorithm of the FAISS index library as the preset search algorithm, the retrieval efficiency of high-dimensional vectors in the multi-source heterogeneous data knowledge base is effectively improved, avoiding the time-consuming problem of traditional retrieval methods when the data volume is large. By setting 0.65 as the similarity threshold as the confidence condition, the search results with strong relevance to the user's query intention can be accurately filtered, preventing low-relevance results from entering the subsequent processing process. This provides a clear and quantifiable judgment standard for the quality of semantic search results, ensuring the effectiveness of subsequent search enhancement processing or structured query generation.

[0052] Under the technical scheme framework disclosed in step 104, the structured query statement generation process is triggered to convert the query text into an executable structured query statement, which specifically includes: using a syntax parser to perform natural language parsing on the query text and mapping it to a standard query template; using a named entity recognition algorithm to identify entity information from the query text, and combining the metadata of the target database to dynamically fill the entity information into the corresponding parameter position in the query template to form a structured query statement.

[0053] Specifically, the user query text is parsed by a natural language parser constructed by ANTLR4. The natural language parser is pre-constructed based on the syntax rules of a structured query statement (such as SQL) to construct a syntax tree model, which can identify the core semantic intent (such as operation types such as "query", "statistics", "filter", etc.) in the query text and map it to a standard query template preset by the system. For example, when the query text is "query marketing activity participant number data", the natural language parser can accurately map it to the SQL standard query template "SELECT COUNT(*) FROM [table name] WHERE [activity identification field]=?". Subsequently, the named entity recognition algorithm is used to extract entities from the query text to identify the key entity information (such as "marketing activity" corresponding to the business entity and "participant number" corresponding to the statistical dimension entity), and the metadata (including table structure, field name, data type, etc.) of the target business database is called to match and associate the identified entity information with the database metadata, and the identified entity information is dynamically filled into the corresponding parameter position of the standard query template, for example, the activity_id field value associated with "marketing activity" is filled into the parameter position of "?" in the template, and finally a complete structured query statement (such as "SELECT COUNT(*) FROM activity_user_info WHERE activity_id = '2024_market_001'") is formed. In addition, after generating the structured query statement, the syntax of the statement can be checked by syntax tree analysis to ensure that it meets the execution specifications of the target database and ensures that the statement can be directly executed.

[0054] The ANTLR4 syntax parser is used to accurately map the query text to the standard template, avoiding template matching errors caused by natural language semantic ambiguity. At the same time, combined with named entity recognition and dynamic parameter filling of database metadata, the association between entities and fields does not need to be manually maintained, solving the tedious problem of manually adjusting parameters in traditional template matching, which not only improves the generation efficiency of structured query statements, but also ensures the adaptability of the statements to the target database, and guarantees the accuracy and effectiveness of the statement execution.

[0055] Under the technical solution framework disclosed in the foregoing embodiments, after converting the query text into an executable structured query statement, the embodiments of the present application further include: connecting to the database through the query executor, executing the structured query statement and obtaining the returned data result; converting the data result into a standardized data format required by the front-end visual chart component for graphical display.

[0056] Specifically, the embodiments of the present application realize data acquisition and visualization adaptation through two links of query executor and data format conversion. Among them, the specific implementation of the query executor adopts the JDBC technology in Java, first configures the connection parameters of the target business database (including database URL, username and password, such as URL format "jdbc:mysql: / / [database IP]:[port] / [database name]"), and then creates a database connection (DriverManager.getConnection(url, username, password)) and a Statement object in the query executor class (such as SQLExecutor class) through the try-with-resources syntax, calls the executeQuery() method of the Statement to execute the generated structured query statement (such as SQL statement), and obtains the returned ResultSet result set. In the process, the exception handling mechanism is used to capture and handle SQLException, so as to ensure the stability of database interaction. After obtaining the data result, the ResultSet result set is converted into a standardized JSON format that can be recognized by the Apache ECharts component according to the front-end visualization requirements: for example, when the query result is the statistical data of "SELECT region, COUNT(*), AVG(sales) FROM data GROUP BY region", each row of data in the ResultSet is traversed, the "region" field is extracted as "xAxis.data" (horizontal axis data) in JSON, and the "COUNT(*)" and "AVG(sales)" fields are extracted as two "series.data" (vertical axis data), respectively, to construct a JSON structure containing "xAxis", "series", "legend" and other key nodes, so that the converted data can be directly called by the front-end visualization chart component.

[0057] Through the JDBC implementation of the query executor, the stable execution of the structured query statement in the target database and the accurate acquisition of the data result are ensured, and the try-with-resources syntax guarantees the automatic release of the database connection and other resources, avoiding resource leakage; and for the standardized JSON format conversion of Apache ECharts, the secondary processing step of the original data in the front-end is saved, so that the data result can be directly used for graphical display, which not only improves the efficiency of data visualization, but also guarantees the accuracy of data display, helping business personnel to more intuitively understand the data statistical result.

[0058] It should be noted that the embodiments of the present disclosure can include a plurality of steps, which are numbered for the convenience of description, but these numbers are not a limitation on the execution time slot and execution order between the steps; these steps can be implemented in any order, and the embodiments of the present disclosure do not limit this.

[0059] Corresponding to the knowledge base query method described above, the present disclosure also proposes a knowledge base query device. Since the device embodiment of the present disclosure corresponds to the method embodiment described above, for the details not disclosed in the device embodiment, reference can be made to the method embodiment described above, which will not be described in detail in the present disclosure.

[0060] Figure 2 A structural schematic diagram of a knowledge base query device provided by the embodiments of the present disclosure is shown in Figure 2 , which includes: The construction unit 21 is configured to parse and structure process a plurality of format documents, and construct a multi-source heterogeneous data knowledge base. The generation unit 22 is configured to respond to a user query instruction, perform semantic understanding on query text in the user query instruction through a large language model deployed on a local server, and generate a vector representation thereof. The retrieval unit 23 is configured to perform semantic retrieval in the multi-source heterogeneous data knowledge base based on the vector representation. The optimization unit 24 is configured to perform conditional testing on the result of semantic retrieval, determine whether to perform retrieval enhancement processing, and trigger a structured query statement generation process when the result of semantic retrieval does not satisfy a confidence condition, and convert the query text into an executable structured query statement.

[0061] The present disclosure provides a knowledge base query device, which constructs a multi-source heterogeneous data knowledge base by parsing and structure processing a plurality of format documents, performs semantic understanding and vector representation generation of a user query instruction relying on a large language model deployed on a local server, performs semantic retrieval in the multi-source heterogeneous data knowledge base based on the vector representation, and performs conditional testing on the result of semantic retrieval to dynamically determine whether to trigger a structured query statement generation process, and converts the query text into an executable structured query statement when the confidence condition is not satisfied. Therefore, the problems of data privacy risk due to dependence on external large model services, lack of unified analysis and retrieval capability of multi-source heterogeneous data, and inability to generate structured query statements on demand in the prior art can be solved, and technical effects of guaranteeing enterprise data security compliance, improving multi-source data retrieval accuracy and efficiency, and meeting the demand for structured data acquisition in complex query scenarios can be achieved.

[0062] Further, in a possible implementation manner of the present embodiment, as shown in Figure 3 , the construction unit 21 includes: The first extraction module 211 is used to process documents of various formats through a text parsing engine and extract text content and its logical structure. The second extraction module 212 is used to access the version control system through the code parser, extract function definitions, related comments and embedded query statements in the code file, and establish a semantic association index between them and the natural language description.

[0063] Furthermore, in one possible implementation of this embodiment, the text parsing engine at least supports parsing portable document formats and word processing document formats, and is able to recognize and convert table structures in the document; The code parser obtains the target code repository by performing a cloning operation and parses the structured query language files and script files therein to build a mapping table between function names and functional descriptions.

[0064] Furthermore, in one possible implementation of this embodiment, such as Figure 3 As shown, it also includes: Compression unit 25 is used to quantize and compress the large language model, and then deploy the quantized and compressed large language model on a local server using a containerization algorithm. The large language model enables a parallel computing engine during inference and accelerates the computation process by setting tensor parallel parameters.

[0065] Furthermore, in one possible implementation of this embodiment, such as Figure 3 As shown, the retrieval unit 23 includes: The retrieval module 231 is used to retrieve the vector index of a multi-source heterogeneous data knowledge base based on a preset search algorithm; The judgment module 232 is used to preset the confidence condition as a set similarity threshold. When the highest similarity score of the retrieved content is lower than the threshold, it is determined that the confidence condition is not met.

[0066] Furthermore, in one possible implementation of this embodiment, such as Figure 3 As shown, the optimization unit 24 includes: Mapping module 241 is used to perform natural language parsing on query text using a syntax parser and map it into a standard query template; The forming module 242 is used to identify entity information from the query text using a named entity recognition algorithm, and dynamically fill the entity information into the corresponding parameter positions in the query template in combination with the metadata of the target database to form a structured query statement.

[0067] Furthermore, in one possible implementation of this embodiment, such as Figure 3 As shown, after converting the query text into an executable structured query statement, the process also includes: The execution unit 26 is configured to connect to a database through a query executor, execute a structured query language statement, and obtain returned data results; and convert the data results into a standardized data format required by a front-end visual chart component for graphical display.

[0068] It should be noted that the foregoing explanation of the method embodiments also applies to the device of the present embodiment, and the principle is the same, which is not limited in the present embodiment.

[0069] According to embodiments of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0070] Figure 4 A schematic block diagram of an example electronic device 300 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptops, desktops, tablets, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular telephones, smartphones, wearable devices, and other similar computing devices. The components shown here, their connections and relationships, and their functions, are meant to be examples only, and are not meant to limit implementations of the present disclosure described and / or claimed in this document.

[0071] As shown in Figure 4 The electronic device 300 includes a computing unit 301 that can perform various appropriate actions and processes according to a computer program stored in a ROM (Read-Only Memory) 302 or a computer program loaded into a RAM (Random Access Memory) 303 from a storage unit 308. Various programs and data required for the operation of the electronic device 300 can also be stored in the RAM 303. The computing unit 301, the ROM 302, and the RAM 303 are connected to each other through a bus 304. An I / O (Input / Output) interface 305 is also connected to the bus 304.

[0072] Various components in the electronic device 300 are connected to the I / O interface 305, including an input unit 306, such as a keyboard, a mouse, etc.; an output unit 307, such as various types of displays, a speaker, etc.; a storage unit 308, such as a magnetic disk, an optical disk, etc.; and a communication unit 309, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 309 allows the electronic device 300 to exchange information / data with other devices through a computer network, such as the Internet, and / or various telecommunication networks.

[0073] The computing unit 301 can be various general and / or special purpose processing components with processing and computing capabilities. Some examples of the computing unit 301 include, but are not limited to, a CPU (Central Processing Unit), a GPU (Graphic Processing Unit), various special-purpose AI (Artificial Intelligence) computing chips, various computing units running machine learning model algorithms, a DSP (Digital Signal Processor), and any appropriate processor, controller, microcontroller, etc. The computing unit 301 performs various methods and processes described above, such as the knowledge base query method. For example, in some embodiments, the knowledge base query method can be implemented as a computer software program tangibly embodied in a machine-readable medium, such as the storage unit 308. In some embodiments, part or all of the computer program can be loaded and / or installed onto the electronic device 300 via the ROM 302 and / or the communication unit 309. When the computer program is loaded onto the RAM 303 and executed by the computing unit 301, one or more steps of the methods described above can be performed. Alternatively, in other embodiments, the computing unit 301 can be configured to perform the aforementioned knowledge base query method by any other appropriate means, such as by means of firmware.

[0074] Various implementations of the systems and techniques described above herein can be realized in digital electronic circuitry, integrated circuitry, a Field Programmable Gate Array (FPGA), an Application-Specific Integrated Circuit (ASIC), an Application Specific Standard Product (ASSP), a System on a Chip (SOC), a Complex Programmable Logic Device (CPLD), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementation in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which can be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0075] Program code for carrying out methods of the present disclosure can be written in any combination of one or more programming languages. The program code can be provided to a processor or controller of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the program code, when executed by the processor or controller, produces the functions / operations specified in the flowcharts and / or the block diagrams. The program code can be executed entirely on a machine, partially on a machine, partially on a machine as a stand-alone software package, partially on a machine and partially on a remote machine or entirely on a remote machine or server.

[0076] In the context of the present disclosure, a machine-readable medium can be a tangible medium that can contain or store program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. Machine-readable storage medium can include but are not limited to an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of the machine-readable storage medium would include one or more lines of electrical wire, portable computer diskette, hard disk, RAM, ROM, EPROM (Electrically Programmable Read-Only-Memory) or flash memory, fiber optics, CD-ROM (Compact Disc Read-Only Memory), optical storage device, magnetic storage device, or any suitable combination of the foregoing.

[0077] To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having a display device (e.g., a CRT (Cathode-Ray Tube) or LCD (Liquid Crystal Display) monitor) for displaying information to the user and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, speech, or tactile input.

[0078] The systems and techniques described herein can be implemented in a computing system that includes a back end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front end component (e.g., a user computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a LAN (Local Area Network), a WAN (Wide Area Network), the Internet, and a blockchain network.

[0079] The computer system can include clients and servers. The clients and servers are generally remote from each other and typically interact through a communication network. The relationship of client and server is one of communication and distribution, with the server generally providing communication and distribution services to the clients. The servers can be cloud servers, also known as cloud computing servers or cloud hosts, which are a host product in the cloud computing service system to solve the defects of large management difficulty and weak business scalability in traditional physical hosts and VPS services ("Virtual Private Server", or simply "VPS"). The servers can also be servers of a distributed system, or servers combined with a blockchain.

[0080] It should be noted that artificial intelligence is a discipline that studies enabling computers to simulate some thinking processes and intelligent behaviors (such as learning, reasoning, thinking, planning, etc.) of people, both hardware and software technologies. Artificial intelligence hardware technologies generally include technologies such as sensors, special artificial intelligence chips, cloud computing, distributed storage, big data processing, etc.; artificial intelligence software technologies mainly include computer vision technology, speech recognition technology, natural language processing technology, and machine learning / deep learning, big data processing technology, knowledge graph technology, etc. several major directions.

[0081] The first, second, and various other numerical designations involved in the present disclosure are only for differentiation for convenience of description, and do not limit the scope of the embodiments of the present disclosure, nor represent a sequence.

[0082] At least one of the present disclosure can also be described as one or more, multiple can be two, three, four or more, the present disclosure does not make restrictions. In the embodiments of the present disclosure, for a technical feature, the technical features in the technical feature are distinguished by "first", "second", "third", "A", "B", "C" and "D" and the like. The technical features described by "first", "second", "third", "A", "B", "C" and "D" have no order or size order.

[0083] It should be understood that the steps shown above can be reordered, added or deleted. For example, the steps described in the present disclosure can be executed in parallel, sequentially or in different order, as long as the desired results of the technical solutions disclosed in the present disclosure can be achieved, which is not limited herein.

[0084] The above specific embodiments do not constitute a limitation on the protection scope of the present disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modifications, equivalent replacements and improvements made within the spirit and principles of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A method of querying a knowledge base, characterized by, The method comprises the following steps: parsing and structuring multiple format documents to build a multi-source heterogeneous data knowledge base; in response to a user query instruction, a large language model deployed on a local server is used to perform semantic understanding on the query text in the user query instruction and generate a vector representation thereof; based on the vector representation, semantic retrieval is performed in the multi-source heterogeneous data knowledge base; the results of the semantic retrieval are conditionally tested to determine whether to perform retrieval enhancement processing, and when the results of the semantic retrieval do not meet the confidence condition, a structured query statement generation process is triggered to convert the query text into an executable structured query statement.

2. The knowledge base query method of claim 1, wherein, The method comprises the following steps: processing multiple format documents through a text parsing engine to extract text content and its logical structure; a code parser accesses a version control system to extract function definitions, associated comments and embedded query statements in code files, and establishes a semantic association index thereof with natural language descriptions.

3. The knowledge base query method of claim 1, wherein, The method further comprises the following steps: quantizing and compressing the large language model, and deploying the quantized and compressed large language model on the local server through a containerization algorithm, wherein the large language model enables a parallel computing engine during inference and accelerates the computing process by setting tensor parallel parameters.

4. The knowledge base query method of claim 1, wherein, The method comprises the following steps: based on a preset search algorithm, the vector index of the multi-source heterogeneous data knowledge base is searched; the preset confidence condition is a set similarity threshold, and when the highest similarity score of the search return content is lower than the threshold, it is determined that the confidence condition is not met.

5. The knowledge base query method of claim 1, wherein, The method comprises the following steps: using a syntax parser to perform natural language parsing on the query text and mapping it to a standard query template; using a named entity recognition algorithm to identify entity information from the query text, and combining the metadata of the target database to dynamically fill the entity information into the corresponding parameter positions in the query template to form the structured query statement.

6. The knowledge base query method of claim 1, wherein, After converting the query text into an executable structured query statement, the method further comprises the following steps: connecting to a database through a query executor to execute the structured query statement and obtain the returned data results; converting the data results into a standardized data format required by a front-end visual chart component for graphical display.

7. A knowledge base query apparatus characterized by comprising: The method comprises the following steps: a construction unit for parsing and structuring multiple format documents to build a multi-source heterogeneous data knowledge base; a generation unit for responding to a user query instruction by using a large language model deployed on a local server to perform semantic understanding on the query text in the user query instruction and generate a vector representation thereof; a retrieval unit for performing semantic retrieval in the multi-source heterogeneous data knowledge base based on the vector representation; An optimization unit is configured to conditionally check the result of the semantic retrieval, determine whether to perform a retrieval enhancement process, and trigger a structured query statement generation process to convert the query text into an executable structured query statement when the result of the semantic retrieval does not satisfy a confidence condition.

8. An electronic device, comprising: Comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-6.

9. A non-transitory computer-readable storage medium having stored thereon computer instructions, wherein, The computer instructions are configured to cause the computer to perform the method of any one of claims 1-6.

10. A computer program product, characterised in that, A computer program that, when executed by a processor, implements the method of any one of claims 1-6.

Images