A lightweight knowledge base construction method and system based on hierarchical coding

Abstract

This invention provides a lightweight knowledge base construction method based on hierarchical coding. The method includes processing documents in a knowledge document library using a lightweight semantic coding model on a cloud server to generate corresponding high-dimensional semantic vectors. Each high-dimensional semantic vector is mapped to a fixed-length binary fingerprint using a pre-defined Locality Sensitive Hash (LSH) function cluster, constructing a global fingerprint index library. On a mobile terminal, fingerprint index data from the cloud server is received and stored. Responding to user-inputted query text, a query fingerprint is generated using the same lightweight semantic coding model and LSH function cluster as on the cloud. By calculating the similarity between the query fingerprint and the fingerprints in the locally stored fingerprint index data, relevant document information is retrieved, and the results are returned. The beneficial effects of this invention are that it can reduce the knowledge base storage overhead by more than an order of magnitude while keeping the accuracy loss of semantic retrieval within 5%.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence computing technology, and in particular to a lightweight knowledge base construction method and system based on hierarchical coding. Background Technology

[0002] With the widespread adoption of mobile smart devices, knowledge-intensive applications (such as smart assistants, legal consultations, and medical Q&A) are becoming increasingly popular. The effectiveness of these applications heavily relies on the ability to quickly and accurately retrieve relevant knowledge locally to provide immediate and accurate services, while reducing reliance on cloud queries and network latency.

[0003] Currently, mobile knowledge retrieval mainly relies on two types of technologies: one is keyword-based inverted indexing, which has the advantages of high retrieval efficiency and low storage consumption, but cannot understand semantics, resulting in poor relevance of search results; the other is semantic indexing technology based on deep learning, especially pre-trained models represented by BERT, which can convert text into high-dimensional semantic vectors and achieve accurate semantic matching. However, the original BERT model has a huge number of parameters and cannot be directly deployed on mobile devices. To address this, the industry has adopted model compression techniques, such as knowledge distillation (DistilBERT), pruning, quantization, and dedicated lightweight architectures (such as MobileBERT), to reduce the computational and storage overhead of the model. In addition, near-nearest neighbor search algorithms such as Locality Sensitive Hashing (LSH) are often used to accelerate high-dimensional vector retrieval.

[0004] Nevertheless, existing solutions still face significant challenges in building dynamic, large-scale knowledge bases for mobile devices: 1. The contradiction between storage overhead and accuracy loss is prominent. Although the parameters of the lightweight model (such as MobileBERT) are reduced after distillation or pruning, the generated semantic vector dimension is still high (usually hundreds of dimensions). Directly storing and indexing the vectors corresponding to massive documents will cause mobile storage space to be exhausted quickly. If crude quantization or dimensionality reduction is further adopted, it will bring unacceptable loss of semantic information (accuracy loss often exceeds 10%), affecting the retrieval quality. 2. The update mechanism is inefficient. The knowledge base needs to be continuously updated to maintain its timeliness. Existing solutions usually require the complete index or model to be downloaded from the cloud periodically. Even if only a small amount of content changes, for example, a 1GB knowledge base, even if only 1% of the content is updated, may require the transfer of tens to hundreds of MB of data, resulting in huge waste of network bandwidth and update delays, and it is difficult to support real-time synchronization. 3. Fragmented system design: Existing methods often optimize the "semantic vector generation model" and the "vector indexing mechanism" as two independent modules, lacking an integrated design. This fragmented design makes it difficult for the overall system to achieve global optimization in multiple dimensions such as model accuracy, index compression rate, and update efficiency, and also increases the system complexity.

[0005] Therefore, there is an urgent need for a new knowledge base construction solution that can simultaneously achieve high-precision semantic retrieval, extremely low storage consumption, and efficient incremental updates in a mobile environment. Summary of the Invention

[0006] To address the problems in existing technologies, this invention provides a lightweight knowledge base construction method based on hierarchical coding. Through a hierarchical coding strategy combining "lightweight semantic coding" and "locality-sensitive hash compression," high-dimensional semantic information is efficiently compressed into fixed-length binary fingerprints. An incremental update mechanism is designed to synchronize only the fingerprint differences of changed documents. This achieves a reduction in knowledge base storage overhead by more than an order of magnitude while keeping the accuracy loss of semantic retrieval within 5%, forming a lightweight, compact, and efficient closed loop for mobile knowledge retrieval. This solves the "accuracy-storage-update" triangle problem that exists in existing knowledge base construction technologies.

[0007] The present invention provides a lightweight knowledge base construction method based on hierarchical coding, applied to a system consisting of a cloud server and a mobile terminal, comprising the following steps: Step 1: Cloud-based offline construction service. On the cloud server, a lightweight semantic coding model is used to process the documents in the knowledge document library, generating high-dimensional semantic vectors for each document. The high-dimensional semantic vectors are mapped to fixed-length binary fingerprints through a pre-defined cluster of Local Sensitive Hash (LSH) functions. A global fingerprint index library is built based on the binary fingerprints of all documents. Step 2: Deploy and provide online retrieval services on mobile devices. On the mobile device, fingerprint index data from the cloud server is received and stored. In response to the query text entered by the user, a query fingerprint is generated using the same lightweight semantic coding model and Locality Sensitive Hash (LSH) function cluster as in the cloud. By calculating the similarity between the query fingerprint and the fingerprints in the locally stored fingerprint index data, relevant document information is retrieved, and the results are finally returned.

[0008] The present invention is further improved in that, in step 1, the lightweight semantic encoding model is a model obtained by processing the original BERT model or its variants through at least one of knowledge distillation, model pruning or quantization.

[0009] The present invention is further improved in that, in step 2, the similarity between the query fingerprint and the fingerprint in the locally stored fingerprint index data is calculated, specifically by calculating the Hamming distance between the query fingerprint and the fingerprint in the locally stored fingerprint index data.

[0010] The present invention is further improved in that, in step 2, relevant document information is retrieved. Specifically, one or more target fingerprints that are most similar are determined based on the Hamming distance between the queried fingerprint and the fingerprints in the locally stored fingerprint index data, and the document identifier or document summary associated with the target fingerprint is returned.

[0011] The present invention is further improved by including an incremental update mechanism in steps 1 and 2. The cloud server continuously monitors change events in the knowledge document base, recalculates the corresponding binary fingerprint for the changed document, and generates an incremental update package based on the difference between the recalculated fingerprint and the historical fingerprint, which is then sent to the mobile terminal. The mobile terminal receives the incremental update package and merges and updates the fingerprint index data stored locally according to the incremental update package.

[0012] In a further improvement to this invention, the change events of the knowledge document base include the addition, deletion or content modification of documents, and the incremental update package includes a change type identifier, a new fingerprint of the changed document and version number information.

[0013] The present invention is further improved such that the triggering condition for the mobile terminal to receive the incremental update package is that the network connection is detected as Wi-Fi, the mobile terminal is in an idle state, and a push notification is received from the cloud server.

[0014] This invention also provides a system consisting of a cloud server and a mobile terminal for implementing the aforementioned lightweight knowledge base construction method based on hierarchical coding. The system is characterized by comprising a cloud subsystem and a mobile terminal system, wherein... The cloud subsystem includes: The semantic encoding module is used to process documents in the knowledge document repository using a lightweight semantic encoding model to generate high-dimensional semantic vectors. The fingerprint generation module is used to map the high-dimensional semantic vector into a fixed-length binary fingerprint using a preset cluster of Local Sensitive Hash (LSH) functions. The index building module is used to build a global fingerprint index library based on the binary fingerprints of all documents; The update management module is used to monitor changes in the knowledge document base and generate incremental update packages. When the knowledge document base changes, the processing flow of the semantic encoding module and fingerprint generation module is re-executed only for the changed documents, and the differences between the new fingerprint and the old fingerprint, as well as the change operation type, are packaged to generate an incremental update package. The mobile terminal system includes: The local storage module is used to store fingerprint index data from the cloud subsystem; The query processing module is used to respond to user queries and generates query fingerprints using the same lightweight semantic coding model and Locality Sensitive Hash (LSH) function family. The retrieval module is used to retrieve documents by calculating the similarity between the query fingerprint and the local fingerprint index data; The update synchronization module is used to receive and apply incremental update packages.

[0015] The beneficial effects of this invention are as follows: This invention provides a lightweight knowledge base construction method based on hierarchical coding. Through a hierarchical coding strategy of "lightweight semantic coding" and "locality-sensitive hash compression," high-dimensional semantic information is efficiently compressed into fixed-length binary fingerprints, achieving an order-of-magnitude reduction in storage overhead (compressed to 1%-3% of the original vector storage). At the same time, due to the similarity preservation characteristics of LSH, the accuracy loss of semantic retrieval is strictly controlled at a low level (within 5%). By designing an incremental update mechanism, only the fingerprint differences of changed documents are synchronized, reducing the update traffic by more than an order of magnitude (more than 90%), achieving efficient near real-time knowledge synchronization. The integrated design of coding, indexing, and updating forms a lightweight, compact, and efficient mobile knowledge retrieval closed loop, which has extremely high practical value and commercial prospects, and solves the problem of the "accuracy-storage-update" triangle contradiction in the construction of knowledge bases in the prior art. Attached Figure Description

[0016] Figure 1 This is a flowchart of a lightweight knowledge base construction method based on hierarchical coding according to the present invention. Detailed Implementation

[0017] The present invention will now be described in further detail with reference to the accompanying drawings and embodiments.

[0018] Please see Figure 1 The present invention discloses a lightweight knowledge base construction method based on hierarchical coding, applied to a system consisting of a cloud server and a mobile terminal, comprising the following steps: Step 1: Cloud-based offline construction service. On the cloud server, a lightweight semantic coding model is used to process the documents in the knowledge document library, generating high-dimensional semantic vectors for each document. Each high-dimensional semantic vector is mapped to a fixed-length binary fingerprint using a pre-defined Locality Sensitive Hash (LSH) function cluster. A global fingerprint index library is built based on the binary fingerprints of all documents. The lightweight semantic coding model is a model obtained by processing the original BERT model or its variants through at least one of knowledge distillation, model pruning, or quantization.

[0019] Step 2: Deploy and provide online retrieval services on the mobile device. On the mobile terminal, fingerprint index data from the cloud server is received and stored. Responding to user-inputted query text, a query fingerprint is generated using the same lightweight semantic coding model and Locality Sensitive Hash (LSH) function family as in the cloud. By calculating the similarity between the query fingerprint and fingerprints in the locally stored fingerprint index data, relevant document information is retrieved, and the final result is returned. Specifically, calculating the similarity between the query fingerprint and fingerprints in the locally stored fingerprint index data involves calculating the Hamming distance between them. Retrieving relevant document information involves identifying one or more most similar target fingerprints based on the Hamming distance between the query fingerprint and fingerprints in the locally stored fingerprint index data, and returning the document identifier or document summary associated with the target fingerprint.

[0020] Please see Figure 1 In steps 1 and 2, an incremental update mechanism is also included. The cloud server continuously monitors change events in the knowledge document repository. For documents that have been changed, it recalculates their corresponding binary fingerprints. Based on the difference between the recalculated fingerprint and the historical fingerprint, an incremental update package is generated and sent to the mobile terminal. The mobile terminal receives the incremental update package and merges and updates the fingerprint index data stored locally according to the incremental update package. The change events in the knowledge document repository include the addition, deletion, or modification of documents. The incremental update package contains a change type identifier, the new fingerprint of the changed document, and version number information. The trigger conditions for the mobile terminal to receive the incremental update package are: detecting a Wi-Fi network connection, being in an idle state, and receiving a push notification from the cloud server.

[0021] Please see Figure 1 In this embodiment, the entire system consists of two main parts: "cloud-based offline building service" and "mobile-based deployment and online retrieval service".

[0022] The cloud-based offline build service includes: 1. The original knowledge document repository contains structured or unstructured text data that needs to be retrieved, such as legal provisions, medical literature, encyclopedic knowledge, etc. 2. Prune BERT semantic encoding: Use a pruned, lightweight BERT model (e.g., retain the core Transformer layers, remove some redundant layers and attention heads, reducing the size to about 30% of the original model) to encode each document, outputting a high-dimensional (e.g., 768-dimensional) semantic vector. This vector captures the deep semantic information of the document; 3. LSH hash mapping uses a cluster of Locality Sensitive Hash (LSH) functions (e.g., k) to process the input high-dimensional semantic vector. Each LSH function outputs a binary value (0 or 1) based on the projection of the vector onto a random hyperplane. The outputs of the k LSH functions are concatenated to obtain a binary hash fingerprint of fixed length k (e.g., 512 bits). The properties of LSH functions ensure that semantically similar documents generate fingerprints with high similarity (small Hamming distance). 4. Generate a global fingerprint index library. After generating fingerprints for all documents, establish a mapping relationship of "fingerprint -> document ID / summary" to form a global index library. The size of this index library is much smaller than the size of storing all high-dimensional vectors.

[0023] Mobile deployment and online search services include: 1. Local fingerprint index library: After initialization or update, the mobile application locally stores (all or part) of the fingerprint index library downloaded from the cloud. For example, a knowledge base containing millions of documents may only require 500MB of storage space for its fingerprint index.

[0024] 2. The query retrieval module, when the user enters query text (such as a question or keywords), uses the same trimmed BERT model and Locality Sensitive Hash (LSH) function family as the cloud-based module to convert the query text into a query fingerprint. Subsequently, this module quickly calculates the Hamming distance (i.e., the number of distinct binary bits) between the query fingerprint and each fingerprint in the local fingerprint index. Hamming distance calculation is extremely efficient. Finally, it returns the document IDs or summaries corresponding to the N fingerprints with the smallest Hamming distances as the retrieval results.

[0025] 3. The incremental update engine is responsible for keeping synchronized with the cloud. The cloud service continuously monitors changes (additions, deletions, and modifications) to the knowledge document repository. Once a change is detected, the cloud only re-encodes the changed document using "semantic encoding -> hash mapping," calculates a new fingerprint, compares it with the old version, and generates an "incremental update package" (Delta index) containing only the change information. The mobile update engine downloads this small incremental package at appropriate times (such as when connected to Wi-Fi) and merges it into the local fingerprint index library, for example, replacing the old fingerprint of a modified document, adding a fingerprint of a new document, or marking the fingerprint of a deleted document. This allows daily updates to potentially only transfer a few megabytes of data, rather than hundreds of megabytes of a complete index.

[0026] The semantic encoding layer is not limited to pruned BERT; it can also use DistilBERT, TinyBERT obtained through knowledge distillation, or models specifically designed for mobile devices such as MobileBERT and ALBERT, or future lightweight architectures. The hash compression layer is not limited to classic LSH; any binary hashing algorithm that can maintain the similarity relationship of high-dimensional vectors can be used, such as SimHash and Spectral Hashing. The incremental update strategy allows for flexible adjustment of the generation and synchronization strategies for incremental packets. For example, generation can be triggered by a fixed time window (e.g., hourly), a certain amount of accumulated changes (e.g., 100 accumulated changes), or more complex version trees and differential algorithms can be used to handle concurrent updates and version rollbacks.

[0027] Please see Figure 1 The present invention also provides a system consisting of a cloud server and a mobile terminal for implementing the above-mentioned lightweight knowledge base construction method based on hierarchical coding, characterized in that it includes a cloud subsystem and a mobile terminal system, wherein... The cloud subsystem includes: The semantic encoding module is used to process documents in the knowledge document repository using a lightweight semantic encoding model to generate high-dimensional semantic vectors. The fingerprint generation module is used to map high-dimensional semantic vectors into fixed-length binary fingerprints using a preset set of Local Sensitive Hash (LSH) functions. The index building module is used to build a global fingerprint index library based on the binary fingerprints of all documents; The update management module is used to monitor changes in the knowledge document base and generate incremental update packages. When the knowledge document base changes, the processing flow of the semantic encoding module and fingerprint generation module is re-executed only for the changed documents, and the differences between the new fingerprint and the old fingerprint, as well as the change operation type, are packaged to generate an incremental update package. The mobile terminal system includes: The local storage module is used to store fingerprint index data from the cloud subsystem; The query processing module is used to respond to user queries and generates query fingerprints using the same lightweight semantic coding model and Locality Sensitive Hash (LSH) function family. The retrieval module is used to retrieve documents by calculating the similarity between the query fingerprint and the local fingerprint index data; The update synchronization module is used to receive and apply incremental update packages.

[0028] In this embodiment, a mobile legal case retrieval app is used as an example. A cloud server stores the full text of millions of legal cases. Through an offline build service, 512-bit binary fingerprints of all cases are generated, forming a global index. Upon initial launch after installation, the app downloads approximately 500MB of fingerprint index data to the user's local device. Subsequently, the user can input natural language questions such as "How should an employee compensate if they are dismissed without cause?" in an offline environment. The app quickly generates a query fingerprint locally and, through Hamming distance comparison, returns the most relevant case titles and summaries from the millions of indexed cases within seconds. Every day, a small number of new cases are added to the cloud database or old cases are revised. The cloud generates an incremental package (approximately 10MB) containing only the fingerprints of these changed cases. The app automatically downloads and silently merges this incremental package when connected to Wi-Fi at night, completing the knowledge base update. The user can retrieve the latest content the following day.

[0029] As can be seen from the above, the beneficial effects of the present invention are as follows: The lightweight knowledge base construction method based on hierarchical coding provided by the present invention efficiently compresses high-dimensional semantic information into fixed-length binary fingerprints through a hierarchical coding strategy of "lightweight semantic coding" and "locality-sensitive hash compression," achieving an order-of-magnitude reduction in storage overhead (compressed to 1%-3% of the original vector storage). At the same time, due to the similarity preservation characteristics of LSH, the accuracy loss of semantic retrieval is strictly controlled at a low level (within 5%). By designing an incremental update mechanism, only the fingerprint differences of changed documents are synchronized, reducing the update traffic by more than an order of magnitude (reduced by more than 90%), achieving efficient near real-time knowledge synchronization. The integrated collaborative design of coding, indexing, and updating forms a lightweight, compact, and efficient mobile knowledge retrieval closed loop, which has extremely high practical value and commercial prospects, and solves the problem of the "accuracy-storage-update" triangle contradiction in the construction of knowledge bases in the prior art.

[0030] The specific embodiments described above are preferred embodiments of the present invention and are not intended to limit the specific scope of the present invention. The scope of the present invention includes, but is not limited to, these specific embodiments. All equivalent changes made in accordance with the present invention are within the protection scope of the present invention.

Claims

1. A lightweight knowledge base construction method based on hierarchical coding, applied to a system consisting of a cloud server and a mobile terminal, characterized in that, Includes the following steps: Step 1: Cloud-based offline construction service. On the cloud server, a lightweight semantic coding model is used to process the documents in the knowledge document library, generating high-dimensional semantic vectors for each document. The high-dimensional semantic vectors are mapped to fixed-length binary fingerprints through a pre-defined cluster of Local Sensitive Hash (LSH) functions. A global fingerprint index library is built based on the binary fingerprints of all documents. Step 2: Deploy and provide online retrieval services on mobile devices. On the mobile device, fingerprint index data from the cloud server is received and stored. In response to the query text entered by the user, a query fingerprint is generated using the same lightweight semantic coding model and Locality Sensitive Hash (LSH) function cluster as in the cloud. By calculating the similarity between the query fingerprint and the fingerprints in the locally stored fingerprint index data, relevant document information is retrieved, and the results are finally returned.

2. The lightweight knowledge base construction method based on hierarchical coding as described in claim 1, characterized in that: In step 1, the lightweight semantic encoding model is a model obtained by processing the original BERT model or its variants through at least one of knowledge distillation, model pruning, or quantization.

3. The lightweight knowledge base construction method based on hierarchical coding as described in claim 2, characterized in that: In step 2, the similarity between the query fingerprint and the fingerprint in the locally stored fingerprint index data is calculated, specifically by calculating the Hamming distance between the query fingerprint and the fingerprint in the locally stored fingerprint index data.

4. The lightweight knowledge base construction method based on hierarchical coding as described in claim 3, characterized in that: In step 2, relevant document information is retrieved. Specifically, one or more target fingerprints that are most similar are determined based on the Hamming distance between the queried fingerprint and the fingerprints in the locally stored fingerprint index data, and the document identifier or document summary associated with the target fingerprint is returned.

5. The lightweight knowledge base construction method based on hierarchical coding as described in claim 4, characterized in that: In steps 1 and 2, an incremental update mechanism is also included. The cloud server continuously monitors change events in the knowledge document base. For documents that have been changed, the corresponding binary fingerprint is recalculated. Based on the difference between the recalculated fingerprint and the historical fingerprint, an incremental update package is generated and sent to the mobile terminal. The mobile terminal receives the incremental update package and merges and updates the fingerprint index data stored locally based on the incremental update package.

6. The lightweight knowledge base construction method based on hierarchical coding as described in claim 5, characterized in that: The change events in the knowledge document base include the addition, deletion, or modification of documents. The incremental update package contains a change type identifier, a new fingerprint of the changed document, and version number information.

7. The lightweight knowledge base construction method based on hierarchical coding as described in claim 6, characterized in that: The triggering conditions for a mobile terminal to receive the incremental update package are: detecting a Wi-Fi network connection, being in an idle state, and receiving a push notification from a cloud server.

8. A system comprising a cloud server and a mobile terminal, used to implement the lightweight knowledge base construction method based on hierarchical coding as described in any one of claims 1-7, characterized in that, Including cloud subsystems and mobile terminal systems, among which, The cloud subsystem includes: The semantic encoding module is used to process documents in the knowledge document repository using a lightweight semantic encoding model to generate high-dimensional semantic vectors. The fingerprint generation module is used to map the high-dimensional semantic vector into a fixed-length binary fingerprint using a preset cluster of Local Sensitive Hash (LSH) functions. The index building module is used to build a global fingerprint index library based on the binary fingerprints of all documents; The update management module is used to monitor changes in the knowledge document base and generate incremental update packages. When the knowledge document base changes, the processing flow of the semantic encoding module and fingerprint generation module is re-executed only for the changed documents, and the differences between the new fingerprint and the old fingerprint, as well as the change operation type, are packaged to generate an incremental update package. The mobile terminal system includes: The local storage module is used to store fingerprint index data from the cloud subsystem; The query processing module is used to respond to user queries and generates query fingerprints using the same lightweight semantic coding model and Locality Sensitive Hash (LSH) function family. The retrieval module is used to retrieve documents by calculating the similarity between the query fingerprint and the local fingerprint index data; The update synchronization module is used to receive and apply incremental update packages.

Images