Storage and retrieval system based on knowledge base and data processing method
By converting the knowledge base into compact video file storage and searching using semantic vector models, the complex and cost-effective problems of traditional knowledge base systems are solved, and rapid retrieval and compressed storage are achieved with personal controllable and low-cost.
Patent Information
- Application Number
- CN202510934863.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-07-08
- Publication Date
- 2025-08-12
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
In the prior art, the search and storage systems of traditional knowledge bases mostly serve large-scale people, and lack solutions that are simple to use, low cost and fully controllable in individuals. In particular, the needs of personal knowledge bases and home knowledge bases have not been met.
A knowledge base-based storage and search system is designed, including a knowledge storage module, a knowledge carrier module, an index carrier module, a search processing module, an index search module and a knowledge search module. By generating metadata and converting it into a video file for storage, the semantic vector model is used for searching, and the rapid search and compressed storage are realized.
It realizes compressed storage of million-level text blocks, providing fast retrieval and sub-second retrieval speed, simple to use, low cost, full control for individuals, and no professional skills installation and maintenance required.
Smart Images
Figure CN120470145A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of data processing technology, and in particular to a storage and retrieval system based on a knowledge base and a data processing method. Background Art
[0002] A knowledge base, like a library, consists of two components: knowledge storage and knowledge retrieval. Traditional knowledge base solutions utilize relational or vector databases for storage and retrieval, typically hosted in the cloud and leased to individuals. This solution is designed for large-scale knowledge bases or for serving large populations, requiring database servers and specialized computer personnel for installation and maintenance. This is costly and complex. Crucially, storing personal knowledge on public cloud services poses a challenge: ensuring personal privacy is not a guarantee.
[0003] With the promotion of AI applications, there is an urgent need for more and more small knowledge bases, such as personal knowledge bases and family knowledge bases, which require solutions that are simple to use, low-cost, and fully controllable by individuals.
[0004] Therefore, the existing technology still needs to be improved and developed. Summary of the Invention
[0005] The main purpose of the present invention is to provide a storage and retrieval system and data processing method based on a knowledge base, aiming to solve the problem in the prior art that the retrieval and storage of traditional knowledge bases mostly serve large groups of people and lack a storage and retrieval system that is simple to use, low cost, and fully controllable by individuals.
[0006] To achieve the above objectives, the present invention provides a knowledge base-based storage and retrieval system, the knowledge base-based storage and retrieval system comprising: A knowledge storage module, a knowledge carrier module, an index carrier module, a retrieval processing module, an index search module, and a knowledge search module, wherein the knowledge storage module is connected to the knowledge carrier module and the index carrier module respectively, the retrieval processing module is connected to the index search module and the knowledge search module respectively, and the index search module and the knowledge search module are connected to the index carrier module and the knowledge carrier module respectively; The knowledge storage module is used to receive input information from the knowledge storage interface module, process the input information, generate metadata, create an index for the metadata, send the index and the metadata to the index carrier module for storage, and obtain video data based on the metadata, and send the video data to the knowledge carrier module for storage; The retrieval processing module is used to receive the information to be processed from the knowledge retrieval interface module and send the information to be processed to the index search module; The index search module is used to generate a search vector according to the information to be processed, obtain a target video frame number from the index carrier module according to the search vector, and send the target video frame number to the search processing module; The retrieval processing module is further configured to send the target video frame number to the knowledge search module; The knowledge search module is used to read the target video data corresponding to the target video frame number from the knowledge carrier module, decode the target video data to obtain target text information, and send the target text information to the knowledge retrieval interface module for output.
[0007] Optionally, in the storage and retrieval system based on a knowledge base, the knowledge storage interface module is used to receive initial input data from other systems and pre-process the initial input data to obtain input information; Wherein, the initial input data includes a data list, a text file, a PDF document and an original text string; The knowledge storage interface module is further configured to receive operation results from the knowledge storage module and output the operation results.
[0008] Optionally, in the storage and retrieval system based on a knowledge base, the knowledge storage module includes: a storage processing unit, a knowledge generation unit, and an index generation unit, wherein the storage processing unit is connected to the knowledge generation unit and the index generation unit respectively; The storage processing unit is used to receive input information from the knowledge storage interface module and send the input information to the knowledge generation unit; The knowledge generation unit is configured to segment the input information according to a preset standard to obtain a plurality of text segments, set a video frame number for each of the text segments, generate metadata based on the plurality of text segments and the video frame number corresponding to each of the text segments, and send the metadata to the storage processing unit; The knowledge generation unit is further configured to generate a QR code image corresponding to each of the text segments in the metadata using QR code rules, treat each QR code image as a frame of video content, compress each frame of video content according to the format of the knowledge carrier module, and send all compressed frames of video content and corresponding video frame numbers to the knowledge carrier module for storage; The storage processing unit is further configured to send the metadata to the index generating unit; The index generation unit is used to convert all the text segments in the metadata into vector representations through a semantic vector model, use the vector representation of each text segment as an index item, and send each index item and the video frame number corresponding to each index item to the index carrier module for storage.
[0009] Optionally, in the storage and retrieval system based on a knowledge base, the knowledge retrieval interface module is used to receive initial information to be processed from other systems, and pre-process the initial information to be processed to obtain information to be processed; The knowledge retrieval interface module is further configured to receive the target text information from the retrieval processing module, and format the target text information based on the format of the information to be processed, thereby obtaining and outputting the final retrieval result.
[0010] Optionally, in the storage and retrieval system based on a knowledge base, the index search module includes: a first information receiving unit and a retrieval vector generating unit; The first information receiving unit is configured to receive the information to be processed sent by the search processing module, and send the information to be processed to the search vector generating unit; The retrieval vector generating unit is configured to receive the information to be processed, generate a retrieval vector based on the information to be processed using a semantic vector model, and send the retrieval vector to the index carrier module; The retrieval vector generation unit is further configured to receive the target video frame number from the index carrier module and send the target video frame number to the retrieval processing module.
[0011] Optionally, in the storage and retrieval system based on a knowledge base, the knowledge search module comprises: a second information receiving unit, a video decompression unit, and a decoding unit, wherein the second information receiving unit, the video decompression unit, and the decoding unit are connected in sequence; The second information receiving unit is used to receive the target video frame number from the retrieval processing module and send the target video frame number to the knowledge carrier module; The video decompression unit is used to receive the target video data from the knowledge carrier module, decompress the target video data to obtain the two-dimensional code image data, and send the two-dimensional code image data to the decoding unit; The decoding unit is used to receive the two-dimensional code image data, decode the two-dimensional code image data according to QR code rules, obtain target text information, and send the target text information to the knowledge retrieval interface module for output.
[0012] Optionally, in the storage and retrieval system based on the knowledge base, the index carrier module includes: an index storage unit and an index reading unit; The index storage unit is used to receive each of the index items and the video frame number corresponding to each of the index items from the index generation unit, and store each of the index items and the video frame number corresponding to each of the index items; The index reading unit is used to receive the retrieval vector, perform a similarity search based on the retrieval vector according to a preset threshold, obtain a target index item whose similarity meets the preset threshold, obtain a corresponding target video frame number based on the target index item, and send the target video frame number to the index search module.
[0013] Optionally, in the storage and retrieval system based on the knowledge base, the knowledge carrier module includes: a knowledge storage unit and a knowledge reading unit; The knowledge storage unit is used to receive all video content and the video frame number corresponding to each video content, and store all video content and the video frame number corresponding to each video content in the form of an MP4 file; The knowledge reading unit is used to receive the target video frame number from the knowledge search module, read out the corresponding text segment according to the target video frame number, use the corresponding text segment as target video data, and send the target video data to the knowledge search module.
[0014] In addition, to achieve the above-mentioned object, the present invention further provides a data processing method for a storage and retrieval system based on a knowledge base, wherein the data processing method comprises: The knowledge storage module receives input information from the knowledge storage interface module, processes the input information, and generates metadata; The knowledge storage module creates an index for the metadata, sends the index and the metadata to the index carrier module for storage, obtains video data based on the metadata, and sends the video data to the knowledge carrier module for storage; The retrieval processing module receives the information to be processed from the knowledge retrieval interface module and sends the information to be processed to the index search module; The index search module generates a search vector according to the information to be processed, obtains a target video frame number from the index carrier module according to the search vector, and sends the target video frame number to the search processing module; The retrieval processing module sends the target video frame number to the knowledge search module, and the knowledge search module reads the target video data corresponding to the target video frame number from the knowledge carrier module, decodes the target video data, obtains target text information, and sends the target text information to the knowledge retrieval interface module for output.
[0015] Optionally, the data processing method of the storage and retrieval system based on the knowledge base, wherein the knowledge storage module creates an index for the metadata, sends the index and the metadata to the index carrier module for storage, obtains video data based on the metadata, and sends the video data to the knowledge carrier module for storage, specifically includes: The knowledge storage module converts all text segments in the metadata into vector representations through a semantic vector model, uses the vector representation of each text segment as an index item, and sends each index item and the video frame number corresponding to each index item to the index carrier module for storage; The knowledge storage module generates QR code images corresponding to all text segments in the metadata through QR code rules, takes each QR code image as a frame of video content, compresses each frame of video content according to the format of the knowledge carrier module, and sends all compressed frames of video content and corresponding video frame numbers to the knowledge carrier module for storage.
[0016] In the present invention, the knowledge storage module receives input information from the knowledge storage interface module, processes the input information, and generates metadata; the knowledge storage module creates an index for the metadata, sends the index and metadata to the index carrier module for storage, obtains video data based on the metadata, and sends the video data to the knowledge carrier module for storage; the retrieval processing module receives the information to be processed from the knowledge retrieval interface module, and sends the information to be processed to the index search module; the index search module generates a retrieval vector based on the information to be processed, obtains the target video frame number from the index carrier module based on the retrieval vector, and sends the target video frame number to the retrieval processing module; the retrieval processing module sends the target video frame number to the knowledge search module, the knowledge search module reads the target video data corresponding to the target video frame number from the knowledge carrier module, decodes the target video data, obtains the target text information, and sends the target text information to the knowledge retrieval interface module for output. The present invention compresses the knowledge base into a compact video file while maintaining instant access to any information. It is simple to use, does not require professional skills for installation and maintenance, is low cost, and individuals can have full control over their own knowledge. It realizes compressed storage of millions of text blocks and can also provide fast retrieval and sub-second retrieval speed. BRIEF DESCRIPTION OF THE DRAWINGS
[0017] Figure 1 This is the overall architecture diagram of the knowledge base-based storage and retrieval system of the present invention; Figure 2 It is an architectural diagram of a knowledge storage module in the knowledge base-based storage and retrieval system of the present invention; Figure 3 It is a flow chart of a preferred embodiment of the data processing method of the knowledge base-based storage and retrieval system of the present invention. DETAILED DESCRIPTION
[0018] In order to make the purpose, technical solutions and advantages of the present invention more clear and distinct, the present invention is further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.
[0019] To solve the problems in the prior art, this embodiment provides a storage and retrieval system based on a knowledge base, such as Figure 1 As shown, the storage and retrieval system based on the knowledge base includes: a knowledge storage module, a knowledge carrier module, an index carrier module, a retrieval processing module, an index search module and a knowledge search module.
[0020] Among them, the knowledge storage module is connected to the knowledge carrier module and the index carrier module respectively, the retrieval processing module is connected to the index search module and the knowledge search module respectively, and the index search module and the knowledge search module are connected to the index carrier module and the knowledge carrier module respectively.
[0021] The knowledge storage module is used to receive input information from the knowledge storage interface module, process the input information, generate metadata, create an index for the metadata, send the index and the metadata to the index carrier module for storage, and obtain video data based on the metadata, and send the video data to the knowledge carrier module for storage; The retrieval processing module is used to receive the information to be processed from the knowledge retrieval interface module and send the information to be processed to the index search module; The index search module is used to generate a search vector according to the information to be processed, obtain a target video frame number from the index carrier module according to the search vector, and send the target video frame number to the search processing module; The retrieval processing module is further configured to send the target video frame number to the knowledge search module; The knowledge search module is used to read the target video data corresponding to the target video frame number from the knowledge carrier module, decode the target video data to obtain target text information, and send the target text information to the knowledge retrieval interface module for output.
[0022] Furthermore, the present invention also provides a data processing unit included in each module, which is used to perform corresponding data collection and processing operations.
[0023] The knowledge storage interface module is used to receive initial input data from other systems and pre-process the initial input data to obtain input information; wherein, the initial input data includes data lists, text files, PDF documents and original text strings; the knowledge storage interface module is also used to receive operation results from the knowledge storage module and output the operation results.
[0024] As you can understand, the knowledge storage interface module is responsible for receiving and initially processing various input data from other systems, including pre-processed data lists, text files, PDF documents, and raw text strings, and outputting text. It is also responsible for receiving responses from the storage processing unit, formatting and integrating them, and then returning them to the input party.
[0025] Further, if Figure 2 As shown, the knowledge storage module includes: a storage processing unit, a knowledge generation unit, and an index generation unit, and the storage processing unit is connected to the knowledge generation unit and the index generation unit respectively.
[0026] The storage processing unit is used to receive input information from the knowledge storage interface module and send the input information to the knowledge generation unit.
[0027] The knowledge generation unit is used to segment the input information according to preset standards to obtain multiple text segments, set a video frame number for each text segment, generate metadata based on the multiple text segments and the video frame number corresponding to each text segment, and send the metadata to the storage processing unit.
[0028] The knowledge generation unit is also used to generate QR code images corresponding to all text segments in the metadata through QR code rules, use each QR code image as a frame of video content, compress each frame of video content according to the format of the knowledge carrier module, and send all compressed frames of video content and corresponding video frame numbers to the knowledge carrier module for storage.
[0029] The storage processing unit is further configured to send the metadata to the index generating unit.
[0030] The index generation unit is used to convert all the text segments in the metadata into vector representations through a semantic vector model, use the vector representation of each text segment as an index item, and send each index item and the video frame number corresponding to each index item to the index carrier module for storage.
[0031] Specifically, the storage processing unit is used to receive input information from the knowledge storage interface module, send the input information to the knowledge generation unit, and wait for the knowledge generation unit to return metadata. At the same time, the storage processing unit is also used to hand over the metadata to the index generation unit, wait for the index to be returned, and finally reply the index to the knowledge storage interface unit.
[0032] The knowledge generation unit is used to receive data from the storage and processing unit, segment the input data, and generate text segments one by one, and each text segment has a unique ID, called a text segment ID. Among them, the standard for segmenting the input data is: not exceeding the capacity standard of the QR code format. The capacity standard of the QR code format is: digital mode, up to 7089 numbers; alphanumeric mode, up to 4296 characters; byte mode, up to 2953 bytes; Chinese character mode, up to 1817 Chinese characters. In this embodiment, 1000 bytes is preferred, so that the QR code generation and decoding speed can be taken into account.
[0033] In this embodiment, the knowledge generation unit creates a new unique video frame number for each text segment. This video frame number represents a unique storage location of the knowledge carrier unit. Metadata is generated based on the multiple text segments and the video frame number corresponding to each text segment, and the metadata is sent to the storage processing unit. The metadata includes: the text segment and the corresponding video frame number.
[0034] The knowledge generation unit is further configured to segment the text into segments and generate a QR code image according to QR code (Quick Response Code) rules. This QR code image is used as the video content to generate a frame of video content. The video content is then compressed according to the format of the knowledge carrier unit, for example, into a frame of MP4 video. The corresponding position of the knowledge carrier unit is found according to the video frame number, and the compressed frame of video is written.
[0035] The index generating unit is used to receive data (metadata) from the storage processing unit and perform an index creation operation on the metadata.
[0036] The index creation process is as follows: Using a semantic vector model, the metadata text segments are converted into vectors. For example, the SentenceTransformer model can be used to convert each text segment into a vector representation. Next, the generated vector representation is used to determine a unique storage location in the index carrier unit (e.g., at the end of the index carrier unit). The vector is written into the index carrier unit as an index item, along with its corresponding video frame number. Finally, the operation result (success or failure) is returned to the storage processing unit.
[0037] Furthermore, the knowledge retrieval interface module is used to receive initial information to be processed from other systems, and pre-process the initial information to be processed to obtain information to be processed.
[0038] The knowledge retrieval interface module is further configured to receive the target text information from the retrieval processing module, and format the target text information based on the format of the information to be processed, thereby obtaining and outputting the final retrieval result.
[0039] It can be understood that the knowledge retrieval interface module is responsible for receiving and preliminarily processing various information to be processed from other systems, and sending the information to be processed to the retrieval processing unit, wherein the information to be processed includes: preprocessed data lists, text files, PDF documents and original text strings.
[0040] At the same time, the knowledge retrieval interface module is also responsible for receiving replies from the retrieval processing unit and formatting and integrating them: formatting according to the input data format and the content format of the retrieval processing unit's reply, for example, converting the picture replied by the retrieval processing unit into a marked text format according to the format requirements of the input data, and replying to the input party.
[0041] Furthermore, the index search module includes: a first information receiving unit and a retrieval vector generating unit.
[0042] The first information receiving unit is configured to receive the information to be processed sent by the search processing module, and send the information to be processed to the search vector generating unit; The retrieval vector generating unit is configured to receive the information to be processed, generate a retrieval vector based on the information to be processed using a semantic vector model, and send the retrieval vector to the index carrier module; The retrieval vector generation unit is further configured to receive the target video frame number from the index carrier module and send the target video frame number to the retrieval processing module.
[0043] In this embodiment, the retrieval vector generating unit generates a retrieval vector using a semantic vector model according to the information to be processed (it is necessary to ensure that the generation method of the retrieval vector is consistent with the text segmentation vector generation method of the index generating unit).
[0044] Furthermore, the knowledge search module includes: a second information receiving unit, a video decompression unit and a decoding unit, and the second information receiving unit, the video decompression unit and the decoding unit are connected in sequence.
[0045] Wherein, the second information receiving unit is used to receive the target video frame number from the retrieval processing module and send the target video frame number to the knowledge carrier module; The video decompression unit is used to receive the target video data from the knowledge carrier module, decompress the target video data to obtain the two-dimensional code image data, and send the two-dimensional code image data to the decoding unit; The decoding unit is used to receive the two-dimensional code image data, decode the two-dimensional code image data according to QR code rules, obtain target text information, and send the target text information to the knowledge retrieval interface module for output.
[0046] In this embodiment, the knowledge search module receives information from the retrieval processing module, locates and reads the knowledge carrier modules one by one according to the video frame number, decompresses the knowledge carrier modules according to their format, such as MP4, to obtain the QR code data. The QR code data is then decoded into text information according to QR code rules. Finally, all the text information is returned to the retrieval processing module as knowledge.
[0047] Furthermore, the index carrier module includes: an index storage unit and an index reading unit.
[0048] The index storage unit is configured to receive each of the index items and the video frame number corresponding to each of the index items from the index generation unit, and store each of the index items and the video frame number corresponding to each of the index items.
[0049] The index reading unit is used to receive the retrieval vector, perform a similarity search based on the retrieval vector according to a preset threshold, obtain a target index item whose similarity meets the preset threshold, obtain a corresponding target video frame number based on the target index item, and send the target video frame number to the index search module.
[0050] It can be understood that the index storage unit receives each of the index items and the video frame number corresponding to each of the index items, performs an index creation operation on each of the index items and the video frame number corresponding to each of the index items, and stores the indexes.
[0051] The index reading unit receives the search vector from the index search module and performs a similarity search on the index items based on the search vector, finding all indexes whose similarity meets a preset threshold, such as 0.8 (indicating 80% similarity). For each index, the corresponding video frame number stored in the index storage unit is found and read, and all video frame numbers are returned to the search processing module.
[0052] In particular, in order to improve the search speed, the contents of the index storage unit can be loaded into the computer's memory and arranged in ascending order by index item, and then the index can be searched for similarity and the video frame number can be read according to the retrieval vector, which will greatly improve the speed.
[0053] Furthermore, the knowledge carrier module includes: a knowledge storage unit and a knowledge reading unit.
[0054] The knowledge storage unit is configured to receive all video contents and the video frame number corresponding to each video content, and store all video contents and the video frame number corresponding to each video content in the form of an MP4 file.
[0055] The knowledge reading unit is used to receive the target video frame number from the knowledge search module, read out the corresponding text segment according to the target video frame number, use the corresponding text segment as target video data, and send the target video data to the knowledge search module.
[0056] It is understood that the knowledge storage unit is a computer resource that stores knowledge, and for example, can be an MP4 file. Similarly, to increase search speed, the contents of the knowledge storage unit can be loaded into the computer's memory, and then located and read according to the video frame number.
[0057] As can be seen, by encoding digital text data into video, the present invention not only achieves compressed storage of millions of text blocks, but also provides fast retrieval and sub-second retrieval speeds. Unlike traditional vector databases that consume large amounts of memory and storage, the present invention compresses the knowledge base into a compact video file while maintaining instant access to any information. It is simple to use, requires no professional skills to install and maintain, is low-cost, and gives individuals full control over their own knowledge.
[0058] Based on the storage and retrieval system based on the knowledge base described in the above embodiment, the present invention also provides a data processing method for the storage and retrieval system based on the knowledge base, specifically as follows Figure 3 As shown in , the data processing method of the storage and retrieval system based on the knowledge base includes the following steps: Step S10: The knowledge storage module receives input information from the knowledge storage interface module, processes the input information, and generates metadata.
[0059] In this embodiment, the storage processing unit of the knowledge storage module receives input information from the knowledge storage interface module and sends the input information to the knowledge generation unit.
[0060] The knowledge generation unit divides the input information according to preset standards to obtain multiple text segments, sets a video frame number for each text segment, and generates metadata based on the multiple text segments and the video frame number corresponding to each text segment. It can be understood that the metadata includes text segments and corresponding video frame numbers.
[0061] Step S20: the knowledge storage module creates an index for the metadata, sends the index and the metadata to the index carrier module for storage, obtains video data based on the metadata, and sends the video data to the knowledge carrier module for storage.
[0062] Specifically, the knowledge storage module converts all text segments in the metadata into vector representations through a semantic vector model, uses the vector representation of each text segment as an index item, and sends each index item and the video frame number corresponding to each index item to the index carrier module for storage.
[0063] Furthermore, the knowledge storage module generates QR code images corresponding to all text segments in the metadata through QR code rules, takes each QR code image as a frame of video content, compresses each frame of video content according to the format of the knowledge carrier module, and sends all compressed frames of video content and corresponding video frame numbers to the knowledge carrier module for storage.
[0064] Step S30: The retrieval processing module receives the information to be processed from the knowledge retrieval interface module, and sends the information to be processed to the index search module.
[0065] Specifically, the knowledge retrieval interface module receives various initial data to be processed from other systems, preprocesses the initial data to be processed, obtains data to be processed, and sends the data to be processed to the retrieval processing module, wherein the initial data to be processed includes a preprocessed data list, a text file, a PDF document, and original text characters.
[0066] Furthermore, the retrieval processing module receives the data to be processed from the knowledge retrieval interface module, sends the data to be processed to the index search module, and waits for the index search module to reply with a video frame number.
[0067] Step S40: The index search module generates a retrieval vector according to the information to be processed, obtains a target video frame number from the index carrier module according to the retrieval vector, and sends the target video frame number to the retrieval processing module.
[0068] Specifically, the first information receiving unit of the index search module receives the information to be processed sent by the retrieval processing module, and sends the information to be processed to the retrieval vector generating unit.
[0069] Furthermore, the retrieval vector generating unit of the index search module receives the information to be processed, generates a retrieval vector through a semantic vector model according to the information to be processed, and sends the retrieval vector to the index carrier module.
[0070] Furthermore, the retrieval vector generation unit of the index search module receives the target video frame number from the index carrier module, and sends the target video frame number to the retrieval processing module.
[0071] Step S50: The retrieval processing module sends the target video frame number to the knowledge search module. The knowledge search module reads the target video data corresponding to the target video frame number from the knowledge carrier module, decodes the target video data, obtains target text information, and sends the target text information to the knowledge retrieval interface module for output.
[0072] Specifically, the second information receiving unit of the knowledge search module receives the target video frame number from the retrieval processing module, and sends the target video frame number to the knowledge carrier module.
[0073] Furthermore, the video decompression unit of the knowledge search module receives the target video data from the knowledge carrier module, decompresses the target video data to obtain two-dimensional code image data, and sends the two-dimensional code image data to the decoding unit.
[0074] Furthermore, the decoding unit of the knowledge search module receives the two-dimensional code image data, decodes the two-dimensional code image data according to QR code rules to obtain target text information, and sends the target text information to the knowledge retrieval interface module for output.
[0075] In an exemplary embodiment, the operation process of the knowledge base-based storage and retrieval system is as follows: Storage process: A user stores a WORD file containing computer knowledge. The knowledge storage interface module receives the WORD file, preprocesses it, extracts the plain text content, and sends it to the knowledge storage module's storage processing unit. The storage processing unit then sends the plain text content to the knowledge storage module's knowledge generation unit. The knowledge generation unit receives the plain text content, segments it into multiple text segments, and generates a unique text segment ID and a video frame number for each text segment. These data are then combined with the text segment to form metadata. The metadata is then encoded to generate a QR code. All QR codes are then added to the corresponding locations of an MP4 video file according to the video frame number. The metadata is then sent to the storage processing unit. The storage processing unit then sends all metadata to the index generation unit of the knowledge storage module. The index generation unit generates a vector for each metadata text segment and adds the vector as an index and metadata to the end of the index carrier unit. Finally, the operation results are returned to the storage processing unit, which then sends the results to the knowledge storage interface module for feedback to the user.
[0076] Retrieval process: The knowledge retrieval interface unit receives user input text content, titled "Computer Knowledge Explanation," and sends it to the retrieval processing module. The retrieval processing module then sends it to the index search module. The index search module receives the text content, segments it into blocks, generates retrieval vectors, and loads the contents of the index carrier module into computer memory. The module then sorts the content by index term and uses a binary sorting algorithm to quickly compare the retrieval vectors with the index terms. The module then finds the target video frame numbers corresponding to all index terms that meet a preset similarity threshold (80%) and sends these target video frame numbers to the retrieval processing module. The retrieval processing module then sends all target video frame numbers to the knowledge search module. The knowledge search module then loads the contents of the knowledge carrier module into computer memory, retrieves the corresponding frame content based on the video frame number, decompresses the frame content, obtains a QR code, and decodes the QR code to obtain the corresponding target text information. The knowledge search module then processes all the target text information into knowledge and sends it to the retrieval processing module. The retrieval processing module then returns the knowledge information as the operation result to the knowledge retrieval interface module, which then returns the operation result to the user who sent the "Computer Knowledge Explanation" request.
[0077] The beneficial effects of the present invention are as follows: (1) Storing millions of text blocks in a single MP4 file is a simple and mature technology that can be widely used.
[0078] (2) Sub-second semantic search of massive data meets the efficiency needs of non-professional users and enables instant retrieval.
[0079] (3) Video compression significantly reduces storage and memory usage, reduces usage costs, and facilitates large-scale promotion.
[0080] (4) No database server or GPU is required, only files that can be copied anywhere are needed, and no new IT investment is required.
[0081] (5) After the video is generated, it works completely offline and can be used at any time on mobile phones, computers and self-service devices.
[0082] (6) A single video file contains the entire knowledge base, making it easy to manage. Videos can be streamed from cloud storage, making them easy to circulate.
[0083] In summary, the present invention provides a storage and retrieval system and data processing method based on a knowledge base, wherein the knowledge storage module receives input information from the knowledge storage interface module, processes the input information, and generates metadata; the knowledge storage module creates an index for the metadata, sends the index and metadata to the index carrier module for storage, obtains video data based on the metadata, and sends the video data to the knowledge carrier module for storage; the retrieval processing module receives information to be processed from the knowledge retrieval interface module, and sends the information to be processed to the index search module; the index search module generates a retrieval vector based on the information to be processed, and obtains the target video from the index carrier module based on the retrieval vector. Frame number, and send the target video frame number to the retrieval processing module; the retrieval processing module sends the target video frame number to the knowledge search module, the knowledge search module reads the target video data corresponding to the target video frame number from the knowledge carrier module, decodes the target video data, obtains the target text information, and sends the target text information to the knowledge retrieval interface module for output. The present invention compresses the knowledge base into a compact video file while maintaining instant access to any information. It is simple to use, does not require professional skills for installation and maintenance, and is low in cost. Individuals can have full control over their own knowledge, realizes compressed storage of millions of text blocks, and can also provide fast retrieval and sub-second retrieval speed.
[0084] It should be noted that, in this document, the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or terminal comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or terminal. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or terminal comprising the element.
[0085] Of course, those skilled in the art will appreciate that all or part of the processes in the above-described method embodiments can be implemented by instructing related hardware (such as a processor, controller, etc.) through a computer program. The program can be stored in a computer-readable storage medium that can be read by a computer. When executed, the program can include the processes in the above-described method embodiments. The computer-readable storage medium can be a memory, a magnetic disk, an optical disk, etc.
[0086] It should be understood that the application of the present invention is not limited to the above examples. For those skilled in the art, improvements or changes can be made based on the above description. All these improvements and changes should fall within the scope of protection of the claims attached to the present invention.
Claims
1. A storage and retrieval system based on a knowledge base, characterized in that: The storage and retrieval system based on the knowledge base includes: a knowledge storage module, a knowledge carrier module, an index carrier module, a retrieval processing module, an index search module and a knowledge search module, wherein the knowledge storage module is connected to the knowledge carrier module and the index carrier module respectively, the retrieval processing module is connected to the index search module and the knowledge search module respectively, and the index search module and the knowledge search module are connected to the index carrier module and the knowledge carrier module respectively; The knowledge storage module is used to receive input information from the knowledge storage interface module, process the input information, generate metadata, create an index for the metadata, send the index and the metadata to the index carrier module for storage, and obtain video data based on the metadata, and send the video data to the knowledge carrier module for storage; The retrieval processing module is used to receive the information to be processed from the knowledge retrieval interface module and send the information to be processed to the index search module; The index search module is used to generate a search vector according to the information to be processed, obtain a target video frame number from the index carrier module according to the search vector, and send the target video frame number to the search processing module; The retrieval processing module is further configured to send the target video frame number to the knowledge search module; The knowledge search module is used to read the target video data corresponding to the target video frame number from the knowledge carrier module, decode the target video data to obtain target text information, and send the target text information to the knowledge retrieval interface module for output.
2. The knowledge base based storage and retrieval system according to claim 1, characterized in that: The knowledge storage interface module is used to receive initial input data from other systems and pre-process the initial input data to obtain input information; Wherein, the initial input data includes a data list, a text file, a PDF document and an original text string; The knowledge storage interface module is further configured to receive operation results from the knowledge storage module and output the operation results.
3. The knowledge base based storage and retrieval system according to claim 1, characterized in that: The knowledge storage module includes: a storage processing unit, a knowledge generation unit and an index generation unit, wherein the storage processing unit is connected to the knowledge generation unit and the index generation unit respectively; The storage processing unit is used to receive input information from the knowledge storage interface module and send the input information to the knowledge generation unit; The knowledge generation unit is configured to segment the input information according to a preset standard to obtain a plurality of text segments, set a video frame number for each of the text segments, generate metadata based on the plurality of text segments and the video frame number corresponding to each of the text segments, and send the metadata to the storage processing unit; The knowledge generation unit is further configured to generate a QR code image corresponding to each of the text segments in the metadata using QR code rules, treat each QR code image as a frame of video content, compress each frame of video content according to the format of the knowledge carrier module, and send all compressed frames of video content and corresponding video frame numbers to the knowledge carrier module for storage; The storage processing unit is further configured to send the metadata to the index generating unit; The index generation unit is used to convert all the text segments in the metadata into vector representations through a semantic vector model, use the vector representation of each text segment as an index item, and send each index item and the video frame number corresponding to each index item to the index carrier module for storage.
4. The knowledge base based storage and retrieval system according to claim 1, characterized in that: The knowledge retrieval interface module is used to receive initial information to be processed from other systems and pre-process the initial information to be processed to obtain information to be processed; The knowledge retrieval interface module is further configured to receive the target text information from the retrieval processing module, and format the target text information based on the format of the information to be processed, thereby obtaining and outputting the final retrieval result.
5. The knowledge base based storage and retrieval system according to claim 1, characterized in that: The index search module includes: a first information receiving unit and a search vector generating unit; The first information receiving unit is configured to receive the information to be processed sent by the search processing module, and send the information to be processed to the search vector generating unit; The retrieval vector generating unit is configured to receive the information to be processed, generate a retrieval vector based on the information to be processed using a semantic vector model, and send the retrieval vector to the index carrier module; The retrieval vector generation unit is further configured to receive the target video frame number from the index carrier module and send the target video frame number to the retrieval processing module.
6. The knowledge base based storage and retrieval system according to claim 1, characterized in that: The knowledge search module includes: a second information receiving unit, a video decompression unit and a decoding unit, wherein the second information receiving unit, the video decompression unit and the decoding unit are connected in sequence; The second information receiving unit is used to receive the target video frame number from the retrieval processing module and send the target video frame number to the knowledge carrier module; The video decompression unit is used to receive the target video data from the knowledge carrier module, decompress the target video data to obtain the two-dimensional code image data, and send the two-dimensional code image data to the decoding unit; The decoding unit is used to receive the two-dimensional code image data, decode the two-dimensional code image data according to QR code rules, obtain target text information, and send the target text information to the knowledge retrieval interface module for output.
7. The knowledge base based storage and retrieval system according to claim 3, characterized in that: The index carrier module includes: an index storage unit and an index reading unit; The index storage unit is used to receive each of the index items and the video frame number corresponding to each of the index items from the index generation unit, and store each of the index items and the video frame number corresponding to each of the index items; The index reading unit is used to receive the retrieval vector, perform a similarity search based on the retrieval vector according to a preset threshold, obtain a target index item whose similarity meets the preset threshold, obtain a corresponding target video frame number based on the target index item, and send the target video frame number to the index search module.
8. The knowledge base based storage and retrieval system according to claim 1, characterized in that: The knowledge carrier module includes: a knowledge storage unit and a knowledge reading unit; The knowledge storage unit is used to receive all video content and the video frame number corresponding to each video content, and store all video content and the video frame number corresponding to each video content in the form of an MP4 file; The knowledge reading unit is used to receive the target video frame number from the knowledge search module, read out the corresponding text segment according to the target video frame number, use the corresponding text segment as target video data, and send the target video data to the knowledge search module.
9. A data processing method based on the knowledge base storage and retrieval system according to any one of claims 1 to 8, characterized in that: The data processing method includes: The knowledge storage module receives input information from the knowledge storage interface module, processes the input information, and generates metadata; The knowledge storage module creates an index for the metadata, sends the index and the metadata to the index carrier module for storage, obtains video data based on the metadata, and sends the video data to the knowledge carrier module for storage; The retrieval processing module receives the information to be processed from the knowledge retrieval interface module and sends the information to be processed to the index search module; The index search module generates a search vector according to the information to be processed, obtains a target video frame number from the index carrier module according to the search vector, and sends the target video frame number to the search processing module; The retrieval processing module sends the target video frame number to the knowledge search module, and the knowledge search module reads the target video data corresponding to the target video frame number from the knowledge carrier module, decodes the target video data, obtains target text information, and sends the target text information to the knowledge retrieval interface module for output.
10. The data processing method of the knowledge base based storage and retrieval system according to claim 9, characterized in that: The knowledge storage module creates an index for the metadata, sends the index and the metadata to the index carrier module for storage, obtains video data based on the metadata, and sends the video data to the knowledge carrier module for storage, specifically including: The knowledge storage module converts all text segments in the metadata into vector representations through a semantic vector model, uses the vector representation of each text segment as an index item, and sends each index item and the video frame number corresponding to each index item to the index carrier module for storage; The knowledge storage module generates QR code images corresponding to all text segments in the metadata through QR code rules, takes each QR code image as a frame of video content, compresses each frame of video content according to the format of the knowledge carrier module, and sends all compressed frames of video content and corresponding video frame numbers to the knowledge carrier module for storage.