Data processing and storage method and device based on graph database and vector database

By combining the graph database and vector database methods, processing and storing document pictures, and using LayoutLMv3 and Transformer models for layout analysis and table conversion, the problem of unstructured data storage accuracy is solved, efficient data retrieval and conversion is achieved, and improved Data utilization and retrieval consistency.

CN118964514AActive Publication Date: 2024-11-15ZHEJIANG GONGSHANG UNIVERSITY +1

Patent Information

Application Number
CN202411421476.2
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-10-12
Publication Date
2024-11-15
Estimated Expiration
2044-10-12

AI Technical Summary

Technical Problem

The prior art has problems of accuracy and reliability when processing and storing unstructured and semi-structured data (such as documents, pictures, tables, etc.), especially when the data is low in structure, resulting in insufficient information retrieved. Accurate or contain noise, affecting the final output quality.

Method used

Using the data processing and storage method based on the graph database Neo4j and the vector database Elasticsearch, combined with the LayoutLMv3 document layout analysis model, the table conversion model based on Transformer and OCR technology, it can identify and convert different elements in the document, build structureable data, and Efficient similarity search and data clustering are supported through BGE-embedding vectorization technology.

Benefits of technology

It realizes efficient layout analysis and data conversion processing of document pictures, provides more efficient, more accurate and more reliable data preprocessing methods, comprehensively identify multimodal data, build multimodal knowledge graphs, and supports fast and accurate semantics. search.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118964514A_ABST
    Figure CN118964514A_ABST
Patent Text Reader

Abstract

The invention discloses a data processing and storing method and device based on a graph database and a vector database. Based on a graph database and a vector database, in combination with a LayoutLMv3 model, a Transform model and an OCR (Optical Character Recognition) technology, the method aims to efficiently analyze, store and retrieve unstructured documents. According to the method, firstly, a document is converted into an image, a layout analysis model LayoutLMv3 model is used for recognizing a text region, an image region and a table region in the image, then three types of analyzers are used for analyzing a region containing data, particularly, due to the complexity of a table data structure, a table analysis model is used for converting a table into a text to be expressed, and finally, the table data structure is analyzed. All obtained data are subjected to structured segmentation and are respectively stored in a graph database and a vector database, so that high accuracy and high efficiency of data retrieval are realized, and powerful support is provided for big data analysis and big language model application.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of big data processing technology, and in particular to a data processing and storage method and device based on a graph database and a vector database. Background Art

[0002] In recent years, with the rapid development of LLM (Large Language Model), applications based on large language models have emerged in an endless stream. However, LLM has problems with "nonsense" and hallucinations, so researchers have proposed a language model that combines retrieval and generation, namely the RAG model. The RAG model first needs to build a knowledge base, then retrieve relevant information from the knowledge base based on the user's questions, and output answers in combination with the generation function of LLM, thereby improving the accuracy of LLM question and answer. However, the effectiveness of RAG technology depends to a large extent on the quality of structured data in the knowledge base. If the data is poorly structured, the retrieved information may not be accurate enough or contain noise, which in turn causes the generation module to generate results based on unreliable context, thus affecting the final output quality.

[0003] In order to solve this problem, the present invention proposes a data processing and storage method combining the graph database Neo4j and the vector database Elasticsearch, aiming to realize the layout analysis and data conversion processing of document images. In this method, Neo4j is good at storing and processing data containing complex relationships, and intuitively represents data and their relationships in the form of nodes and edges. As a vector database, Elasticsearch provides an efficient way to store and retrieve vector-based feature data for processing unstructured data search tasks, such as similarity search of text and images.

[0004] In order to better analyze the layout of the document, the present invention adopts the LayoutLMv3 document layout analysis model, which can identify and analyze different elements in the document, such as text, tables, image areas, etc., making document processing more intelligent and efficient. Combined with the table conversion model based on the Transformer model and OCR technology, these tools can automatically convert unstructured data in the document (such as images, tables, text, etc.) into text data that can be structured and processed, thereby greatly improving the accessibility and utilization of document information.

[0005] In summary, the present invention proposes a data processing and storage method based on the graph database Neo4j and the vector database Elasticsearch, combining LayoutLMv3, the Transformer-based table conversion model and OCR technology to realize the layout analysis and data conversion processing of document images. In this method, the graph database can effectively store and query complex relationship networks with its flexible data model and powerful semantic expression ability; the vector database converts text, image and other data into numerical vectors through vectorization technology to support efficient similarity search and data clustering, thereby ensuring the high-quality data input required by RAG technology. Summary of the invention

[0006] In view of the shortcomings of the prior art, namely the limitations of effective storage of unstructured data and semi-structured data (such as documents, pictures, tables, etc.), the present invention proposes a data processing and storage method and device based on a graph database and a vector database, providing users with a more efficient, accurate and reliable data preprocessing method.

[0007] The method proposed in the present invention is implemented by the following technical solution: a data processing and storage method based on a graph database and a vector database, the method comprising the following steps:

[0008] Step 1: Identify the document layout and convert the table format, convert the identified content into Markdown format and store it;

[0009] Step 2: Extract the subject, reference files, and appendix key information of the document in the Markdown format file based on the large language model;

[0010] Step 3: Convert the Markdown format file into non-structured data, divide the data blocks and store them in the vector database. Based on the key information of the document, build a set of relationship types between documents and generate a visual knowledge graph.

[0011] Furthermore, in step 1, the document layout is analyzed by constructing a layout detection model. The construction steps are as follows:

[0012] 1) Obtain the dataset for model training, including the Publaynet dataset and the self-built dataset with manual collection and annotation, and divide it into training set and test set;

[0013] 2) Design a layout detection model architecture, use document images to identify the structure of documents based on the LayoutLMv3 architecture, and implement the function of dividing document images into three types of areas: text, image, and table;

[0014] 3) Train the layout detection model, set the learning rate, measure the difference between the model prediction and the actual label through the cross entropy loss function, and update the model parameters through the Adam optimizer.

[0015] Furthermore, in step 1, the table is converted into JSON format by constructing a table conversion model. The construction steps are as follows:

[0016] 1) Obtain data sets for model training, including FeTaQA, TAT-QA data sets, and manually collected and annotated self-built Chinese table data sets, and divide them into training sets and test sets;

[0017] 2) Design the architecture of the table detection model and add a trainable prompt learning module based on the Transformer architecture to assist the model in generating tasks;

[0018] 3) Train the table conversion model, set the learning rate, use the cross-entropy loss function to measure the difference between the model prediction and the actual label, and use the AdamW optimizer to improve training stability.

[0019] Furthermore, after converting the document into a document image, the layout detection model is used to analyze the content layout of the document image, and the document image is divided into three types of areas: text, image, and table, which are represented by different colors; the layout detection model outputs the four boundary values ​​of all area boxes, indicating the position of the area in the image, and each detected area is assigned a category label; the content of the text, image, and table areas is extracted, and the extracted data is saved in Markdown format.

[0020] Furthermore, in step 3, a data storage module is constructed to structure the converted Markdown format documents and save them in the graph database neo4j and the vector database elasticsearch. The construction steps are as follows:

[0021] 1) Structural division of Markdown documents; for text data, the title is used as the basis for paragraph division, and the text within the paragraph is divided into data blocks using the sliding window technology; for table and image data, each entity is taken as a unit, and each table or image is regarded as an independent data block;

[0022] 2) Define entity type sets and relationship type sets, where the entity type set includes document name and document type; the relationship type set generates connections between documents based on the subject, reference files, and appendix key information of the document; import the obtained entity relationship tuples into the neo4j database to form a visual document relationship knowledge graph;

[0023] 3) Create an associated index in elasticsearch, which stores the specific data of the document. The index structure consists of attribute information, including the unique id of the index, the unique file_id of the document, the theme, content, embedding, type and level of the document;

[0024] 4) Based on the divided data blocks, use the BGE-embedding vectorization model to expand all data blocks and save them in the elasticsearch vector database.

[0025] Furthermore, the layout detection model uses a text-image multimodal transformer to learn cross-modal features, and captures text information and image information by using three modules: masked language modeling MLM, masked image modeling MIM, and word block alignment WPA; wherein masked language modeling randomly masks a part of the text word vector, but retains the corresponding two-dimensional position information, and the task goal is to restore the masked words in the text based on the unmasked image and layout information; masked image modeling randomly masks a part of the image block, and the task goal is to restore the discretized ID of the masked image block based on the unmasked text and image information; word block alignment learns the fine-grained alignment relationship between language and visual modalities by explicitly predicting whether the corresponding image block of a text word is masked.

[0026] Furthermore, the table conversion model superimposes a 12-layer Transformer. In particular, a special symbol "instruction:" is added before the input table content to prompt text generation, and a trainable vector is added in the multi-head attention module of each layer to pre-train task prompts; during the training process, the table is first flattened into a sequence so that it can be directly input into the model; and the boundaries of the table are indicated by inserting special markers.

[0027] Furthermore, a graph database with "topic" as the core is constructed to form an interconnected information network. In this network, the "topic" node occupies a central position and is connected to multiple "file" nodes. Each "file" node is a rich information collection, including "reference file", "es index", and "attachment" sub-nodes; all nodes are connected to other nodes through the relationship types of reference file and appendix, forming a complex semantic network.

[0028] Furthermore, the data blocks are represented by vectorization. For image data, after being converted into vector representation, they are stored separately in elasticsearch. Specifically, the BGE-embedding vectorization model is used to vectorize the divided data blocks. Files are used as indexes in the elasticsearch database. Each file includes multiple data blocks, and each data block has its own data type. In particular, the original version of the file data block, that is, the version that has not been vectorized, is saved in the elasticsearch database at the same time to achieve hybrid retrieval.

[0029] On the other hand, the present invention also provides a data processing and storage device based on a graph database and a vector database, comprising a memory and one or more processors, wherein the memory stores executable code, and when the processor executes the executable code, it implements the data processing and storage method based on a graph database and a vector database.

[0030] The beneficial effects of the present invention are as follows:

[0031] 1. By using a variety of professional parsers to parse different data types in documents, the invention can fully identify and accurately extract multimodal data such as text, tables, images, etc.

[0032] 2. Build a multimodal knowledge graph to effectively convert unstructured data into a structured form and store it in a graph database while retaining the hierarchical structure and semantic information of the data.

[0033] 3. Save the original data and vectorized data simultaneously in Elasticsearch, support mixed retrieval of original data and vectorized data, and enhance the consistency and flexibility of retrieval.

[0034] 4. Combining the advantages of graph database and vector database, this invention provides an efficient information retrieval mechanism that can achieve fast and accurate semantic search. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 A schematic flow chart of a data processing and storage method based on a graph database and a vector database provided in an embodiment of the present invention.

[0036] Figure 2 This is an example diagram of custom data and labels for layout detection provided by an embodiment of the present invention.

[0037] Figure 3 An example diagram of a knowledge graph provided for an embodiment of the present invention.

[0038] Figure 4 A text division structure diagram provided for an embodiment of the present invention.

[0039] Figure 5 A schematic diagram of a table conversion model provided in an embodiment of the present invention.

[0040] Figure 6 A structural diagram of a data processing and storage device based on a graph database and a vector database provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0041] In order to more clearly explain the purpose, technical solutions and advantages of this specification, we will describe it in detail through specific embodiments and related drawings. It should be clear that the described embodiments only represent some cases and do not cover all possible implementation methods. Based on these embodiments, other embodiments that can be derived by technicians in this field without performing creative work are all within the scope of protection of this specification.

[0042] In summary, this specification aims to fully demonstrate the details of the technical solution through specific embodiments and drawings, while retaining the protection of other possible embodiments. All operations are within the scope permitted by law and respect the rights and interests of data owners.

[0043] The present invention solves the data processing and storage problems in the prior art by constructing a multimodal unified graph database and vector database. On the one hand, by constructing a knowledge graph, the structured information inside the file is stored, the correlation between files is reflected, and rich semantic information can be provided; on the other hand, by constructing a vector database, efficient retrieval of file information is achieved, and data such as text and images are converted into numerical vectors, which greatly improves the speed and accuracy of retrieval. Combining these two database technologies, the present invention can fully consider other files related to the query file when retrieving knowledge, thereby expanding the retrieval scope and enhancing the retrieval depth.

[0044] like Figure 1 As shown, the present invention is a data processing and storage method based on a graph database and a vector database, the method comprising the following steps:

[0045] (1) Train the document layout detection model and analyze the document layout based on the layout detection model LayoutLMv3.

[0046] First, we obtain the dataset for model training, based on the Publaynet dataset and the self-built dataset that is manually collected and annotated, and divide it into a training set and a test set. Publaynet contains a large number of PDF documents and document page images, but lacks other types of documents, such as word, html, etc., and cannot cope with the problem of missing content in document conversion. In addition, the lack of Chinese data makes the dataset's expression of Chinese low. Therefore, we collected other types of Chinese documents, converted them into images, and combined them with the Publaynet dataset as a training dataset. Figure 2 As shown, after collecting Chinese samples, a label with a blue background is added to represent text, including the title, abstract, and keywords of the document. A label with a purple background is added to represent images, each image has an image box. A label with a yellow background is added to represent each table.

[0047] Then, the layout detection model architecture is designed. The model is based on the LayoutLMv3 architecture, uses document images to identify the structure of documents, and realizes the function of dividing document images into three types of areas: text, image, and table. This model is based on the LayoutLMv3 model, uses the text-image multimodal transformer to learn cross-modal features, and captures text and image information by using three modules: Masked Language Modeling (MLM), Masked Image Modeling (MIM), and Word-Patch Alignment (WPA).

[0048] Specifically, masked language modeling embeds text, including word embedding and position embedding (layout). The document image is preprocessed using an off-the-shelf OCR toolkit to obtain text content and corresponding 2D position information. The word embedding is initialized using the embedding model. The position embedding includes 1D position and 2D layout position embedding, where the 1D position refers to the index of the tag in the text sequence, and the 2D layout position refers to the bounding box coordinates of the text sequence. Since adjacent words in a text usually express similar semantics, a piece of text shares a two-dimensional position vector.

[0049] Mask image modeling uses image embedding to split all document images into a series of uniform P×P patches, obtains image feature sequences through linear mapping, and then adds a learnable one-dimensional position vector to obtain an image vector. Furthermore, image embedding resizes the document image to H×W and uses I∈R C×H×WRepresents an image, where C, H, and W are the channel size, width, and height of the image, respectively. Then, the image is split into a series of uniform P×P blocks, the image blocks are linearly projected into D dimensions, and they are flattened into a vector sequence with length M=HW / P 2 .

[0050] Among them, masked language modeling randomly masks 30% of the text word vectors, but retains the corresponding two-dimensional position (layout) information. The task goal is to restore the masked words in the text based on the unmasked images and layout information; masked image modeling randomly masks about 40% of the image blocks. The task goal is to restore the discretized ID of the masked image blocks based on the unmasked text and image information; word block alignment learns the fine-grained alignment relationship between language and visual modalities by explicitly predicting whether the corresponding image block of a text word is masked.

[0051] Given an input document image and its corresponding text and 2D position information (layout) information obtained using OCR, the model takes as input the linear projection of patches and word tokens and encodes them into contextualized vector representations. The model is trained with the discrete token reconstruction objectives of the Masked Language Model (MLM) and Masked Image Model (MIM). In addition, LayoutLMv3 is pre-trained with a Word-Patch Alignment (WPA) objective to learn cross-modal alignment by predicting whether the image patch corresponding to a text word is masked.

[0052] In the process of training the layout detection model, the learning rate is set to 5e-5. At the same time, the cross entropy loss function is selected to measure the difference between the model prediction and the actual label, and the model parameters are updated through the Adam optimizer.

[0053] (2) Training a table conversion model. To cope with the large amount of table data and complex structure, the table data in the document needs to be specially processed. The table conversion model is used to convert the table data in the document into a more readable JSON format.

[0054] First, obtain the data set for model training, and divide it into training set and test set based on data sets such as FeTaQA, TAT-QA, and self-built Chinese table data sets that are manually collected and annotated.

[0055] Design the architecture of the table detection model, such as Figure 5As shown in the figure, the model is based on the standard Transformer architecture and adds a trainable prompt learning module to assist the model's generation task. Specifically, a 12-layer Transformer is used in the stacking. In particular, a special symbol "instruction:" is added before the input table content to prompt text generation, and a trainable vector is added to the multi-head attention module of each layer to pre-train the prompt of the task, and the prompt length is set to 100. During the training process, the table is first flattened into a sequence so that it can be directly input into the model. By inserting several special markers to indicate the boundaries of the table, a flattened table can be represented as:

[0056] TABLE = [HEAD], col1, col2,..coln, [ROW], 1, row1,row2,..rowm,[ROW],..

[0057] Here [HEAD] and [ROW] are special tags, indicating the table header and row area respectively. The number after [ROW] is used to indicate the row index. Before entering the table data into the model, add the special symbol "instruction:" to assist in text generation. Table 1 described below will be converted to:

[0058] {[HEAD],"Serial Number","Time Period","Content",[ROW],"1","2014.1-2014.3","Preliminary Research and Technical Preparation Phase: We will conduct an in-depth analysis of the characteristics of point cloud data and investigate existing convolutional neural network technology to establish the theoretical basis and technical roadmap of the project.",[ROW],"2","2014.10-2014.12","Technical Optimization and Application Promotion Phase: In the final stage, we will further optimize the algorithm and solve the problems found in the experiment."}

[0059] Table 1

[0060] The self-built Chinese table data is as follows:

[0061] {

[0062] "instruction":"Please read the table in the following input and convert it into JSON representation according to the table content. [HEAD] and [ROW] are special marks in the table, indicating the header and row areas respectively. The number after [ROW] is used to indicate the row index.",

[0063] "input_serialize":"["[HEAD]","Serial number","Time period","Content","[ROW]","1","2014.1-2014.3","Preliminary research and technical preparation stage: We will conduct in-depth analysis of the characteristics of point cloud data and investigate the existing convolutional neural network technology to establish the theoretical basis and technical roadmap of the project. ","[ROW]","2",2014.10-2014.12","Technical optimization and application promotion stage: In the final stage, we will further optimize the algorithm and solve the problems found in the experiment. "]",

[0064] "input":"{"research_phases": [{"index": 1,"time_period": "2014.1-2014.3","content": "Preliminary research and technical preparation phase: We will conduct in-depth analysis of the characteristics of point cloud data and investigate existing convolutional neural network technology to establish the theoretical basis and technical roadmap of the project. "},{"index": 2,"time_period": "2014.10-2014.12","content": "Technical optimization and application promotion phase: In the final stage, we will further optimize the algorithm and solve the problems found in the experiment. "}]}",

[0065] "output":"From January to March 2014, we will conduct an in-depth analysis of the characteristics of point cloud data and investigate existing convolutional neural network technology to establish the theoretical basis and technical roadmap of the project. From October to December 2014, this is the final stage of the project, we will further optimize the algorithm and solve the problems found during the experiment."

[0066] }

[0067] When training the table conversion model, the learning rate is set to 3×5e-5 to ensure that the model can fully learn and generalize to unseen data during training. The cross entropy loss function is selected to measure the difference between the model prediction and the actual label, and the , , The AdamW optimizer is used to improve training stability.

[0068] (3) Based on the layout detection model and table conversion model obtained in step (1) and step (2), a layout detection module and a content analysis module are constructed to perform layout detection, OCR text recognition, and table conversion on the source document, and the recognized content is converted into Markdown format and stored. The specific implementation process is as follows:

[0069] (3.1) Identify the type of document and use corresponding tools, such as pdfkit, python-docx, pptx, etc., to convert the document into a document image.

[0070] (3.2) Use the layout detection module to analyze the content layout of the document image and divide the document image into three types of areas: text, image, and table. Specifically, a blue background is used to represent text lines, including the title, abstract, and keywords of the document, a purple background is used to represent images, each image has an image box, and a yellow background is used to represent each table. Finally, the layout detection module outputs the four boundary values ​​of all area boxes, indicating the location of the area in the image. Each detected area will be assigned a category label, such as "TEXT", "IMAGE", "TABLE", etc.

[0071] (3.3) Use the content analysis module to extract the content of three types of areas: text, images, and tables, and save the extracted data in Markdown format.

[0072] For the extraction of text areas, use the OCR (Optical Character Recognition) tool to identify and extract text areas in the document. OCR technology analyzes the characters in the image and converts them into editable text format. In this process, OCR can accurately identify the paragraph structure of the document. The extracted text will be saved according to the paragraphs of the original document to ensure the integrity of the document structure and the coherence of the content. For the formulas in the document, choose to save them in latex format. Latex is a standard format for typesetting complex mathematical formulas, which can accurately represent the structure and symbols of the formulas.

[0073] For the table area, the identified table is flattened and input into the table conversion model. The model can parse the structure of the table and convert it into a JSON format. JSON format is a lightweight data exchange format that is easy for machine processing and storage, and can retain the row and column structure and data content of the table. The extracted table will be embedded in the Markdown file for subsequent data storage.

[0074] For the image area identified in the document, the present invention chooses to directly save a copy of the image and store it locally. The Markdown file saves the relative path of the image locally, retains the original information of the image, and also provides direct call materials for subsequent tasks. In particular, important data in the document is extracted, such as file name, file subject, reference file, attachment, etc.

[0075] (4) Based on the Markdown format file obtained in step (3), input the content into the Qwen72B large language model to extract key information such as the subject, reference files, and appendix of the document; deploy the LLM locally and design the prompt as follows:

[0076] prompt = "Please extract the following key information from the document below:

[0077] 1. Title of the document

[0078] 2. References or reference list

[0079] 3. Contents or lists of the appendix

[0080] The content of the document is as follows: [Insert document content]

[0081] Please output in the following format: Title: [extracted title] References: [extracted reference list] Appendix: [extracted appendix content]"

[0082] Record the extracted key information and save it at the beginning of the Markdown file in (4) for building the knowledge graph.

[0083] (5) Construct a data storage module. According to the Markdown format file obtained in step (3), convert the unstructured data in the file, divide it into structured data chunks of different sizes, and store them in the vector database; according to the key information of the document obtained in step (4), construct a set of relationship types between documents and generate a visual knowledge graph. The specific implementation process is as follows:

[0084] (5.1) Structural division of Markdown documents. When segmenting documents, targeted processing strategies are adopted for different types of data. For text data, Figure 4 As shown in the figure, the title is used as the basis for dividing paragraphs, and all paragraphs of the entire document are abstracted into a title text tree. For the text in the paragraph, the sliding window technology is used to divide the data blocks. Assuming that each data block contains 100 words, each slide may slide 50 words, that is, 50 words are extended before and after, so that there are overlapping parts between adjacent chunks, ensuring the coherence and integrity of the text information. For table and image data, each entity is taken as a unit, and each table or image is regarded as an independent data block.

[0085] (5.2) Define entity type set and relationship type set based on the document data extracted in step (3.1) and step (4), where the entity type set is the document name and document type; the relationship type set is based on the key information such as document subject, reference file, appendix, etc. extracted by the large language model in step (4) to generate the connection between documents. Import the obtained entity relationship tuples into the neo4j database to form a visual document relationship knowledge graph.

[0086] Build a graph database with "topic" as the core to form a highly interconnected information network. In this network, the "topic" node occupies a central position and is connected to multiple "file" nodes. Each "file" node is a rich information collection, including sub-nodes such as "reference file", "es index", and "attachment". Among them, "reference file" and "appendix" act as bridges between files to build a complex knowledge network, and the "es index" stores the index information about the file in elasticsearch. All nodes are connected to other nodes through various relationship types such as reference files and appendices, forming a complex semantic network. Furthermore, store the key information such as reference files and attachments of the document obtained according to step (4) to build a connection network with other related documents. Figure 3 The network shown in the figure has multiple topics. Each topic contains multiple files. The files may contain multiple attributes. These files are independent of each other, but are connected to other files through reference files and appendices. Figure 3 The arrows from the subject to the file and from the file to the attribute indicate the inclusion relationship, and the arrow from the attribute to the file indicates the association relationship. In particular, the es index is extended to the elasticsearch database to build a hybrid retrieval of the graph database and the vector database.

[0087] (5.3) Create an associative index in elasticsearch. This index stores the specific data of the document. The structure of the index is mainly composed of attribute information. The attributes include the unique id of the index, the unique file_id of the document, and the theme, content, embedding, type, and level of the document. When creating the index, set the number of shards to 3 and the number of replicas to 0.

[0088] The index design is as follows:

[0089] {

[0090] 'file_id': {

[0091] 'type': 'keyword'

[0092] },

[0093] 'trunk_id': {

[0094] 'type': 'keyword'

[0095] },

[0096] 'file_theme':{

[0097] 'type':'text',

[0098] 'fields':{

[0099] 'keyword': {

[0100] 'type': 'keyword'

[0101] }

[0102] },

[0103] },

[0104] 'content': {

[0105] 'type': 'text',

[0106] 'similarity': 'BM25'

[0107] },

[0108] 'embedding': {

[0109] 'type': 'dense_vector',

[0110] 'dims': EMBEDDING_DIMS,

[0111] 'index': True,

[0112] 'similarity': 'cosine'

[0113] },

[0114] 'type': {

[0115] 'type': 'text'

[0116] },

[0117] 'level': {

[0118] 'type': 'text'

[0119] }

[0120] (5.4) Based on the divided data blocks obtained in step (5.1), all data blocks are treated as a token using the BGE-embedding vectorization model and expanded to 1024 dimensions and then saved in the elasticsearch vector database. In the elasticsearch database, files are used as indexes. Each file includes multiple data blocks, and each data block has its own data type. Each data block contains file_id, which represents the unique id of the file, trunk_id, which represents the unique id of the data block, file_theme, which represents the theme of the file, content, which represents the initial expression of the content of the data block, embedding, which represents the expression of the content of the data block after vectorization, type, which represents the initial category of the data block, such as table, image, text, etc., and level, which represents the level of title of the data block. In particular, the original version of the file data block, that is, the unvectorized version, is saved in the elasticsearch database at the same time to achieve hybrid retrieval. Traverse the divided data blocks obtained in step (5.1), extract the attribute values ​​required for the elasticsearch database index, and write them into the index in sequence.

[0121] Corresponding to the aforementioned embodiment of a data processing and storage method based on a graph database and a vector database, the present invention also provides an embodiment of a data processing and storage device based on a graph database and a vector database.

[0122] See also Figure 6 A data processing and storage device based on a graph database and a vector database provided in an embodiment of the present invention includes a memory and one or more processors, wherein the memory stores executable code, and when the processor executes the executable code, it is used to implement a data processing and storage method based on a graph database and a vector database in the above embodiment.

[0123] The embodiment of a data processing and storage device based on a graph database and a vector database provided by the present invention can be applied to any device with data processing capabilities, and the device with data processing capabilities can be a device or apparatus such as a computer. The device embodiment can be implemented through software, or through hardware or a combination of software and hardware. Taking software implementation as an example, as a device in a logical sense, it is formed by the processor of any device with data processing capabilities in which it is located reading the corresponding computer program instructions in the non-volatile memory into the internal memory for execution. From a hardware perspective, if Figure 6 As shown, it is a hardware structure diagram of a data processing and storage device based on a graph database and a vector database provided by the present invention, in which any device with data processing capability is located, except Figure 6In addition to the processor, memory, network interface, and non-volatile memory shown, any device with data processing capabilities in which the apparatus in the embodiments is located may also include other hardware, generally based on the actual functions of the device with data processing capabilities, which will not be described in detail.

[0124] The implementation process of the functions and effects of each unit in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned method, and will not be repeated here.

[0125] For the device embodiment, since it basically corresponds to the method embodiment, the relevant parts can refer to the partial description of the method embodiment. The device embodiment described above is only schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of the present invention. Ordinary technicians in this field can understand and implement it without paying creative work.

[0126] An embodiment of the present invention further provides a computer-readable storage medium on which a program is stored. When the program is executed by a processor, a data processing and storage method based on a graph database and a vector database in the above embodiment is implemented.

[0127] The computer-readable storage medium may be an internal storage unit of any device with data processing capability described in any of the aforementioned embodiments, such as a hard disk or a memory. The computer-readable storage medium may also be an external storage device of any device with data processing capability, such as a plug-in hard disk, a smart media card (SMC), an SD card, a flash card, etc. equipped on the device. Furthermore, the computer-readable storage medium may also include both an internal storage unit and an external storage device of any device with data processing capability. The computer-readable storage medium is used to store the computer program and other programs and data required by any device with data processing capability, and may also be used to temporarily store data that has been output or is to be output.

[0128] The present invention also provides a computer program product, including a computer program, which, when executed by a processor, implements the data processing and storage method based on a graph database and a vector database.

[0129] The above embodiments are used to illustrate the present invention rather than to limit the present invention. Any modification and change made to the present invention within the spirit of the present invention and the protection scope of the claims shall fall within the protection scope of the present invention.

Claims

1. A data processing and storage method based on a graph database and a vector database, characterized in that: The method comprises the following steps: Step 1: Identify the document layout and convert the table format, convert the identified content into Markdown format and store it; Step 2: Extract the subject, reference files, and appendix key information of the document in the Markdown format file based on the large language model; Step 3: Convert the Markdown format file into non-structured data, divide the data blocks and store them in the vector database. Based on the key information of the document, build a set of relationship types between documents and generate a visual knowledge graph.

2. A data processing and storage method based on a graph database and a vector database according to claim 1, characterized in that: In step 1, the document layout is analyzed by building a layout detection model. The construction steps are as follows: 1) Obtain the dataset for model training, including the Publaynet dataset and the self-built dataset with manual collection and annotation, and divide it into training set and test set; 2) Design a layout detection model architecture, use document images to identify the structure of documents based on the LayoutLMv3 architecture, and implement the function of dividing document images into three types of areas: text, image, and table; 3) Train the layout detection model, set the learning rate, measure the difference between the model prediction and the actual label through the cross entropy loss function, and update the model parameters through the Adam optimizer.

3. A data processing and storage method based on a graph database and a vector database according to claim 1, characterized in that: In step 1, the table is converted into JSON format by building a table conversion model. The construction steps are as follows: 1) Obtain data sets for model training, including FeTaQA, TAT-QA data sets, and manually collected and annotated self-built Chinese table data sets, and divide them into training sets and test sets; 2) Design the architecture of the table detection model and add a trainable prompt learning module based on the Transformer architecture to assist the model in generating tasks; 3) Train the table conversion model, set the learning rate, use the cross-entropy loss function to measure the difference between the model prediction and the actual label, and use the AdamW optimizer to improve training stability.

4. A data processing and storage method based on a graph database and a vector database according to claim 2, characterized in that: After converting the document into a document image, the layout detection model is used to analyze the content layout of the document image, and the document image is divided into three types of areas: text, image, and table, which are represented by different colors; the layout detection model outputs the four boundary values ​​of all area boxes, indicating the position of the area in the image, and each detected area is assigned a category label; the content of the text, image, and table areas is extracted, and the extracted data is saved in Markdown format.

5. The data processing and storage method based on a graph database and a vector database according to claim 1, characterized in that: In step 3, a data storage module is built to structure the converted Markdown documents and save them in the graph database neo4j and the vector database elasticsearch. The construction steps are as follows: 1) Structural division of Markdown documents; for text data, the title is used as the basis for paragraph division, and for the text in the paragraph, the sliding window technology is used to divide the data blocks; For table and image data, each entity is taken as a unit and each table or image is regarded as an independent data block; 2) Define entity type sets and relationship type sets, where the entity type set includes document name and document type; the relationship type set generates connections between documents based on the document's subject, reference files, and appendix key information; Import the obtained entity-relationship tuples into the neo4j database to form a visual document relationship knowledge graph; 3) Create an associated index in elasticsearch, which stores the specific data of the document. The index structure consists of attribute information, including the unique id of the index, the unique file_id of the document, the theme, content, embedding, type and level of the document; 4) Based on the divided data blocks, use the BGE-embedding vectorization model to expand all data blocks and save them in the elasticsearch vector database.

6. A data processing and storage method based on a graph database and a vector database according to claim 2, characterized in that: The layout detection model uses a text-image multimodal transformer to learn cross-modal features, and captures text and image information by using three modules: masked language modeling (MLM), masked image modeling (MIM), and word block alignment (WPA). Among them, masked language modeling randomly masks a part of the text word vector, but retains the corresponding two-dimensional position information. The task goal is to restore the masked words in the text based on the unmasked images and layout information; masked image modeling randomly masks a part of the image block. The task goal is to restore the discretized ID of the masked image block based on the unmasked text and image information; word block alignment learns the fine-grained alignment relationship between language and visual modalities by explicitly predicting whether the corresponding image block of a text word is masked.

7. A data processing and storage method based on a graph database and a vector database according to claim 3, characterized in that: The table conversion model uses a 12-layer Transformer stack. In particular, a special symbol "instruction:" is added before the input table content to prompt text generation, and a trainable vector is added to the multi-head attention module of each layer to pre-train the task prompt; During training, the table is first flattened into a sequence so that it can be directly input into the model; special markers are inserted to indicate the boundaries of the table.

8. The data processing and storage method based on a graph database and a vector database according to claim 5, characterized in that: Build a graph database with "topic" as the core to form an interconnected information network. In this network, the "topic" node occupies a central position and is connected to multiple "file" nodes. Each "file" node is a rich information collection, including "reference file", "es index", and "attachment" sub-nodes; all nodes are connected to other nodes through the relationship types of reference file and appendix, forming a complex semantic network.

9. The data processing and storage method based on a graph database and a vector database according to claim 5, characterized in that: The data blocks are represented by vectorization. For image data, after being converted into vector representation, they are stored separately in elasticsearch. Specifically, the BGE-embedding vectorization model is used to vectorize the divided data blocks. Files are used as indexes in the elasticsearch database. Each file includes multiple data blocks, and each data block has its own data type. In particular, the original version of the file data block, that is, the unvectorized version, is saved in the elasticsearch database at the same time to achieve hybrid retrieval.

10. A data processing and storage device based on a graph database and a vector database, comprising a memory and one or more processors, wherein the memory stores executable code, characterized in that: When the processor executes the executable code, it implements a data processing and storage method based on a graph database and a vector database as described in any one of claims 1 to 9.

Citation Information

Patent Citations

  • Search terminal control method and search terminal

    CN112966007A

  • Data relation identification method and data processing method for data relation identification

    CN116403038A

  • Question and answer method and system based on knowledge graph and large language model

    CN116775847A

  • Data management system for unifying multi-modal data structured processing based on GPT model

    CN117235276A

  • Document layout analysis method and system based on domain noise self-adaption

    CN118097693A

Cited By

  • Document analysis and index construction method

    CN119445603A

  • Method and device for automatically converting cross-format document into Markdown document and form enhancement method and system

    CN119598980A

  • RAG knowledge base large model enhanced dialogue method and system for financial bank

    CN119938823A

  • Water conservancy data quality inspection rule automatic generation method and system based on large language model

    CN120067090A

  • Online document processing method and system based on structured storage

    CN121009068A