Data processing and storage method and apparatus based on graph database and vector database

Abstract

Disclosed are a data processing and storage method and apparatus based on a graph database and a vector database. On the basis of a graph database and a vector database in combination with a LayoutLMv3 model, a Transformer model, and OCR technology, the invention aims to efficiently parse, store, and retrieve unstructured documents. In the present invention, first, a document is converted into an image, and a layout analysis model, namely the LayoutLMv3 model, is used to identify several types of regions in the image, such as text, images, and tables. Then, three types of parsers are used to analyze the regions containing data. In particular, due to the complexity of table data structures, a table analysis model is used to convert tables into textual representations. Finally, all obtained data is structurally partitioned and respectively stored in a graph database and a vector database to achieve high accuracy and high efficiency in data retrieval, thereby providing strong support for big data analysis and large language model applications.

Description

Data processing and storage methods and apparatus based on graph databases and vector databases Technical Field

[0001] This invention relates to the field of big data processing technology, and in particular to a data processing and storage method and apparatus based on graph databases and vector databases. Background Technology

[0002] In recent years, with the rapid development of LLM (Large Language Model), numerous applications based on LLM have emerged. However, LLM suffers from problems such as "nonsense" and "illusions." Therefore, researchers have proposed a language model that combines retrieval and generation, namely the RAG model. The RAG model first constructs a knowledge base, then retrieves relevant information from the knowledge base based on the user's question, and combines this with the generation function of LLM to output the answer, thereby improving the accuracy of LLM question answering. However, the effectiveness of RAG technology largely depends on the quality of the structured data in the knowledge base. If the data is poorly structured, the retrieved information may be inaccurate or contain noise, leading the generation module to generate results based on unreliable context, thus affecting the final output quality.

[0003] To address this issue, this invention proposes a data processing and storage method combining the graph database Neo4j and the vector database Elasticsearch, aiming to achieve layout analysis and data transformation of document images. In this method, Neo4j excels at storing and processing data containing complex relationships, intuitively representing data and their interrelationships using nodes and edges. Elasticsearch, as a vector database, provides an efficient way to store and retrieve vector-based feature data for handling unstructured data search tasks, such as text and image similarity searches.

[0004] To better analyze document layout, this invention employs the LayoutLMv3 document layout analysis model, which can identify and analyze different elements in a document, such as text, tables, and image regions, making document processing more intelligent and efficient. Combined with a Transformer-based table transformation model and OCR technology, these tools can automatically convert unstructured data (such as images, tables, and text) in a document into structured text data, thereby greatly improving the accessibility and usability of document information.

[0005] In summary, this invention proposes a data processing and storage method based on the graph database Neo4j and the vector database Elasticsearch. It combines LayoutLMv3, a Transformer-based table transformation model, and OCR technology to achieve layout analysis and data transformation of document images. In this method, the graph database, with its flexible data model and powerful semantic expression capabilities, can effectively store and query complex relationship networks; the vector database, through vectorization technology, converts text, images, and other data into numerical vectors to support efficient similarity search and data clustering, thereby ensuring the high-quality data input required for RAG technology. Summary of the Invention

[0006] This invention addresses the shortcomings of existing technologies, namely the limitations in effectively storing unstructured and semi-structured data (such as documents, images, tables, etc.), by proposing a data processing and storage method and apparatus based on graph databases and vector databases, providing users with a more efficient, accurate, and reliable data preprocessing method.

[0007] The method proposed in this invention is achieved through the following technical solution: a data processing and storage method based on graph databases and vector databases, the method comprising the following steps:

[0008] Step 1: Identify the document layout and convert the table format, then convert the identified content into Markdown format and save it;

[0009] Step 2: Extract key information about the document's theme, references, and appendices from the Markdown file based on a large language model;

[0010] Step 3: Convert the Markdown file into unstructured data, divide the data into blocks and store them in a vector database. Based on the key information of the documents, construct a set of relationship types between documents and generate a visual knowledge graph.

[0011] Furthermore, in step 1, a layout detection model is constructed to analyze the document layout. The construction steps are as follows:

[0012] 1) Obtain the dataset for model training, including the Publaynet dataset and a self-built dataset with manually collected annotations, and divide it into training set and test set;

[0013] 2) Design a layout detection model architecture. Based on the LayoutLMv3 architecture, use document images to identify the document structure and realize the function of dividing the document image into three types of regions: text, image, and table.

[0014] 3) Train the layout detection model, set the learning rate, use the cross-entropy loss function to measure the difference between the model prediction and the actual label, and update the model parameters using the Adam optimizer.

[0015] Furthermore, in step 1, the table is converted into JSON format by building a table conversion model. The construction steps are as follows:

[0016] 1) Obtain the dataset for model training, including the FeTaQA, TAT-QA datasets and a self-built Chinese table dataset that was manually collected and labeled, and divide it into training and test sets;

[0017] 2) Design the architecture of the table detection model, and add a trainable prompt learning module based on the Transformer architecture to assist the model in the generation task;

[0018] 3) Train the table transformation model, set the learning rate, use the cross-entropy loss function to measure the difference between the model prediction and the actual label, and use the AdamW optimizer to improve training stability.

[0019] Furthermore, after converting the document into a document image, a layout detection model is used to analyze the content layout of the document image, dividing the document image into three categories: text, image, and table regions, and representing them with different colors. The layout detection model outputs four boundary values ​​of all region boxes, indicating the position of the region in the image, and each detected region is assigned a category label. The content of the text, image, and table regions is extracted, and the extracted data is saved in Markdown format.

[0020] Furthermore, in step 3, a data storage module is constructed to structurally divide the converted Markdown documents and store them in the graph database Neo4j and the vector database Elasticsearch. The construction steps are as follows:

[0021] 1) Structure the Markdown document; for text data, use headings as the basis for paragraph division, and use sliding window technology to divide the text within paragraphs into data blocks; for table and image data, treat each table or image as an independent data block, with each entity as the unit.

[0022] 2) Define an entity type set and a relation type set, where the entity type set contains document names and document types; the relation type set generates relationships between documents based on key information such as the document's topic, references, and appendices; import the resulting entity-relationship tuples into the neo4j database to form a visual document relationship knowledge graph;

[0023] 3) Create a relational index in Elasticsearch that stores the document's data. The index structure consists of attribute information, including the index's unique ID, the document's unique file_id, and the document's theme, content, embedding, type, and level.

[0024] 4) Based on the divided data blocks, use the BGE-embedding vectorization model to expand all data blocks and store them in the Elasticsearch vector database.

[0025] Furthermore, the layout detection model uses a text-image multimodal transformer to learn cross-modal features. It captures textual and image information through three modules: Masked Language Modeling (MLM), Masked Image Modeling (MIM), and Word Patch Alignment (WPA). Masked Language Modeling randomly masks a portion of the text word vectors while retaining the corresponding two-dimensional positional information. The task objective is to reconstruct the masked words in the text based on the unmasked text and layout information. Masked Image Modeling randomly masks a portion of the image blocks. The task objective is to reconstruct the discretized ID of the masked image blocks based on the information from the unmasked text and images. Word Patch Alignment learns fine-grained alignment relationships between language and visual modalities by explicitly predicting whether the corresponding image block of a text word is masked.

[0026] Furthermore, the table transformation model employs a 12-layer Transformer. Specifically, a special symbol "instruction:" is added before the input table content to prompt text generation, and trainable vectors are added to the multi-head attention module of each layer to pre-train task prompts. During training, the table is first flattened into a sequence so that it can be directly input into the model. Special markers are inserted to represent the table boundaries.

[0027] Furthermore, a graph database centered on "topics" is constructed to form an interconnected information network. In this network, "topic" nodes occupy a central position and are connected to multiple "file" nodes. Each "file" node is a rich set of information, containing "reference files", "ES indexes", and "attachments" child nodes. All nodes are connected to other nodes through the relationship types of reference files and appendices, forming a complex semantic network.

[0028] Furthermore, the data blocks are represented in a vectorized manner. For image data, after being converted to a vector representation, it is stored separately in Elasticsearch. Specifically, the BGE-embedding vectorization model is used. The vectorized data blocks are indexed by files in the Elasticsearch database. Each file includes multiple data blocks, and each data block has its own data type. In particular, the original version of the file data blocks, i.e., the unvectorized version, is also stored in the Elasticsearch database to achieve hybrid retrieval.

[0029] On the other hand, the present invention also provides a data processing and storage device based on graph databases and vector databases, including a memory and one or more processors, wherein the memory stores executable code, and when the processor executes the executable code, it implements the aforementioned data processing and storage method based on graph databases and vector databases.

[0030] The beneficial effects of this invention are as follows:

[0031] 1. By using a variety of professional parsers to parse different data types in documents, this invention can comprehensively identify and accurately extract multimodal data such as text, tables, and images.

[0032] 2. Construct a multimodal knowledge graph to effectively convert unstructured data into a structured form and store it in a graph database, while preserving the hierarchical structure and semantic information of the data.

[0033] 3. Elasticsearch stores both raw and vectorized data simultaneously, supporting mixed retrieval of raw and vectorized data, enhancing retrieval consistency and flexibility.

[0034] 4. Combining the advantages of graph databases and vector databases, this invention provides an efficient information retrieval mechanism that enables fast and accurate semantic search. Attached Figure Description

[0035] Figure 1 is a schematic flowchart of a data processing and storage method based on graph database and vector database provided by an embodiment of the present invention.

[0036] Figure 2 is an example diagram of custom data and labels for layout detection provided in an embodiment of the present invention.

[0037] Figure 3 is an example diagram of a knowledge graph provided in an embodiment of the present invention.

[0038] Figure 4 is a text segmentation structure diagram provided in an embodiment of the present invention.

[0039] Figure 5 is a schematic diagram of the table conversion model provided in an embodiment of the present invention.

[0040] Figure 6 is a structural diagram of a data processing and storage device based on a graph database and a vector database provided in an embodiment of the present invention. Detailed Implementation

[0041] To more clearly illustrate the purpose, technical solutions, and advantages of this specification, we will describe them in detail through specific embodiments and related drawings. It should be noted that the described embodiments represent only some examples and do not cover all possible implementations. Other embodiments that can be deduced by those skilled in the art based on these embodiments without creative work are all within the protection scope of this specification.

[0042] In summary, this specification aims to fully demonstrate the details of the technical solution through specific embodiments and accompanying drawings, while reserving protection for other possible embodiments. All operations are within the scope permitted by law and respect the rights of data owners.

[0043] This invention addresses the data processing and storage problems in existing technologies by constructing a unified multimodal graph database and vector database. On one hand, by building a knowledge graph, it stores the structured information within files, reflecting the correlations between files and providing rich semantic information. On the other hand, by constructing a vector database, it achieves efficient retrieval of file information, converting text, images, and other data into numerical vectors, greatly improving the speed and accuracy of retrieval. Combining these two database technologies, this invention comprehensively considers other files related to the query file when retrieving knowledge, thus expanding the scope and enhancing the depth of the retrieval.

[0044] As shown in Figure 1, this invention is a data processing and storage method based on graph databases and vector databases. The method includes the following steps:

[0045] (1) Train a document layout detection model and analyze the document layout based on the layout detection model LayoutLMv3.

[0046] First, the training dataset for the model was obtained, based on the Publaynet dataset and a manually collected and labeled self-built dataset, which was then divided into training and testing sets. Publaynet contains a large number of PDF documents and document page images, but lacks other document types such as Word and HTML, failing to address the issue of missing content during document conversion. Furthermore, the lack of Chinese data resulted in a low representation of Chinese characters in this dataset. Therefore, other types of Chinese documents were collected, converted into images, and then combined with the Publaynet dataset as the training dataset. As shown in Figure 2, after collecting the Chinese samples, labels were added. The first background color represents text, including document titles, abstracts, and keywords; the second background color represents images, each with an image bounding box; and the third background color represents each table.

[0047] Next, a layout detection model architecture was designed. Based on the LayoutLMv3 architecture, the model utilizes document images to identify document structure, enabling the segmentation of document images into three categories: text, images, and tables. This model, based on the LayoutLMv3 model, uses a text-image multimodal transformer to learn cross-modal features. It captures textual and image information through three modules: Masked Language Modeling (MLM), Masked Image Modeling (MIM), and Word-Patch Alignment (WPA).

[0048] Specifically, masked language modeling embeds text, including word embedding and positional embedding (layout). It uses readily available OCR toolkits to preprocess document images to obtain text content and corresponding 2D positional information. Word embedding is initialized using an embedding model. Positional embedding includes 1D positional embedding and 2D layout positional embedding. 1D positional embedding refers to the index of the marker in the text sequence, and 2D layout positional embedding refers to the bounding box coordinates of the text sequence. Since adjacent words in text usually express similar semantics, a piece of text shares a two-dimensional positional vector.

[0049] Masked image modeling uses image embedding to segment all document images into a series of uniform P×P size patches. Image feature sequences are obtained through linear mapping, and then a learnable one-dimensional position vector is added to obtain the image vector. Further, image embedding resizes the document images to H×W and uses I∈R... C×H×WLet C represent the image, where C, H, and W are the image's channel size, width, and height, respectively. Then, the image is divided into a series of uniform P×P blocks. These blocks are linearly projected onto a D-dimensional plane and flattened into a vector sequence of length M = HW / P. 2 .

[0050] In the masked language modeling, 30% of the text word vectors are randomly masked, but the corresponding two-dimensional position (layout) information is retained. The task is to restore the masked words in the text based on the unmasked text and layout information. In the masked image modeling, about 40% of the image patches are randomly masked. The task is to restore the discretized ID of the masked image patches based on the information of the unmasked text and images. In the word block alignment, fine-grained alignment relationships between language and visual modalities are learned by explicitly predicting whether the corresponding image patch of a text word is masked.

[0051] Given an input document image and its corresponding text and 2D positional information (layout) obtained using OCR, the model takes the linear projection of patches and word tokens as input and encodes them into contextualized vector representations. The model is trained using discrete token reconstruction objectives from Masked Language Model (MLM) and Masked Image Model (MIM). Furthermore, LayoutLMv3 is pre-trained with a Word-Patch Alignment (WPA) objective, learning cross-modal alignment by predicting whether an image patch corresponding to a text word is masked.

[0052] During the training of the layout detection model, the learning rate was set to 5e-5. At the same time, the cross-entropy loss function was used to measure the difference between the model prediction and the actual label, and the model parameters were updated using the Adam optimizer.

[0053] (2) Train a table conversion model. In order to cope with the large amount of table data and the complex structure, the table data in the document needs to be specially processed. The table conversion model is used to convert the table data in the document into a more readable JSON format.

[0054] First, obtain the dataset for model training. Based on datasets such as FeTaQA and TAT-QA, as well as a self-built Chinese table dataset that was manually collected and labeled, divide it into training and test sets.

[0055] The architecture of the table detection model is shown in Figure 5. The model is based on the standard Transformer architecture, with the addition of a trainable cue learning module to assist in the model's generation task. Specifically, a 12-layer Transformer is used. Notably, a special symbol "instruction:" is added before the input table content for cue text generation. Furthermore, trainable vectors are added to the multi-head attention module of each layer to pre-train the cue for the task, and the cue length is set to 100. During training, the table is first flattened into a sequence so that it can be directly input into the model. By inserting several special markers to represent the table boundaries, a flattened table can be represented as follows:

[0056] TABLE=[HEAD],col1,col2,..coln,[ROW],1,row1,row2,..rowm,[ROW],..

[0057] Here, [HEAD] and [ROW] are special markers, representing the header and row areas respectively. The number after [ROW] indicates the row index. The special symbol "instruction:" is added before inputting the table data into the model to assist in text generation. Table 1, as described below, will be converted to:

[0058] {[HEAD],"Serial Number","Time Period","Content",[ROW],"1","2014.1-2014.3","Preliminary Research and Technical Preparation Phase: We will conduct in-depth analysis of the characteristics of point cloud data and investigate existing convolutional neural network technologies to establish the theoretical foundation and technical roadmap of the project.",[ROW],"2","2014.10-2014.12","Technology Optimization and Application Promotion Phase: In the final stage, we will further optimize the algorithm and solve the problems found in the experiment."}

[0059] Table 1

[0060] The data, presented in a self-built Chinese table format, is as follows:

[0061] {

[0062] "instruction": "Please read the table in the following input and convert it into JSON representation based on its content. The table uses [HEAD] and [ROW] as special markers to represent the header and row areas respectively, and the number after [ROW] indicates the row index."

[0063] "input_serialize":"["[HEAD]","Serial Number","Time Period","Content","[ROW]","1","2014.1-2014.3","Preliminary Research and Technical Preparation Phase: We will conduct in-depth analysis of the characteristics of point cloud data and investigate existing convolutional neural network technologies to establish the theoretical foundation and technical roadmap of the project. ","[ROW]","2",2014.10-2014.12","Technology Optimization and Application Promotion Phase: In the final stage, we will further optimize the algorithm and solve the problems found in the experiment."]",

[0064] "input":"{"research_phases":[{"index":1,"time_period":"2014.1-2014.3","content":"Preliminary Research and Technical Preparation Phase: We will conduct in-depth analysis of the characteristics of point cloud data and investigate existing convolutional neural network technologies to establish the theoretical foundation and technical roadmap of the project. "},{"index":2,"time_period":"2014.10-2014.12","content":"Technology Optimization and Application Promotion Phase: In the final stage, we will further optimize the algorithm and solve the problems found in the experiments. "}]}",

[0065] "output": "From January to March 2014, we will conduct an in-depth analysis of the characteristics of point cloud data and investigate existing convolutional neural network technologies to establish the theoretical foundation and technical roadmap for the project. From October to December 2014, this is the final stage of the project, where we will further optimize the algorithm and address the problems discovered during the experiments."

[0066] }

[0067] When training the table transformation model, the learning rate was set to 3×5e-5 to ensure that the model could learn sufficiently and generalize to unseen data during training. The cross-entropy loss function was used to measure the difference between the model's predictions and the actual labels, with β1 = 0.9, β2 = 0.98, and λ = 1×10⁻⁵. -6 The AdamW optimizer is used to improve training stability.

[0068] (3) Based on the layout detection model and table conversion model obtained in steps (1) and (2), construct a layout detection module and a content analysis module to perform layout detection, OCR text recognition, and table conversion on the source document. Convert the recognized content into Markdown format and store it. The specific implementation process is as follows:

[0069] (3.1) Identify the document type and use the appropriate tools, such as pdfkit, python-docx, pptx, etc., to convert the document into a document image.

[0070] (3.2) The layout detection module analyzes the content layout of the document images, dividing them into three categories: text, images, and tables. Specifically, a first background color is used to represent text lines, including the document's title, abstract, and keywords; a second background color is used to represent images, each with an image bounding box; and a third background color is used to represent each table. Finally, the layout detection module outputs four boundary values ​​for all region bounding boxes, indicating the region's position in the image. Each detected region is assigned a category label, such as "TEXT," "IMAGE," or "TABLE."

[0071] (3.3) Use the content analysis module to extract the content of three types of areas: text, images, and tables, and save the extracted data in Markdown format.

[0072] For text extraction, OCR (Optical Character Recognition) tools are used to identify and extract text regions from the document. OCR technology analyzes characters in an image and converts them into an editable text format. During this process, OCR accurately identifies the document's paragraph structure, and the extracted text is saved according to the original document's paragraphs, ensuring the integrity of the document structure and the coherence of the content. For formulas in the document, they are saved in LaTeX format. LaTeX is a standard format for typesetting complex mathematical formulas and can accurately represent the structure and symbols of formulas.

[0073] For table areas, the identified tables are flattened and input into the table conversion model. The model can parse the table structure and convert it into a JSON format representation. JSON is a lightweight data-interchange format that is easy for machines to process and store, and it can preserve the row and column structure and data content of the table. The extracted table will be embedded in a Markdown file for subsequent data storage.

[0074] For image regions identified in the document, this invention chooses to directly save copies of the images locally. The Markdown file stores the relative path of the image locally, preserving the original image information and providing directly usable material for subsequent tasks. Specifically, it extracts important data from the document, such as filenames, document topics, references, and attachments.

[0075] (4) Based on the Markdown format file obtained in step (3), input the content into the Qwen72B large language model to extract key information such as the document's topic, references, and appendices; deploy the LLM locally and design the prompt as follows:

[0076] prompt="Please extract the following key information from the document below: "

[0077] 1. Document title

[0078] 2. List of references or bibliography

[0079] 3. Contents or list in the appendix section

[0080] The document content is as follows: [Insert document content]

[0081] Please output in the following format: Title: [Extracted Title] References: [Extracted List of References] Appendix: [Extracted Appendix Contents]

[0082] The key information extracted is recorded and stored at the beginning of the Markdown file in (4) for the purpose of constructing a knowledge graph.

[0083] (5) Construct a data storage module. Based on the Markdown format file obtained in step (3), convert the unstructured data in the file, divide it into structured data chunks of different sizes, and store them in the vector database. Based on the key document information obtained in step (4), construct a set of relationship types between documents and generate a visual knowledge graph. The specific implementation process is as follows:

[0084] (5.1) Structured partitioning of Markdown documents. When segmenting the document, targeted processing strategies were adopted for different types of data. For text data, as shown in Figure 4, the headings were used as the basis for paragraph partitioning. All paragraphs in the entire document were abstracted into a heading text tree. For the text within a paragraph, a sliding window technique was used to divide the data into blocks. Assuming each data block contains 100 words, each slide might move 50 words, expanding by 50 words before and after, ensuring overlap between adjacent chunks and maintaining the coherence and integrity of the text information. For table and image data, each table or image was treated as an independent data block, with each entity as the unit.

[0085] (5.2) Based on the document data extracted in steps (3.1) and (4), define the entity type set and the relation type set, where the entity type set is the document name and document type; the relation type set is based on the key information such as document topic, reference documents, and appendices extracted from the large language model in step (4) to generate the relationships between documents. Import the obtained entity-relationship tuples into the neo4j database to form a visual document relationship knowledge graph.

[0086] A graph database centered on "topics" is constructed, forming a highly interconnected information network. In this network, "topic" nodes occupy a central position and are connected to multiple "file" nodes. Each "file" node is a rich collection of information, containing sub-nodes such as "reference files," "Elasticsearch index," and "attachments." Among them, "reference files" and "attachments" act as bridges between files, constructing a complex knowledge network, while "Elasticsearch index" stores index information about the file in Elasticsearch. All nodes are connected to other nodes through various relationship types such as reference files and appendices, forming a complex semantic network. Furthermore, key information such as reference files and attachments of the documents obtained in step (4) is stored to construct a connection network with other related documents. As shown in Figure 3, the network contains multiple topics, each of which contains a plurality of files. These files may contain multiple attributes. They are independent of each other but are linked to other files through reference files and appendices. In Figure 3, arrows pointing from topics to files and from files to attributes indicate inclusion relationships, while arrows pointing from attributes to files indicate association relationships. In particular, the network extends to the Elasticsearch database through the Elasticsearch index, thus constructing a hybrid retrieval system that combines graph and vector databases.

[0087] (5.3) Create a relational index in Elasticsearch that stores the specific data of the documents. The structure of the index is mainly composed of attribute information, including the unique ID of the index, the unique file_id of the document, and the theme, content, embedding, type, and level of the document. When creating the index, set the number of shards to 3 and the number of replicas to 0.

[0088] The index is designed as follows:

[0089] (5.4) Based on the data blocks obtained in step (5.1), the BGE-embedding vectorization model is used to treat all data blocks as a token, expand it to 1024 dimensions, and then store it in the Elasticsearch vector database. In the Elasticsearch database, files are used as the index, and each file includes multiple data blocks, each with its own data type. Each data block contains: file_id (representing the file's unique ID), trunk_id (representing the data block's unique ID), file_theme (representing the file's theme), content (representing the initial description of the data block's content), embedding (representing the vectorized description of the data block's content), type (representing the initial category of the data block, such as table, image, text, etc.), and level (representing the level of the data block's heading). Specifically, the original version of the file data blocks, i.e., the unvectorized version, is also stored in the Elasticsearch database to enable mixed retrieval. The data blocks obtained in step (5.1) are traversed, and the attribute values ​​required for the Elasticsearch database index are extracted and written into the index sequentially.

[0090] Corresponding to the aforementioned embodiment of a data processing and storage method based on graph databases and vector databases, the present invention also provides an embodiment of a data processing and storage apparatus based on graph databases and vector databases.

[0091] Referring to Figure 6, an embodiment of the present invention provides a data processing and storage device based on a graph database and a vector database, including a memory and one or more processors. The memory stores executable code, and when the processor executes the executable code, it is used to implement a data processing and storage method based on a graph database and a vector database as described in the above embodiment.

[0092] The present invention provides an embodiment of a data processing and storage device based on graph and vector databases, which can be applied to any device with data processing capabilities, such as a computer. The device embodiment can be implemented in software, hardware, or a combination of both. Taking software implementation as an example, as a logical device, it is formed by the processor of the device with data processing capabilities reading the corresponding computer program instructions from non-volatile memory into memory and executing them. From a hardware perspective, as shown in Figure 6, which is a hardware structure diagram of any device with data processing capabilities where the data processing and storage device based on graph and vector databases provided by the present invention is located, in addition to the processor, memory, network interface, and non-volatile memory shown in Figure 6, the device with data processing capabilities in the embodiment may also include other hardware depending on the actual function of the device, which will not be elaborated further.

[0093] The specific implementation process of the functions and roles of each unit in the above device can be found in the implementation process of the corresponding steps in the above method, and will not be repeated here.

[0094] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of the present invention according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0095] This invention also provides a computer-readable storage medium storing a program thereon, which, when executed by a processor, implements a data processing and storage method based on a graph database and a vector database as described in the above embodiments.

[0096] The computer-readable storage medium can be an internal storage unit of any data processing device described in any of the foregoing embodiments, such as a hard disk or memory. The computer-readable storage medium can also be an external storage device of any data processing device, such as a plug-in hard disk, smart media card (SMC), SD card, flash card, etc., equipped on the device. Furthermore, the computer-readable storage medium can include both internal storage units and external storage devices of any data processing device. The computer-readable storage medium is used to store the computer program and other programs and data required by the data processing device, and can also be used to temporarily store data that has been output or will be output.

[0097] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the aforementioned data processing and storage method based on a graph database and a vector database.

[0098] The above embodiments are used to explain and illustrate the present invention, but not to limit the present invention. Any modifications and changes made to the present invention within the spirit and scope of the claims shall fall within the protection scope of the present invention.

Claims

1. A data processing and storage method based on graph databases and vector databases, characterized in that, The method includes the following steps: Step 1: Identify document layout based on the LayoutLMv3 model and perform table format conversion based on the Transformer model. Convert the identified content into Markdown format and store it. The document layout is analyzed by constructing a layout detection model. The construction steps are as follows: 1) Obtain the dataset for model training, including the Publaynet dataset and a self-built dataset with manually collected annotations, and divide it into training set and test set; 2) Design a layout detection model architecture. Based on the LayoutLMv3 architecture, it uses document images to identify document structure. After converting the document into a document image, the layout detection model analyzes the content layout of the document image, dividing the document image into three categories: text, image, and table regions, and representing them with different colors. The layout detection model outputs four boundary values ​​of all region boxes, indicating the position of the region in the image. Each detected region is assigned a category label. The content of the text, image, and table regions is extracted, and the extracted data is saved in Markdown format. The layout detection model uses a text-image multimodal transformer for learning. Cross-modal features are achieved by capturing textual and image information through three modules: Masked Language Modeling (MLM), Masked Image Modeling (MIM), and Word Patch Alignment (WPA). Masked Language Modeling randomly masks a portion of the text word vectors while retaining the corresponding two-dimensional positional information. The task is to reconstruct the masked words in the text based on the unmasked text and layout information. Masked Image Modeling randomly masks a portion of the image patches. The task is to reconstruct the discretized IDs of the masked image patches based on the information from the unmasked text and images. Word Patch Alignment learns fine-grained alignment relationships between language and visual modalities by explicitly predicting whether the corresponding image patch of a text word is masked. 3) Train the layout detection model, set the learning rate, use the cross-entropy loss function to measure the difference between the model prediction and the actual label, and update the model parameters using the Adam optimizer; Step 2: Extract key information about the document's theme, references, and appendices from the Markdown file based on a large language model; Step 3: Convert the Markdown files to structured data, divide the data into blocks and store them in a vector database. Based on the key information of the documents, construct a set of relationship types between documents and generate a visual knowledge graph. Specifically, the converted Markdown documents are structured and stored in the graph database Neo4j and the vector database Elasticsearch. The specific steps are as follows: 1) Structure the Markdown document; for text data, use headings as the basis for paragraph division, and use sliding window technology to divide the text within paragraphs into data blocks; for table and image data, treat each table or image as an independent data block, with each entity as the unit. 2) Define an entity type set and a relation type set. The entity type set includes document names and document types. The relation type set generates relationships between documents based on key information such as the document's topic, references, and appendices. Import the resulting entity-relationship tuples into the Neo4j database to construct a graph database centered on "topics," forming an interconnected information network. In this network, the "topic" node occupies a central position and is connected to multiple "file" nodes. Each "file" node is a rich set of information, containing "references," "ES index," and "attachments" child nodes. All nodes are connected to other nodes through the relation types of references and appendices, forming a complex semantic network and obtaining a visualized document relationship knowledge graph. 3) Create a relational index in Elasticsearch that stores the document's data. The index structure consists of attribute information, including the index's unique ID, the document's unique file_id, and the document's theme, content, embedding, type, and level. 4) Based on the divided data blocks, all data blocks are expanded using the BGE-embedding vectorization model and stored in the Elasticsearch vector database. Specifically, the data blocks are represented in vector form. For image data, after being converted to vector representation, it is stored separately in Elasticsearch. The BGE-embedding vectorization model is used to vectorize the divided data blocks. In the Elasticsearch database, files are used as indexes, and each file includes multiple data blocks, each with its own data type. At the same time, the original version of the file data blocks, i.e., the unvectorized version, is stored in the Elasticsearch database to achieve mixed retrieval.

2. The data processing and storage method based on graph database and vector database according to claim 1, characterized in that, In step 1, the table is converted into JSON format by building a table conversion model. The construction steps are as follows: 1) Obtain the dataset for model training, including the FeTaQA, TAT-QA datasets and a self-built Chinese table dataset that was manually collected and labeled, and divide it into training and test sets; 2) Design the architecture of the table detection model, and add a trainable prompt learning module based on the Transformer architecture to assist the model in the generation task; 3) Train the table transformation model, set the learning rate, use the cross-entropy loss function to measure the difference between the model prediction and the actual label, and use the AdamW optimizer to improve training stability.

3. The data processing and storage method based on graph database and vector database according to claim 2, characterized in that, The table transformation model uses a 12-layer Transformer, adding the special symbol "instruction:" before the input table content to prompt text generation, and adding trainable vectors to the multi-head attention module of each layer to pre-train task prompts. During training, the table is first flattened into a sequence so that it can be directly input into the model; special markers are inserted to indicate the boundaries of the table.

4. A data processing and storage device based on graph databases and vector databases, comprising a memory and one or more processors, wherein the memory stores executable code, characterized in that, When the processor executes the executable code, it implements a data processing and storage method based on a graph database and a vector database as described in any one of claims 1-3.

Images