Heterogeneous information-based large language model expansion method and system, and storage medium

Abstract

The application provides a large language model extension method and system based on heterogeneous information and a storage medium, the method comprising: obtaining a target document, and identifying document content in the target document by region to obtain a plurality of content nodes; analyzing the hierarchical relationship between different document contents, and constructing a tree topology structure for representing the hierarchical relationship by taking the content nodes as tree nodes, while analyzing the reference relationship of different document contents and assigning the reference relationship to the corresponding content nodes; storing the tree topology structure as extension data of a large language model in a database; wherein, according to the different types of document content, different storage structures are used to store the content nodes; and based on the large language model, the database is interacted to obtain an interaction result. The application is convenient for supporting dynamic granularity adaptation and multi-round feedback retrieval, and is conducive to improving the intelligent level and interaction ability of an AI system.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a method, system and storage medium for expanding a large language model based on heterogeneous information. Background Technology

[0002] With the development of deep learning technology, large language models based on Transformer have performed well in natural language processing tasks. However, when applying general large models to vertical fields, achieving low-cost and high-efficiency knowledge expansion has become a core technical challenge. The optimization and upgrading of related technical solutions has become an urgent need for industry development.

[0003] Existing large-scale model knowledge expansion technologies mainly include three implementation schemes: The first is full retraining and efficient parameter fine-tuning (LoRA), which retrains or fine-tunes the model by adding new corpora to achieve knowledge internalization; the second is model editing (Knowledge Editing) technology, which updates specific knowledge by directly modifying model parameters; the third is retrieval-augmented generation (RAG), which is the current mainstream solution. It usually adopts the core paradigm of PDF parsing, text slicing (Chunking), and vector retrieval, and supplements the model with relevant contextual information by adding external knowledge bases without retraining the model.

[0004] However, all kinds of knowledge expansion technologies have significant technical defects, and the technical shortcomings of mainstream search enhancement generation schemes are particularly prominent:

[0005] Full retraining and fine-tuning are computationally and time-intensive, and can easily lead to catastrophic forgetting or semantic drift problems, making it difficult to accurately correct learned erroneous knowledge.

[0006] Model editing is complex, making it difficult to guarantee the internal consistency of the model. Furthermore, the complexity of the operation can easily damage the model's original general reasoning ability.

[0007] The core problem with retrieval-enhanced generation schemes is the lack of a universal storage format that preserves the complete logical structure, layout space information, and multimodal associations of documents. Specifically, the parsing process loses document structural information and fragments multimodal information, the retrieval process is rigid in granularity and lacks dynamic interactive capabilities, and lossy data compression limits the visual understanding capabilities of large multimodal models, resulting in poor system scalability. Ultimately, this restricts the improvement of long document reasoning capabilities and the construction of advanced intelligent agents in retrieval-enhanced generation systems.

[0008] Therefore, it is necessary to propose a new knowledge extension scheme for large language models to solve the above problems. Summary of the Invention

[0009] This invention provides a method, system, and storage medium for expanding a large language model based on heterogeneous information, aiming to solve the technical problems of existing expansion methods being unable to fully utilize multimodal information and having poor retrieval and expansion capabilities.

[0010] To address the aforementioned technical problems, in a first aspect, the present invention provides a method for expanding a large language model based on heterogeneous information, comprising the following steps:

[0011] S101. Obtain the target document for knowledge expansion of the large language model, and identify the document content in the target document by region to obtain multiple content nodes;

[0012] S102. Analyze the hierarchical relationship between the document content corresponding to different content nodes, and construct a tree topology structure to represent the hierarchical relationship using the content nodes as tree nodes. Then, analyze the reference relationship between the document content corresponding to different content nodes, and assign the reference relationship to the corresponding content node that has formed the tree topology structure.

[0013] Subsequently, the tree-like topology constructed from the content nodes with the aforementioned referencing relationships is stored in the database as extended data of the large language model; wherein, depending on the type of the document content corresponding to the content node, different storage structures are used to store the content nodes in the database;

[0014] S103. Interact with the database based on the large language model to obtain the interaction result.

[0015] Furthermore, step S102 includes the following sub-steps:

[0016] The hierarchical relationship of inclusion or adjacency between the document content corresponding to different content nodes is obtained based on a preset hierarchical derivation method.

[0017] Based on the hierarchical relationship, the content node is used as a tree node to construct the tree topology, so that the hierarchical relationship becomes the structural edge of the tree topology, and a corresponding node ID is assigned to each content node in the tree topology.

[0018] The reference relationships between the document content corresponding to different content nodes are obtained based on a preset semantic analysis method, and the reference relationships are assigned to the corresponding content nodes that have formed the tree topology, so that the reference relationships become the semantic edges of the tree topology.

[0019] Different storage structures are used to define the content nodes according to the different types of document content corresponding to the content nodes;

[0020] The storage structure is used as extended data of the large language model in the database;

[0021] Based on the target document, the content nodes, the hierarchical relationships, the referencing relationships, and the tree topology, a related table is constructed for querying the content nodes in the database, and the related table is stored in the database.

[0022] Furthermore, the document content includes textual information and non-textual information, and step S101 further includes the following steps:

[0023] Extract the text contained in the text class information, and use the obtained text as the content node corresponding to the text class information;

[0024] Extract the original coordinate borders of the non-text information in the page of the target document, and capture the corresponding screenshots from the page according to the original coordinate borders. Use the obtained screenshots as the content nodes corresponding to the non-text information.

[0025] Furthermore, the storage structure includes at least one of the following items: node ID, type of document content, metadata of document content, hierarchical relationship, and reference relationship;

[0026] For the text-type information, the storage structure further includes at least one of the following items: text content, feature vector of text content, parent node ID of the corresponding content node, and sibling node ID of the corresponding content node;

[0027] For the non-text information, the storage structure further includes at least one of the following items: screenshot content, original coordinate border, caption text, text in the screenshot content, storage path of the screenshot content, feature vector of the screenshot content, annotation of the screenshot content, and parent node ID of the corresponding content node.

[0028] Furthermore, the database includes:

[0029] A physical sub-database is used to store the document content corresponding to each of the content nodes;

[0030] A logical sub-database is used to store the hierarchical relationship corresponding to each of the content nodes;

[0031] A semantic sub-database is used to store the reference relationships corresponding to each of the content nodes.

[0032] Furthermore, the association table includes at least one of the following:

[0033] A document metadata table is used to store the metadata of the target document;

[0034] The core structure table is used to store the indexes of the hierarchical relationships;

[0035] The content details table is used to store the metadata of the document content corresponding to the content node;

[0036] A semantic graph edge table is used to store indexes of the reference relationships;

[0037] A vector table is used to store the vector indexes of the document content corresponding to the content nodes.

[0038] Furthermore, step S103 includes the following sub-steps:

[0039] Obtain the query entry, perform a vector similarity-based retrieval in the database based on the query entry to obtain multiple content nodes, and use the obtained multiple content nodes as anchor nodes;

[0040] Based on the association table, the anchor node is searched in the database by upward expansion and / or adjacent expansion and / or cross-level expansion based on the hierarchical relationship and the reference relationship, to obtain multiple content nodes associated with the anchor node, and the multiple content nodes obtained are used as extension nodes;

[0041] The document content corresponding to each of the anchor nodes and the extended nodes is uniformly organized into interactive text for the output of the large language model, which serves as the interaction result.

[0042] Secondly, the present invention also provides a large language model extension system based on heterogeneous information, comprising:

[0043] The document parsing module is used to obtain target documents for knowledge expansion of large language models, and to identify the document content in the target documents by region to obtain multiple content nodes;

[0044] The tree structure module is used to analyze the hierarchical relationship between the document content corresponding to different content nodes, and to construct a tree topology structure to represent the hierarchical relationship using the content nodes as tree nodes. Then, it analyzes the reference relationship between the document content corresponding to different content nodes and assigns the reference relationship to the corresponding content node that has formed the tree topology structure.

[0045] Subsequently, the tree-like topology constructed from the content nodes with the aforementioned referencing relationships is stored in the database as extended data of the large language model; wherein, depending on the type of the document content corresponding to the content node, different storage structures are used to store the content nodes in the database;

[0046] The retrieval and reasoning module is used to interact with the database based on the large language model and obtain the interaction results.

[0047] Thirdly, the present invention also provides a computer device, comprising: a memory, a processor, and a large language model extension program based on heterogeneous information stored in the memory and executable on the processor, wherein when the processor executes the large language model extension program based on heterogeneous information, it implements the steps in the large language model extension method based on heterogeneous information as described in any of the above embodiments.

[0048] Fourthly, the present invention also provides a storage medium storing a large language model extension program based on heterogeneous information, wherein when the large language model extension program based on heterogeneous information is executed by a processor, the large language model extension program based on heterogeneous information implements the steps in the large language model extension method based on heterogeneous information as described in any of the above embodiments.

[0049] The beneficial effects achieved by this invention lie in proposing a method for expanding a large language model based on heterogeneous information. This method, by constructing a storage structure corresponding to different types of document content, enables the large language model to reduce the loss of document slice information during expansion and interaction, while preserving structural semantics, multimodal associations, and original physical information, thus exhibiting good scalability. At the same time, flexible interaction paradigms can be constructed based on this storage structure, which facilitates dynamic granularity adaptation and multi-turn feedback retrieval, thereby improving the intelligence level and interaction capabilities of AI systems. Attached Figure Description

[0050] The present invention will now be described in detail with reference to the accompanying drawings. The above and other aspects of the present invention will become clearer and more readily understood through the detailed description following the accompanying drawings. In the drawings:

[0051] Figure 1 This is a flowchart of the steps of the large language model expansion method based on heterogeneous information provided in the embodiments of the present invention;

[0052] Figure 2 This is a schematic diagram of the structure of the large language model extension system based on heterogeneous information provided in an embodiment of the present invention;

[0053] Figure 3 This is a schematic diagram of the structure of a computer device provided in an embodiment of the present invention. Detailed Implementation

[0054] The specific embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0055] The specific embodiments / examples described herein are specific implementations of the present invention, used to illustrate the concept of the invention, and are illustrative and exemplary, and should not be construed as limiting the implementation methods or scope of the present invention. In addition to the embodiments described herein, those skilled in the art can employ other obvious technical solutions based on the content disclosed in the claims and specification of this application. These technical solutions include those that make any obvious substitutions and modifications to the embodiments described herein, all of which are within the protection scope of the present invention.

[0056] Example 1

[0057] Please refer to Figure 1 , Figure 1 This is a flowchart illustrating the steps of a large language model expansion method based on heterogeneous information provided in this embodiment of the invention. The large language model expansion method based on heterogeneous information includes the following steps:

[0058] S101. Obtain the target document for knowledge expansion of the large language model, and identify the document content in the target document by region to obtain multiple content nodes.

[0059] The target document described in this embodiment of the invention is the content document that the large language model needs to learn. The type of the target document includes, but is not limited to, PDF, Word, PPT, etc. The characteristics of the target document are different from those of conventional linear documents. Its content is rich in pictures and texts, and the pictures and texts are interconnected.

[0060] This invention treats the text and image content in the target document as unified image modality data for recognition processing, identifying multiple content nodes from the document page by region. During implementation, the recognition can be performed using deep learning-based visual models (such as LayoutLM, YOLO, etc.) to detect objects on the document page, or by using document parsing tools to extract and structurally transform the content of the document page to obtain the content nodes required in step S101. The content identified based on regions in the target document page includes, but is not limited to, titles, text, images, tables, and captions, and can be selectively identified according to the characteristics of the target document.

[0061] Specifically, in this embodiment of the invention, the document content type includes textual information and non-textual information, and step S101 further includes the following steps:

[0062] Extract the text contained in the text class information, and use the obtained text as the content node corresponding to the text class information;

[0063] Extract the original coordinate borders of the non-text information in the page of the target document, and capture the corresponding screenshots from the page according to the original coordinate borders. Use the obtained screenshots as the content nodes corresponding to the non-text information.

[0064] Due to the different content, for large language models, textual information constitutes the main output content, while non-textual information is mostly used to supplement or illustrate the textual information. In step S101, in order to facilitate the construction of the database for knowledge expansion in subsequent steps, information of different modalities is distinguished based on the content identified in different regions of the target document. Among them, the image modal data divided by region from the target document is still retained as the original data.

[0065] S102. Analyze the hierarchical relationship between the document content corresponding to different content nodes, and construct a tree topology structure to represent the hierarchical relationship using the content nodes as tree nodes. Then, analyze the reference relationship between the document content corresponding to different content nodes, and assign the reference relationship to the corresponding content node that has formed the tree topology structure.

[0066] Subsequently, the tree-like topology constructed from the content nodes with the aforementioned referencing relationships is stored in the database as extended data of the large language model; wherein, depending on the type of the document content corresponding to the content node, different storage structures are used to store the content nodes in the database.

[0067] Step S102 includes the following sub-steps:

[0068] S1021. Obtain the hierarchical relationship of inclusion or adjacency between the document content corresponding to different content nodes based on a preset hierarchical derivation method;

[0069] S1022. Based on the hierarchical relationship, using the content nodes as tree nodes, construct the tree topology structure, making the hierarchical relationship the structural edge of the tree topology structure, and assigning a corresponding node ID to each content node in the tree topology structure.

[0070] S1023. Obtain the reference relationship between the document content corresponding to different content nodes based on a preset semantic analysis method, and assign the reference relationship to the corresponding content node that has formed the tree topology, so that the reference relationship becomes the semantic edge of the tree topology.

[0071] S1024. Define the content node using different storage structures according to the different types of document content corresponding to the content node;

[0072] S1025. Store the storage structure as extended data of the large language model in the database;

[0073] S1026. Based on the target document, the content node, the hierarchical relationship, the reference relationship, and the tree topology, construct an association table for querying the content node in the database, and store the association table in the database.

[0074] As defined in step S101, different content nodes are identified from the pages of the target document. Different content nodes have objective hierarchical relationships, such as the inclusion relationship between titles and body text, and the adjacency relationship between paragraphs. In this embodiment of the invention, a preset hierarchical derivation method is used to analyze these hierarchical relationships. These relationships can be annotated using their format during the content node identification process. For example, based on a specific document parsing tool, the content nodes identified from the target document are constructed into a sequential node array. At this time, the preset hierarchical derivation method can determine the inclusion relationship between text blocks based on font size, text box indentation, and position information; or it can use traditional or rule-based natural language processing algorithms as the preset hierarchical derivation method to discover the reference relationships between text paragraphs in different content nodes.

[0075] In particular, since text or images of the same paragraph may be scattered at the bottom of the previous page or the top of the next page in the target document, in step S1021, the content nodes can be merged based on the prior knowledge that image nodes and title / footnote nodes are basically adjacent, thereby optimizing the complexity of the tree topology.

[0076] Furthermore, in step S1022, the basic tree topology structure is constructed using content nodes as tree nodes and hierarchical relationships as structural edges. Each node in the tree topology structure has a corresponding node ID, which facilitates the rapid description of the relationships between nodes later through the node ID.

[0077] The preset semantic analysis method in step S1023 is used to analyze the reference relationships between text and images. For example, in the target document, the text content of Chapter 2, Section 1 mentions "as described in Chapter 1, Section 1," or the text content of Chapter 3 mentions "Fig. 5 in Chapter 2," etc. Such reference relationships cannot be represented by the hierarchical relationship as in step S1021. Therefore, in this step, the reference relationships between the document content corresponding to different content nodes are obtained, and the reference relationships are assigned to the corresponding content nodes that constitute the tree topology, making the reference relationships semantic edges of the tree topology. At this time, for the tree topology, the structural edges formed by the hierarchical relationship determine its tree structure, while the semantic edges formed by the reference relationships make it a cyclic graph. This design enables the embodiments of the present invention to implement rich retrieval functions based on the tree topology in subsequent processes.

[0078] Specifically, step S1024 uses a specific data structure to define the document content in the content node.

[0079] The storage structure includes at least one of the following items: node ID, type of document content, metadata of document content, hierarchical relationship, and reference relationship;

[0080] For the text-type information, the storage structure further includes at least one of the following items: text content, feature vector of text content, parent node ID of the corresponding content node, and sibling node ID of the corresponding content node;

[0081] For the non-text information, the storage structure further includes at least one of the following items: screenshot content, original coordinate border, caption text, text in the screenshot content, storage path of the screenshot content, feature vector of the screenshot content, annotation of the screenshot content, and parent node ID of the corresponding content node.

[0082] Unlike existing extended knowledge storage designs, this invention defines textual and non-textual information separately based on their data modalities. During implementation, as in step S101 of the above embodiments, both textual and non-textual information are first identified according to image modal data. Textual information can be extracted using technologies such as OCR, while non-textual information has more data. Therefore, defining storage structures containing different content for different types of information is essentially the implementation of heterogeneous information proposed in this invention. Specifically, in the storage entries for non-textual information described in the above embodiments, the content and location information of non-textual information are saved through table appending. This storage structure provides a data interface for subsequent data reading in the implementation of a large language model, facilitating future functional expansion of the large language model.

[0083] In this embodiment of the invention, considering the information query business scenario and the scalability of the interaction system for the entire large language model, the tree topology structure composed of multiple modal storage structures needs to be designed with corresponding performance. Specifically, the database includes:

[0084] The physical sub-database is used to store the document content corresponding to each content node, such as the text content in the storage structure corresponding to each text type information, its own node ID, the screenshot content in the storage structure corresponding to each image type information, its own node ID, original coordinate border, caption text, text in the screenshot, parent node ID, and auxiliary metadata obtained from the target document, etc.

[0085] The logical sub-database is used to store the hierarchical relationship corresponding to each content node, such as storing the feature vector, parent node ID and sibling node ID of the text content in the storage structure corresponding to each text type information, and the feature vector, parent node ID and sibling node ID of the non-screenshot content in the storage structure corresponding to each image type information, etc.

[0086] The semantic sub-database is used to store the reference relationships corresponding to each content node, such as storing the node IDs of other content nodes that each content node points to through the reference relationships.

[0087] Different sub-databases, based on their stored data content, actually embody different logical levels and performance orientations. As described in the above embodiments, the physical sub-database primarily stores structured basic data, the logical sub-database stores high-dimensional data vectors to support semantic retrieval, and the semantic sub-database stores complex node relationships to support semantic extension. During implementation, the overall database is based on a hybrid storage approach using multiple database software to support the functionalities of different sub-databases. For example, the database in this embodiment is implemented using a hybrid storage approach based on MySQL, Neo4J, and Milvus.

[0088] A schema is a crucial tool for logical mapping between sub-databases. In this embodiment, a three-tiered logical sub-database is designed to store different content. For large language models, data retrieval is not only completed within a single sub-database but also involves retrieving related data from different sub-databases. Therefore, designing a schema that supports high-performance retrieval is a vital step in implementing multi-level sub-databases and tree-like topology storage. In this embodiment, the schema includes at least one of the following:

[0089] A document metadata table is used to store the metadata of the target document;

[0090] The core structure table is used to store the indexes of the hierarchical relationships;

[0091] The content details table is used to store the metadata of the document content corresponding to the content node;

[0092] A semantic graph edge table is used to store indexes of the reference relationships;

[0093] A vector table is used to store the vector indexes of the document content corresponding to the content nodes.

[0094] Specifically, in order to achieve lightweight and high-throughput node traversal retrieval in a tree-like topology, some optimization designs can be made during the construction of each data table. The optimization designs made in the implementation of this invention include:

[0095] Data is flattened. For all types of document content, instead of nesting node arrays within document objects of type doc, etc., the data is stored in tables at the database level.

[0096] Node ID version upgrade. Using ordered UUIDs (ULID or UUIDv7) can avoid the overhead of traditional UUIDs for indexing, and also avoid the difficulties of migration or expansion caused by auto-incrementing primary keys.

[0097] Path materialization. A `path` field is added to the metadata (core structure table) of the tree-like topology to identify horizontal and vertical search patterns, giving the large language model stronger context awareness during retrieval and thus enabling it to obtain more information.

[0098] Content separation processing. Text data and vector data are extracted from the main document metadata table to ensure lightweight and high-throughput roaming retrieval.

[0099] S103. Interact with the database based on the large language model to obtain the interaction result.

[0100] Step S103 includes the following sub-steps:

[0101] S1031. Obtain the query entry, perform a vector similarity-based retrieval in the database based on the query entry, obtain multiple content nodes, and use the obtained multiple content nodes as anchor nodes;

[0102] S1032. Based on the association table, the anchor node is searched in the database for upward expansion and / or adjacent expansion and / or cross-level expansion based on the hierarchical relationship and the reference relationship, to obtain multiple content nodes associated with the anchor node, and the obtained multiple content nodes are used as extension nodes;

[0103] S1033. The document content corresponding to the anchor node and the extended node is uniformly organized into interactive text for the output of the large language model, which is used as the interaction result.

[0104] Step S103 is the process of implementing interaction based on the large language model after the target document is set in this embodiment of the invention. This step reflects the function of dynamic retrieval based on the tree topology database. It can be understood that the large language model does not directly participate in the database retrieval. The retrieval process is implemented through a retrieval model. The large language model obtains the input query item and passes it to the retrieval model. After the retrieval model completes the database retrieval, the large language model organizes the retrieval results and outputs the interactive results.

[0105] Based on the retrieval process described in the above embodiments, for example, the query item in step S1031 is obtained by inputting a large language model, and then the query item is transmitted to the retrieval tool corresponding to the database. The retrieval tool performs vector similarity retrieval by retrieving the corresponding relation table and finds the Top-K most relevant anchor nodes. The content of these nodes may be a text paragraph or image deep in the tree.

[0106] In step S1032, by performing upward expansion and / or adjacent expansion and / or cross-level expansion on the anchor node based on the hierarchical relationship and the reference relationship, during the implementation process, since the data stored in the database is represented by a constructed tree topology structure, the parent node of the anchor node can be quickly obtained by passing the node ID, thus supplementing the chapter background to which the paragraph belongs; or by obtaining the previous and next neighbor nodes of the anchor node, thus supplementing the coherent reading context; or by obtaining the nodes referenced by the anchor node, thus supplementing the extended description of the corresponding text paragraph. For example, if the content of the anchor node is "caption", the screenshot content and coordinate selection content attached to other content nodes related to "caption" can be further associated and obtained.

[0107] Finally, the retrieval tool reads this content from the database as extended nodes and returns it to the large language model, realizing content retrieval by traversing the nodes in the tree-like topology. Step S1033, as the output module, involves the large language model automatically organizing and summarizing the document content corresponding to the anchor nodes and extended nodes, and outputting it as the interaction result.

[0108] Throughout the interaction process, the large language model will automatically determine the granularity and specificity of the information retrieved from the database, and feed it back to the retrieval tool through pre-designed association tables and database configuration commands to obtain the required information, thereby achieving multiple rounds of feedback optimization.

[0109] The beneficial effects achieved by this invention lie in proposing a method for expanding a large language model based on heterogeneous information. This method, by constructing a storage structure corresponding to different types of document content, enables the large language model to reduce the loss of document slice information during expansion and interaction, while preserving structural semantics, multimodal associations, and original physical information, thus exhibiting good scalability. At the same time, flexible interaction paradigms can be constructed based on this storage structure, which facilitates dynamic granularity adaptation and multi-turn feedback retrieval, thereby improving the intelligence level and interaction capabilities of AI systems.

[0110] Example 2

[0111] This invention also provides a large language model extension system 200 based on heterogeneous information, please refer to... Figure 2 , Figure 2 This is a schematic diagram of the structure of a large language model extension system based on heterogeneous information provided in an embodiment of the present invention, which includes:

[0112] The document parsing module 201 is used to obtain the target document for knowledge expansion of the large language model, and to identify the document content in the target document by region to obtain multiple content nodes;

[0113] The tree structure module 202 is used to analyze the hierarchical relationship between the document content corresponding to different content nodes, and to construct a tree topology structure to represent the hierarchical relationship using the content nodes as tree nodes. Then, it analyzes the reference relationship between the document content corresponding to different content nodes and assigns the reference relationship to the corresponding content node that has formed the tree topology structure.

[0114] Subsequently, the tree-like topology constructed from the content nodes with the aforementioned referencing relationships is stored in the database as extended data of the large language model; wherein, depending on the type of the document content corresponding to the content node, different storage structures are used to store the content nodes in the database;

[0115] The retrieval reasoning module 203 is used to interact with the database based on the large language model to obtain interaction results.

[0116] The large language model expansion system 200 based on heterogeneous information can implement the steps in the large language model expansion method based on heterogeneous information in the above embodiments, and can achieve the same technical effect. Referring to the description in the above embodiments, it will not be repeated here.

[0117] Example 3

[0118] This invention also provides a computer device, please refer to... Figure 3 , Figure 3 This is a schematic diagram of the structure of a computer device provided in an embodiment of the present invention. The computer device 300 includes: a memory 302, a processor 301, and a large language model extension program based on heterogeneous information stored in the memory 302 and capable of running on the processor 301.

[0119] The processor 301 calls the large language model expansion program based on heterogeneous information stored in the memory 302, and executes the steps in the large language model expansion method based on heterogeneous information provided in this embodiment of the invention. Please refer to... Figure 1 Specifically, it includes the following steps:

[0120] S101. Obtain the target document for knowledge expansion of the large language model, and identify the document content in the target document by region to obtain multiple content nodes.

[0121] The document content includes text information and non-text information. Step S101 also includes the following steps:

[0122] Extract the text contained in the text class information, and use the obtained text as the content node corresponding to the text class information;

[0123] Extract the original coordinate borders of the non-text information in the page of the target document, and capture the corresponding screenshots from the page according to the original coordinate borders. Use the obtained screenshots as the content nodes corresponding to the non-text information.

[0124] S102. Analyze the hierarchical relationship between the document content corresponding to different content nodes, and construct a tree topology structure to represent the hierarchical relationship using the content nodes as tree nodes. Then, analyze the reference relationship between the document content corresponding to different content nodes, and assign the reference relationship to the corresponding content node that has formed the tree topology structure.

[0125] Subsequently, the tree-like topology constructed from the content nodes with the aforementioned referencing relationships is stored in the database as extended data of the large language model; wherein, depending on the type of the document content corresponding to the content node, different storage structures are used to store the content nodes in the database.

[0126] Step S102 includes the following sub-steps:

[0127] The hierarchical relationship of inclusion or adjacency between the document content corresponding to different content nodes is obtained based on a preset hierarchical derivation method.

[0128] Based on the hierarchical relationship, the content node is used as a tree node to construct the tree topology, so that the hierarchical relationship becomes the structural edge of the tree topology, and a corresponding node ID is assigned to each content node in the tree topology.

[0129] The reference relationships between the document content corresponding to different content nodes are obtained based on a preset semantic analysis method, and the reference relationships are assigned to the corresponding content nodes that have formed the tree topology, so that the reference relationships become the semantic edges of the tree topology.

[0130] Different storage structures are used to define the content nodes according to the different types of document content corresponding to the content nodes;

[0131] The storage structure is used as extended data of the large language model in the database;

[0132] Based on the target document, the content nodes, the hierarchical relationships, the referencing relationships, and the tree topology, a related table is constructed for querying the content nodes in the database, and the related table is stored in the database.

[0133] The storage structure includes at least one of the following items: node ID, type of document content, metadata of document content, hierarchical relationship, and reference relationship;

[0134] For the text-type information, the storage structure further includes at least one of the following items: text content, feature vector of text content, parent node ID of the corresponding content node, and sibling node ID of the corresponding content node;

[0135] For the non-text information, the storage structure further includes at least one of the following items: screenshot content, original coordinate border, caption text, text in the screenshot content, storage path of the screenshot content, feature vector of the screenshot content, annotation of the screenshot content, and parent node ID of the corresponding content node.

[0136] The database includes:

[0137] A physical sub-database is used to store the document content corresponding to each of the content nodes;

[0138] A logical sub-database is used to store the hierarchical relationship corresponding to each of the content nodes;

[0139] A semantic sub-database is used to store the reference relationships corresponding to each of the content nodes.

[0140] The association table includes at least one of the following:

[0141] A document metadata table is used to store the metadata of the target document;

[0142] The core structure table is used to store the indexes of the hierarchical relationships;

[0143] The content details table is used to store the metadata of the document content corresponding to the content node;

[0144] A semantic graph edge table is used to store indexes of the reference relationships;

[0145] A vector table is used to store the vector indexes of the document content corresponding to the content nodes.

[0146] S103. Interact with the database based on the large language model to obtain the interaction result.

[0147] Step S103 includes the following sub-steps:

[0148] Obtain the query entry, perform a vector similarity-based retrieval in the database based on the query entry to obtain multiple content nodes, and use the obtained multiple content nodes as anchor nodes;

[0149] Based on the association table, the anchor node is searched in the database by upward expansion and / or adjacent expansion and / or cross-level expansion based on the hierarchical relationship and the reference relationship, to obtain multiple content nodes associated with the anchor node, and the multiple content nodes obtained are used as extension nodes;

[0150] The document content corresponding to each of the anchor nodes and the extended nodes is uniformly organized into interactive text for the output of the large language model, which serves as the interaction result.

[0151] The computer device 300 provided in this embodiment of the invention can implement the steps in the large language model expansion method based on heterogeneous information as described in the above embodiments, and can achieve the same technical effect. Referring to the description in the above embodiments, it will not be repeated here.

[0152] Example 4

[0153] This invention also provides a storage medium storing a large language model extension program based on heterogeneous information. When the large language model extension program based on heterogeneous information is executed by a processor, it implements the various processes and steps in the large language model extension method based on heterogeneous information provided in this invention and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0154] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by hardware related to a large language model extension program or instructions based on heterogeneous information. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0155] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0156] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0157] The embodiments of the present invention have been described above with reference to the accompanying drawings. The disclosed embodiments are merely preferred embodiments of the present invention. However, the present invention is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many equivalent changes in form under the guidance of the present invention without departing from the spirit and scope of the claims, and all such changes are within the protection scope of the present invention.

Claims

1. A method for expanding a large language model based on heterogeneous information, characterized in that, Includes the following steps: S101. Obtain the target document for knowledge expansion of the large language model, and identify the document content in the target document by region to obtain multiple content nodes; S102. Analyze the hierarchical relationship between the document content corresponding to different content nodes, and construct a tree topology structure to represent the hierarchical relationship using the content nodes as tree nodes. Then, analyze the reference relationship between the document content corresponding to different content nodes, and assign the reference relationship to the corresponding content node that has formed the tree topology structure. Subsequently, the tree-like topology constructed from the content nodes with the aforementioned referencing relationships is stored in the database as extended data of the large language model; wherein, depending on the type of the document content corresponding to the content node, different storage structures are used to store the content nodes in the database; S103. Interact with the database based on the large language model to obtain the interaction result; Step S102 includes the following sub-steps: The hierarchical relationship of inclusion or adjacency between the document content corresponding to different content nodes is obtained based on a preset hierarchical derivation method. Based on the hierarchical relationship, the content node is used as a tree node to construct the tree topology, so that the hierarchical relationship becomes the structural edge of the tree topology, and a corresponding node ID is assigned to each content node in the tree topology. The reference relationships between the document content corresponding to different content nodes are obtained based on a preset semantic analysis method, and the reference relationships are assigned to the corresponding content nodes that have formed the tree topology, so that the reference relationships become the semantic edges of the tree topology. Different storage structures are used to define the content nodes according to the different types of document content corresponding to the content nodes; The storage structure is used as extended data of the large language model in the database; Based on the target document, the content nodes, the hierarchical relationships, the referencing relationships, and the tree topology, a related table is constructed for querying the content nodes in the database, and the related table is stored in the database.

2. The method for expanding a large language model based on heterogeneous information according to claim 1, characterized in that, The document content includes text information and non-text information. Step S101 also includes the following steps: Extract the text contained in the text class information, and use the obtained text as the content node corresponding to the text class information; Extract the original coordinate borders of the non-text information in the page of the target document, and capture the corresponding screenshots from the page according to the original coordinate borders. Use the obtained screenshots as the content nodes corresponding to the non-text information.

3. The method for expanding a large language model based on heterogeneous information according to claim 2, characterized in that, The storage structure includes at least one of the following items: node ID, type of document content, metadata of document content, hierarchical relationship, and reference relationship; For the text-type information, the storage structure further includes at least one of the following items: text content, feature vector of text content, parent node ID of the corresponding content node, and sibling node ID of the corresponding content node; For the non-text information, the storage structure further includes at least one of the following items: screenshot content, original coordinate border, caption text, text in the screenshot content, storage path of the screenshot content, feature vector of the screenshot content, annotation of the screenshot content, and parent node ID of the corresponding content node.

4. The method for expanding a large language model based on heterogeneous information according to claim 1, characterized in that, The database includes: A physical sub-database is used to store the document content corresponding to each of the content nodes; A logical sub-database is used to store the hierarchical relationship corresponding to each of the content nodes; A semantic sub-database is used to store the reference relationships corresponding to each of the content nodes.

5. The method for expanding a large language model based on heterogeneous information according to claim 4, characterized in that, The association table includes at least one of the following: A document metadata table is used to store the metadata of the target document; The core structure table is used to store the indexes of the hierarchical relationships; The content details table is used to store the metadata of the document content corresponding to the content node; A semantic graph edge table is used to store indexes of the reference relationships; A vector table is used to store the vector indexes of the document content corresponding to the content nodes.

6. The method for expanding a large language model based on heterogeneous information according to claim 1, characterized in that, Step S103 includes the following sub-steps: Obtain the query entry, perform a vector similarity-based retrieval in the database based on the query entry to obtain multiple content nodes, and use the obtained multiple content nodes as anchor nodes; Based on the association table, the anchor node is searched in the database by upward expansion and / or adjacent expansion and / or cross-level expansion based on the hierarchical relationship and the reference relationship, to obtain multiple content nodes associated with the anchor node, and the multiple content nodes obtained are used as extension nodes; The document content corresponding to each of the anchor nodes and the extended nodes is uniformly organized into interactive text for the output of the large language model, which serves as the interaction result.

7. A large language model extension system based on heterogeneous information, characterized in that, include: The document parsing module is used to obtain target documents for knowledge expansion of large language models, and to identify the document content in the target documents by region to obtain multiple content nodes; The tree structure module is used to analyze the hierarchical relationship between the document content corresponding to different content nodes, and to construct a tree topology structure to represent the hierarchical relationship using the content nodes as tree nodes. Then, it analyzes the reference relationship between the document content corresponding to different content nodes and assigns the reference relationship to the corresponding content node that has formed the tree topology structure. Subsequently, the tree-like topology constructed from the content nodes with the aforementioned referencing relationships is stored in the database as extended data of the large language model; wherein, depending on the type of the document content corresponding to the content node, different storage structures are used to store the content nodes in the database; The retrieval and reasoning module is used to interact with the database based on the large language model and obtain the interaction results; Specifically, the tree structure module is used for: The hierarchical relationship of inclusion or adjacency between the document content corresponding to different content nodes is obtained based on a preset hierarchical derivation method. Based on the hierarchical relationship, the content node is used as a tree node to construct the tree topology, so that the hierarchical relationship becomes the structural edge of the tree topology, and a corresponding node ID is assigned to each content node in the tree topology. The reference relationships between the document content corresponding to different content nodes are obtained based on a preset semantic analysis method, and the reference relationships are assigned to the corresponding content nodes that have formed the tree topology, so that the reference relationships become the semantic edges of the tree topology. Different storage structures are used to define the content nodes according to the different types of document content corresponding to the content nodes; The storage structure is used as extended data of the large language model in the database; Based on the target document, the content nodes, the hierarchical relationships, the referencing relationships, and the tree topology, a related table is constructed for querying the content nodes in the database, and the related table is stored in the database.

8. A computer device, characterized in that, include: The system includes a memory, a processor, and a large language model extension program based on heterogeneous information stored in the memory and executable on the processor. When the processor executes the large language model extension program based on heterogeneous information, it implements the steps of the large language model extension method based on heterogeneous information as described in any one of claims 1-6.

9. A storage medium, characterized in that, The storage medium stores a large language model extension program based on heterogeneous information. When the large language model extension program based on heterogeneous information is executed by the processor, it implements the steps in the large language model extension method based on heterogeneous information as described in any one of claims 1-6.

Images