Entity extraction, storage and retrieval method based on LLM and multiple databases

Abstract

The invention relates to an entity extraction, storage and retrieval method based on LLM and multiple databases, entity information is extracted from a to-be-stored data set by constructing a prompt based on the LLM, an entity ID is generated based on the entity information, and entity attributes and relationships are written into a full-text library and a graph database respectively in the data storage process and are bound through the entity ID, so that the entity extraction efficiency is improved, and the data storage efficiency is improved. And the performance pressure of mass data storage of a single-class database is reduced. And after classification, synchronously writing into a full-text library and a graph database, writing entity attribute entries into the full-text library and binding the entity attribute entries with entity IDs previously written into the full-text library, and writing entity relationships into the graph database and binding the entity relationships with entity IDs previously written into the graph database. The data retrieval service is provided through the unified retrieval entry, the user does not need to increase the learning cost, the search scene is automatically judged to query the target database, the performance advantages of each database are fully played, and the mass data storage and retrieval efficiency is further improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of big data storage and analysis, and particularly relates to a method for entity extraction, storage, and retrieval based on LLM and multiple databases. Background Art

[0002] In recent years, with the highly developed informatization and digitization, the emergence of various big data application platforms has further exploited the data value. Big data has become a force that cannot be ignored. It is like an infinite treasure house, containing countless opportunities and challenges.

[0003] In the current data storage field, full-text databases are often used to store and manage a large amount of text data. Users can search for entity attributes stored in the database through keywords or phrases to find the required information. However, it is difficult to reflect the relationships between data in full-text database storage, and it is hard to discover the connections between data. Graph databases can easily represent and process various types of relationships, but they are not flexible enough in retrieval. If there is a method that can combine these two types of databases during the data storage and retrieval process, giving full play to the performance advantages of each database, the efficiency of massive data storage and retrieval can be further improved. At the same time, when facing massive text data, manual marking for entity extraction requires a huge amount of human and time costs, and the context understanding ability of entity recognition in traditional NLP is weak. It is difficult to handle polysemous words and to a certain extent relies on manual feature engineering, which is time-consuming and laborious. If entity extraction can be combined with large models, giving full play to the semantic understanding ability of large models, the efficiency of entity extraction can be further improved.

[0004] Therefore, the problems existing in the above-mentioned prior art, such as the difficulty in reflecting the relationships between data in full-text database storage, the hard discovery of connections between data, the lack of flexibility in retrieval, the huge human and time costs required for manual marking for entity extraction, and the weak context understanding ability of entity recognition, are the technical problems that need to be solved urgently at present. Summary of the Invention

[0005] The purpose of the present invention is to provide a method for entity extraction, storage, and retrieval based on LLM and multiple databases, so as to solve the technical problems existing in the prior art, such as the difficulty in reflecting the relationships between data in full-text database storage, the hard discovery of connections between data, the lack of flexibility in retrieval, the huge human and time costs required for manual marking for entity extraction, and the weak context understanding ability of entity recognition.

[0006] To solve the above technical problems, the technical solution adopted by the present invention is as follows: A method for entity extraction, storage, and retrieval based on LLM and multiple databases, comprising the following steps: S1: Load the files to be stored in the specified directory, map the data of the files to be stored into continuous vectors and store them, and construct a prompt based on the LLM to extract entity information from the dataset to be stored; S2: Generate entity IDs based on the entity information, classify the entity information and synchronously write it into the full-text database and the graph database; S3: Write the entity attribute entries into the full-text database and bind them to the entity IDs previously written into the full-text database, and write the entity relationships into the graph database and bind them to the entity IDs previously written into the graph database; S4: Implement entity attribute retrieval by directly retrieving the entity attributes stored in the full-text database for quick matching, and perform associated retrieval in the graph database by matching entity IDs.

[0007] Preferably, the specific process of mapping the data of the files to be stored into continuous vectors in step S1 is as follows: S11: Split the data of the files to be stored to obtain a number of discrete data units, and assign an index to each discrete data unit; S12: Construct a trainable embedding matrix with dimensions V×D. For each discrete data unit, look up the corresponding row vector from the embedding matrix according to its index, and the dimension of the row vector is D; S13: If the input is a single file unit, the output is a D-dimensional vector; If the input is a sequence, the vector sequence is calculated by taking the mean of all vector elements, or concatenating the head and tail of the sequence vectors into a long vector, or weighted aggregation into a single continuous vector; S14: Output a continuous vector of fixed length and store it in the vector store.

[0008] Preferably, the specific process of constructing a prompt based on the LLM to extract entity information from the dataset to be stored in step S1 is as follows: S15: Construct a prompt based on the LLM. The constructed prompt includes a candidate label list schema schema and a structured output format format to be extracted. The candidate label list schema schema includes an entityTypeList and a relationTypeList. The entityTypeList is an entity type or an event type, and the relationTypeList is a relation type. The structured output format format includes: an object list definition extractObjectData and a relation list definition extractRelationData; S16: Extract one or two of entity attributes and entity relationships from the dataset to be stored. The extracted entity attributes and entity relationships are in an independent key-value pair structure / set of key-value pairs. The entity relationship includes a relationship start node, a relationship type, and a relationship target node. The relationship start node and the relationship target node are referred to using entity IDs.

[0009] Preferably, the specific process of generating entity IDs based on entity information in step S2 is as follows: Concatenate the representative attribute value of the entity information with the entity or event name in the candidate label entityTypeList hit by the entity in the string buffer using a specified separator. After taking out the concatenated string, create a digital fingerprint using a hashing algorithm.

[0010] Preferably, the specific process of classifying the entity information and synchronously writing it into the full-text database and the graph database in step S2 is as follows: S21: After parsing the key-value pairs of entity attributes, write them into the full-text retrieval database according to the preset field mapping rules to support subsequent text queries and content retrievals. The full-text database is a system that supports full-text indexing and searching; S22: According to the different databases written, set the entity ID generated above as the docId of the full-text database or other identifiers used to represent specific records. The written data includes text fields, keyword fields, labels, and classification information; S23: After parsing the key-value pairs of entity relationships, extract the node information representing the entities, including the relationship start node and the relationship target node, and the edge information representing the association between entities. According to the modeling specifications of the graph database, standardize the node attributes, types, and the relationship types and start and end entities of the edges, and write them into the graph database to complete the construction of the entity graph; S24: According to the different databases written, set the relationship start node and the relationship target node as the node IDs of the graph database or other identifiers used to represent specific instances.

[0011] Preferably, the specific process of step S3 is as follows: S31: Write into the full-text database through the index interface, specifying the entity ID and attribute data. The entity ID is used as the unique key of the full-text database document, and the attribute data is strongly bound to the entity through the entity ID to ensure that it can be quickly located by the ID during subsequent retrievals. After writing, verify whether the data is correctly stored through the get interface to ensure that the attribute value is consistent with the input; S32: Parse the entity relationship data, and query whether from_id and to_id exist in the graph database through node labels and IDs. If the node does not exist, the node needs to be created first or an exception needs to be thrown; S33: Obtain entity node IDs in the graph database. Entity node IDs in the graph database are uniquely identified by labels and attributes. The business ID is mapped to the entity node ID. A relationship is established using the node IDs of the start point and the end point, and the relationship type and attributes are specified.

[0012] Preferably, the specific process of step S4 is as follows: S41: Setting a search keyword, encapsulating the keyword into a full-text search request, wherein the full-text search request includes search fields, matching methods, and sorting strategy information; S42: Submit the full-text search request to the full-text library. The system matches relevant documents in the specified field based on the inverted index mechanism and returns the matching results. The matching results include entity attributes and record identifiers of the results. S43: Performing association search: Selecting a matching result, obtaining the entity ID of the selected result as the node ID in the graph database, querying the graph database through the specified node ID, locating the target node, and performing adjacency search based on the outgoing and incoming edges of the node to obtain the relationship information associated with the node; S44: After locating the associated node, by obtaining the target associated node ID, the node ID is used as a record identifier of the full-text library to initiate a request to the full-text library to obtain the attribute results of the associated node.

[0013] Preferably, the process of encapsulating keywords into full-text search requests in step S41 is: converting keyword input into a full-text search request structure including field range, matching strategy and sorting configuration, and calling the standardized query interface of the full-text library through an HTTP request.

[0014] Preferably, step S4 provides data retrieval services through a unified retrieval portal, supports the use of logical operators, wildcards, and fuzzy searches for precise retrieval when performing entity attribute retrieval, thereby improving the efficiency of data retrieval, and supports association retrieval including path discovery, clustering and multiple graph algorithms to deeply mine hidden associations and patterns in the data.

[0015] The beneficial effects of the present invention include: The entity extraction, storage, and retrieval method based on LLM and multiple databases provided by the present invention constructs a prompt based on LLM to extract entity information from the dataset to be stored, generates entity IDs based on the entity information, writes entity attributes and relationships into the full-text database and the graph database respectively during the data storage process, and binds them through the entity IDs, improving the efficiency of entity extraction and reducing the performance pressure of storing massive data in a single-category database. After classification, it is synchronously written into the full-text database and the graph database. The entity attribute entries are written into the full-text database and bound to the entity IDs previously written into the full-text database. The entity relationships are written into the graph database and bound to the entity IDs previously written into the graph database. Entity attribute retrieval is achieved by directly retrieving the entity attributes stored in the full-text database for quick matching, and associated retrieval is performed in the graph database by matching the entity IDs. Through a unified retrieval entry, data retrieval services are provided. Users do not need to increase the learning cost, and the search scenario is automatically judged to query the target database, giving full play to the performance advantages of each database and further improving the efficiency of massive data storage and retrieval.

[0016] First, by constructing a prompt containing a candidate label list and a structured output format through LLM, the entity attributes and relationships in complex texts can be accurately extracted, supporting the custom extension of entity types and relationship types. There is no need for hard-coded rules, and the adaptability is significantly improved. The key-value pair structured output converts unstructured data into a machine-processable format. The entity relationships directly use entity IDs to refer to nodes, which naturally adapts to the association modeling requirements of the graph database and reduces the manual mapping cost.

[0017] Second, through discrete data unit segmentation and embedding matrix mapping, vector representations of multiple types of files such as text, images, and audio are supported: text data can quickly generate vectors through word segmentation indexing, retaining the context semantics; image / audio data is compressed into vectors of a unified length through a fixed-dimensional embedding matrix, facilitating storage in the Vector Store and supporting subsequent similarity retrieval, flexibly adapting to different data forms.

[0018] Third, through the inverted index mechanism and the support of logical operators / wildcards, a combined retrieval of "keyword fuzzy matching + field precise filtering" is achieved. For example, when retrieving "persons whose labels contain 'NLP' and are classified as 'employees'", the result can be returned in milliseconds, effectively improving the query efficiency compared to traditional databases. The sorting strategy optimization ensures that relevant results are displayed first, combined with front-end highlighting to enhance the user retrieval experience.

[0019] Finally, through the semantic understanding ability of LLM, the automatic extraction of entities and relationships is realized. Combined with a multi-database collaborative storage architecture, namely the full-text database + graph database + Vector Store, a complete link of "data extraction - structured storage - intelligent retrieval - association analysis" is constructed. Description of the Drawings

[0020] Figure 1 This is a schematic diagram of the working principle of the entity extraction, storage, and retrieval method based on LLM and multiple databases of the present invention.

[0021] Figure 2 This is a schematic diagram of the process of the entity extraction, storage, and retrieval method based on LLM and multiple databases of the present invention. Detailed implementation manners

[0022] The following further elaborates on the present invention in conjunction with the attached Figures 1 - 2 : Example 1 Refer to the attached Figure 1 and Figure 2 As shown, an entity extraction, storage, and retrieval method based on LLM and multiple databases includes the following steps: S1: Load the files to be stored in the specified directory. The file formats include: txt, pdf, docx, xlsx, ppt, etc. Map the data of the files to be stored into continuous vectors and store them. Then perform embedding on the file content to convert the text data into vectors and store them in the vector store. Then construct a prompt based on the LLM, ask questions and send them into the LLM model. Finally, obtain the response result and parse out the entity information to realize the extraction of entity information from the dataset to be stored.

[0023] S2: Generate entity IDs based on the entity information, classify the entity information and synchronously write it into the full-text database and the graph database. The classification and synchronous writing of the entity information into the full-text database and the graph database are realized through HTTP requests, RESTful APIs or client SDKs, and are realized through graph database interfaces (such as Cypher, GQL or HTTP API). The data writing supports synchronous or asynchronous modes.

[0024] S3: Write the entity attribute entries into the full-text database and bind them to the entity IDs previously written into the full-text database, and write the entity relationships into the graph database and bind them to the entity IDs previously written into the graph database; S4: Realize entity attribute retrieval by directly retrieving the entity attributes stored in the full-text database for quick matching, and perform associated retrieval in the graph database by matching the entity IDs.

[0025] In the current data storage field, full-text databases are often used to store and manage a large amount of text data. Users can search for entity attributes stored in the database through keywords or phrases to find the required information. However, it is difficult to reflect the relationships between various data in full-text database storage, and it is hard to discover the connections between data. Graph databases can easily represent and process various types of relationships, but they are not flexible enough in terms of retrieval. If there is a method that can combine these two types of databases during the data storage and retrieval process to give full play to the performance advantages of each database, the efficiency of massive data storage and retrieval can be further improved. At the same time, when facing massive text data, manual marking for entity extraction requires a huge amount of human and time costs. Traditional NLP entity recognition has weak context understanding ability, is difficult to handle polysemous words, and to a certain extent relies on manual feature engineering, which is time-consuming and laborious. Therefore, the present invention combines entity extraction with a large model to give full play to the semantic understanding ability of the large model and achieve the improvement of the efficiency of entity extraction.

[0026] Embodiment 2 Based on Embodiment 1, the specific process of mapping the file data to be stored into continuous vectors in step S1 is as follows: S11: Split the file data to be stored to obtain a number of discrete data units, and assign an index to each discrete data unit; S12: Construct a trainable embedding matrix with dimensions V×D. For each discrete data unit, look up the corresponding row vector from the embedding matrix according to its index. The dimension of the row vector is D; S13: If the input is a single file unit, the output is a D-dimensional vector; If the input is a sequence, the vector sequence is calculated by taking the mean of all vector elements, or by concatenating the head and tail of the sequence vectors into a long vector, or by weighted aggregation into a single continuous vector; S14: Output a continuous vector of a fixed length and store it in the vector store.

[0027] In this embodiment, the specific process of extracting entity information from the dataset to be stored based on the LLM to construct a prompt in step S1 is as follows: S15: Construct a prompt based on the LLM. The constructed prompt includes a candidate label list schema to be extracted and a structured output format format. The candidate label list schema includes an entityTypeList and a relationTypeList. The entityTypeList is the entity type or event type, and the relationTypeList is the relation type. In actual embodiments, the candidate label schema must include at least one of the above two types.

[0028] The entityTypeList in the candidate tag schema contains a list of objects, each having at least one entity or event name (name), at least one entity or event proxy name (showName), and at least one attribute field (fields). The relationTypeList in the candidate tag schema contains a list of objects, each having a relation name (showName), a relation type (type), a start node name (startObjectName), and an end node name (endObjectName).

[0029] The structured output format format includes: the object list definition extractObjectData and the relation list definition extractRelationData. The object list definition extractObjectData should include the information of the hit candidate tag entityTypeList and the corresponding entity object. The relation list definition extractRelationData should include the information of the hit candidate tag relationTypeList and the corresponding relation object.

[0030] S16: Extract one or both of the entity attributes and entity relationships from the dataset to be stored. The extracted entity attributes and entity relationships are in an independent key-value pair structure / set of key-value pairs. The entity relationship includes a relationship start node, a relationship type, and a relationship target node. The relationship start node and the relationship target node are referred to by entity IDs.

[0031] The specific process of generating entity IDs based on entity information in step S2 is as follows: Concatenate the representative attribute value of the entity information with the entity or event name in the entityTypeList of the candidate tag hit by the entity in the string buffer using a specified separator. After taking out the concatenated string, create a digital fingerprint using a hashing algorithm.

[0032] The specific process of classifying the entity information and synchronously writing it into the full-text database and the graph database in step S2 is as follows: S21: Parse the key-value pairs of the entity attributes and write them into the full-text retrieval database according to the preset field mapping rules to support subsequent text queries and content retrievals. The full-text database is a system that supports full-text indexing and searching.

[0033] The key-value parsing of entity attributes includes extracting key-value pairs and validating required fields. According to the Mapping configuration of the full-text library index, map the attribute fields to specific types, such as text, keyword, date. For text-type fields, use a tokenizer, such as ik_max_word of elasticsearch, to generate a token list and build an inverted index. Write data using the API provided by the full-text library, specifying the value of the document ID field.

[0034] S22: According to the different databases to which data is written, set the entity ID generated above as the docId of the full-text library or other identifiers used to represent specific records. The data written includes text fields, keyword fields, tags, and classification information.

[0035] If the input data already contains an entity_id, directly use it as the docId. If the input data has no ID, generate it through a globally unique algorithm. Mapping of the full-text library document ID: Specify the docId through the id parameter in the write request. If not specified, the full-text library will automatically generate a random ID, but the mapping relationship between this ID and the business entity needs to be stored additionally. Specification of the data fields written: Required fields: entity_id as the docId. The text field is used for full-text retrieval, the keyword field is used for precise filtering. The classification field is used for faceted search. Use the generated entity ID as the document identifier of the full-text library to ensure data uniqueness and retrievability.

[0036] S23: After parsing the key values of the entity relationship, extract the node information representing the entity, including the relationship start node and relationship target node, and the edge information representing the association between entities. According to the modeling specifications of the graph database, standardize the node attributes, types, and the relationship types, start and end entities of the edges, and write them into the graph database to complete the construction of the entity graph; S24: According to the different databases to which data is written, set the relationship start node and relationship target node as the node ID of the graph database or other identifiers used to represent specific instances.

[0037] Embodiment 3 Based on Embodiment 1 or Embodiment 2, the specific process of step S3 is as follows: S31: Write to the full-text library through the index interface, specifying the entity ID and attribute data. The entity ID is used as the unique key of the full-text library document, and the attribute data is strongly bound to the entity through the entity ID to ensure quick positioning through the ID during subsequent retrieval. After writing, verify whether the data is correctly stored through the get interface to ensure that the attribute values are consistent with the input; S32: Parse entity relationship data and query whether from_id and to_id exist in the graph database through node labels and IDs. If the node does not exist, create the node first or throw an exception. S33: Obtain entity node IDs in the graph database. Entity node IDs in the graph database are uniquely identified by labels and attributes. The business ID is mapped to the entity node ID. A relationship is established using the node IDs of the start point and the end point, and the relationship type and attributes are specified.

[0038] The specific process of step S4 is as follows: S41: Setting a search keyword, encapsulating the keyword into a full-text search request, wherein the full-text search request includes search fields, matching methods, and sorting strategy information; S42: Submit the full-text search request to the full-text library. The system matches relevant documents in the specified field based on the inverted index mechanism and returns the matching results. The matching results include entity attributes and record identifiers of the results. S43: Performing association search: Selecting a matching result, obtaining the entity ID of the selected result as the node ID in the graph database, querying the graph database through the specified node ID, locating the target node, and performing adjacency search based on the outgoing and incoming edges of the node to obtain the relationship information associated with the node; S44: After locating the associated node, by obtaining the target associated node ID, the node ID is used as a record identifier of the full-text library to initiate a request to the full-text library to obtain the attribute results of the associated node.

[0039] Preferably, the process of encapsulating keywords into full-text search requests in step S41 is: converting keyword input into a full-text search request structure including field range, matching strategy and sorting configuration, and calling the standardized query interface of the full-text library through an HTTP request.

[0040] Preferably, step S4 provides data retrieval services through a unified retrieval portal, supports the use of logical operators, wildcards, and fuzzy searches for precise retrieval when performing entity attribute retrieval, thereby improving the efficiency of data retrieval, and supports association retrieval including path discovery, clustering and multiple graph algorithms to deeply mine hidden associations and patterns in the data.

[0041] In this embodiment, taking the file format of docx as an example, part of the file content is as follows: Table 1 Employee ID Statistics Serial number Employee number Name 1 00001201 Zhang XX 2 00001202 Yan XX Then, the file content is embedded and the text data is converted into a vector, that is, the file to be stored is mapped into a continuous vector and stored in the vector store.

[0042] Based on LLM, a prompt is constructed, and questions are asked and sent to the LLM model. The constructed prompt contains: the candidate tag list schema to be extracted and the structured output format format. The candidate tag list schema is divided into two categories: entityTypeList and relationTypeList. The entityTypeList is the entity type or event type, and the relationTypeList is the relationship type. The entityTypeList in the candidate tag schema contains at least one object list with an entity or event name (name), at least one entity or event referent name (showName), and at least one attribute field (fields). The relationTypeList in the candidate tag schema contains at least one object list with a relationship name (showName), a relationship type (type), a start node name (startObjectName), and an end node name (endObjectName).

[0043] In this embodiment, the schema in the constructed prompt includes both entityTypeList and relationTypeList.

[0044] The structure of entityTypeList is as follows: [ { "name": "employee", "showName": "Staff", "fields": [ { "name": "Name", "primary": false }, { "number": "Work number", "primary": true } ] } ].

[0045] The entityTypeList contains 1 entity name (name), 1 entity proxy name (showName), and 2 attribute fields (fields). The entity name (name) is "employee", the entity proxy name (showName) is "employee", and the attribute fields (fields) are "name" and "number", which represent "person's name" and "employee number" respectively. "number" is specified as the representative attribute.

[0046] The structure of the relationTypeList is as follows: { "type": "colleague ", "showName": "colleague", "startObjectName": "employee", "endObjectName": "employee" } This relationTypeList contains 1 relation name (showName), 1 relation type (type), 1 start node name (startObjectName), and 1 end node name (endObjectName). The relation name (showName) is "colleague", the relation type (type) is "colleague", the start node name (startObjectName) is "employee", and the end node name (endObjectName) is "employee".

[0047] The structured output format format contains: the object list definition extractObjectData and the relation list definition extractRelationData. The object list definition extractObjectData should contain the information of the hit candidate label entityTypeList and the corresponding entity object. The relation list definition extractRelationData should contain the information of the hit candidate label relationTypeList and the corresponding relation object.

[0048] format contains both the object list definition extractObjectData and the relation list definition extractRelationData.

[0049] The structure of extractObjectData is as follows:​​ { "entityType": "The name corresponding to Entity 1", "fields": { "key": "Entity Attribute 1", "value": "Entity Attribute Value 1" } } The structure of extractRelationData is as follows: { "relationType": "The type corresponding to the relationship", "entityTypeFrom": "Entity name", "entityTypeTo": "Entity name", "from": "Entity ID", "to": "Entity ID" } Finally, the response result is obtained and the entity information is parsed out.

[0050] In this embodiment, the response result should be: { "extractObjectData": { "entityType": "employee", "id": "8f1b55e8e6a2d99cd9c4c30143bfcfa2", "fields": { "key": "name", "value": "Zhang XX" }, { "key": "number", "value": "00001201" } }, { "entityType": "employee", ​​​​​​"id": "1d504ea54ca863646cf8425a2dcd6475", "fields": { "key": "name", "value": "Yan" }, { "key": "number", "value": "00001202" } } , "extractRelationData": { "relationType": "colleague", "entityTypeFrom": "employee", "entityTypeTo": "employee", "from": "8f1b55e8e6a2d99cd9c4c30143bfcfa2", "to": "1d504ea54ca863646cf8425a2dcd6475" } } The response result of this embodiment contains two instances of "Zhang" and "Yan" with the entity name (name) being "employee", and their entity IDs are "8f1b55e8e6a2d99cd9c4c30143bfcfa2" and "1d504ea54ca863646cf8425a2dcd6475" respectively; the response result of this embodiment contains one relationship with the relationship type (type) being "colleague", the ID of the starting node of the relationship is "8f1b55e8e6a2d99cd9c4c30143bfcfa2", and the ID of the target node of the relationship is "1d504ea54ca863646cf8425a2dcd6475".

[0051] In step B1, it is necessary to parse the response result of Embodiment 1.

[0052] Subsequently, in step B2, the entity attribute extractObjectData and the entity relationship extractRelationData are obtained.​​

[0053] The result of the entity attribute extractObjectData in this embodiment is: { "entityType": "employee", "id": "8f1b55e8e6a2d99cd9c4c30143bfcfa2", "fields": { "key": "name", "value":"Zhang XX" }, { "key": "number", "value": "00001201" } }, { "entityType": "employee", "id": "1d504ea54ca863646cf8425a2dcd6475", "fields": { "key": "name", "value": "Yan XX" }, { "key": "number", "value": "00001202" } } The result of the entity relationship extractRelationData in this embodiment is: { "relationType": "colleague", "entityTypeFrom": "employee", "entityTypeTo": "employee", "from": "8f1b55e8e6a2d99cd9c4c30143bfcfa2",​​​​​ "to": "1d504ea54ca863646cf8425a2dcd6475" } Next, in step B3, parse the entity attributes, write them into the full-text retrieval library according to the preset field mapping rules, and use the entity ID as the docID of the full-text library; at the same time, parse the entity relationship key-value pairs, and write the entity ID as the node ID and the relationship type type as the edge type into the graph database.

[0054] In this embodiment, the method for writing entity attributes into the full-text retrieval library is as follows: Set the field mapping rules, convert the "fields" in the extractObjectData result key-value pairs into ordinary fields, and call the RESTful API to write the entity attributes into the full-text library. The converted entity attribute result is: {"index":{"_index":"employees","_id":"8f1b55e8e6a2d99cd9c4c30143bfcfa2"}} {"entityType":"employee","name":"Zhang XX","number":"00001201"} {"index":{"_index":"employees","_id":"1d504ea54ca863646cf8425a2dcd6475"}} {"entityType":"employee","name":"Yan XX","number":"00001202"} Among them, the values of the key-value pairs of "_index" and "entityType" are the entity names (name) in the response result, "name" and "number" are the attribute fields (fields) defined in the entityTypeList of the candidate label schema, and the value of the key-value pair of "_id" is the entity ID in the response result. By setting the value of the full-text library "_id" (i.e., docId) to the entity ID, the entity attributes are bound to the entity ID in the full-text library.

[0055] In this embodiment, the method for writing entity relationships into the graph database is as follows: ​Parse the start and end points of the relationship data in extractRelationData. For this embodiment, there is an edge of type "colleague" in the entity relationship extractRelationData that points from the node "8f1b55e8e6a2d99cd9c4c30143bfcfa2" to "1d504ea54ca863646cf8425a2dcd6475". In this embodiment, the entity relationship data is stored by constructing GQL and calling the graph database interface. By setting the entity ID as the node ID in the graph database, the entity relationship is bound to the entity ID in the graph database.

[0056] Set the search keywords, encapsulate the keywords into a full-text retrieval request, and this request can contain information such as retrieval fields, matching methods (such as fuzzy matching, exact word matching), and sorting strategies. After the request is submitted to the full-text library, the system matches relevant documents in the specified fields based on the inverted index mechanism and returns the matching results. The matching results should include entity attributes and the record identifier (i.e., entity ID) of the belonging result.

[0057] When performing entity attribute retrieval in step C1, functions such as logical operators, wildcards, and fuzzy search are supported for precise retrieval, thereby improving the efficiency of data retrieval. Select the matching results, obtain the record identifier (entity ID) of the selected results as the node ID in the graph database, query the graph database through the specified node ID, locate the target node, and perform adjacency relationship retrieval based on the outgoing and incoming edges of this node, so as to obtain the relationship information associated with this node. Performing association retrieval supports various graph algorithms such as path discovery and clustering, and can deeply mine the associations and patterns hidden in the data. Then, by obtaining the target associated node ID and using the node ID as the record identifier of the full-text library to initiate a request to the full-text library to further obtain the attribute results of the associated nodes.

[0058] In summary, the entity extraction, storage, and retrieval method based on LLM and multi-databases provided by the present invention extracts entity information from the dataset to be stored based on LLM to construct a prompt, generates entity IDs based on the entity information, writes entity attributes and relationships into the full-text library and the graph database respectively during the data storage process and binds them through the entity ID, improving the efficiency of entity extraction and reducing the performance pressure of a single-category database for storing massive data. After classification, it is synchronously written into the full-text library and the graph database. The entity attribute entries are written into the full-text library and bound to the entity ID previously written into the full-text library, and the entity relationships are written into the graph database and bound to the entity ID previously written into the graph database. Through a unified retrieval entry, data retrieval services are provided. Users do not need to increase the learning cost, automatically judge the search scenario to query the target database, give full play to the performance advantages of each database, and further improve the efficiency of massive data storage and retrieval.

Claims

1. An entity extraction, storage, and retrieval method based on LLM and multiple databases, characterized in that, It includes the following steps: S1: Load the files to be stored in the specified directory, map the data of the files to be stored into continuous vectors and store them, and extract entity information from the dataset to be stored based on a prompt constructed by an LLM; S2: Generate entity IDs based on the entity information, classify the entity information and write it to the full-text database and the graph database synchronously; S3: Write the entity attribute entries to the full-text database and bind them to the entity IDs previously written to the full-text database, and write the entity relationships to the graph database and bind them to the entity IDs previously written to the graph database; S4: Realize entity attribute retrieval by directly retrieving the entity attributes stored in the full-text database for quick matching, and perform associated retrieval in the graph database by matching entity IDs.

2. The entity extraction, storage, and retrieval method based on LLM and multiple databases according to claim 1, characterized in that, The specific process of mapping the data of the files to be stored into continuous vectors in step S1 is as follows: S11: Split the data of the files to be stored to obtain a number of discrete data units, and assign an index to each discrete data unit; S12: Construct a trainable embedding matrix with dimensions V×D. For each discrete data unit, look up the corresponding row vector from the embedding matrix according to its index, and the dimension of the row vector is D; S13: If the input is a single file unit, the output is a D-dimensional vector; If the input is a sequence, the vector sequence is calculated by taking the mean of all vector elements, or by concatenating the head and tail of the sequence vectors into a long vector, or by weighted aggregation into a single continuous vector; S14: Output a continuous vector of fixed length and store it in the vector store.

3. The entity extraction, storage, and retrieval method based on LLM and multiple databases according to claim 2, characterized in that The specific process of extracting entity information from the dataset to be stored based on a prompt constructed by an LLM in step S1 is as follows: S15: Construct a prompt based on the LLM. The constructed prompt includes a candidate label list schema schema and a structured output format format to be extracted. The candidate label list schema schema includes an entityTypeList and a relationTypeList. The entityTypeList is the entity type or event type, and the relationTypeList is the relation type. The structured output format format includes: an object list definition extractObjectData and a relation list definition extractRelationData; S16: Extract one or both of the entity attributes and entity relationships from the dataset to be stored. The extracted entity attributes and entity relationships are in an independent key-value pair structure / set of key-value pairs. The entity relationship includes a relationship start node, a relationship type, and a relationship target node. The relationship start node and the relationship target node are referred to by entity IDs.

4. A method for entity extraction, storage, and retrieval based on LLM and multiple databases according to claim 1, characterized in that, The specific process of generating entity IDs based on the entity information in step S2 is as follows: Concatenate the representative attribute value of the entity information with the entity or event name in the candidate label entityTypeList hit by the entity in the string buffer through a specified delimiter, and after taking out the concatenated string, create a digital fingerprint using a hashing algorithm.

5. A method for entity extraction, storage, and retrieval based on LLM and multiple databases according to claim 3, characterized in that, The specific process of classifying the entity information and writing it to the full-text database and the graph database synchronously in step S2 is as follows: S21: After parsing the key value of the entity attribute, write it into the full-text search library according to the preset field mapping rules to support subsequent text query and content retrieval. The full-text library is a system that supports full-text indexing and searching; S22: according to the different databases to be written, the entity ID generated above is set to the docId of the full-text library or other identifiers used to represent specific records, and the written data includes text fields, keyword fields, tags, and classification information; S23: After parsing the key value of the entity relationship, the node information representing the entity, including the relationship start node and the relationship target node, and the edge information representing the association between the entities are extracted. According to the modeling specification of the graph database, the node attributes, types, and edge relationship types, and the start and end entities are standardized and written into the graph database to complete the construction of the entity graph; S24: According to different databases written, the relationship start node and the relationship target node are set to the node ID of the graph database or other identifiers used to represent specific instances.

6. A method for entity extraction, storage, and retrieval based on LLM and multiple databases according to claim 1, characterized in that, The specific process of step S3 is as follows: S31: Write to the full-text library through the index interface, specify the entity ID and attribute data, the entity ID is used as the unique key of the full-text library document, and the attribute data is strongly bound to the entity through the entity ID to ensure that it can be quickly located through the ID during subsequent retrieval. After writing, verify whether the data is stored correctly through the get interface to ensure that the attribute value is consistent with the input; S32: Parse entity relationship data and query whether from_id and to_id exist in the graph database through node labels and IDs. If the node does not exist, create the node first or throw an exception. S33: Obtain entity node IDs in the graph database. Entity node IDs in the graph database are uniquely identified by labels and attributes. The business ID is mapped to the entity node ID. A relationship is established using the node IDs of the start point and the end point, and the relationship type and attributes are specified.

7. A method for entity extraction, storage, and retrieval based on LLM and multiple databases according to claim 1, characterized in that, The specific process of step S4 is as follows: S41: Setting a search keyword, encapsulating the keyword into a full-text search request, wherein the full-text search request includes search fields, matching methods, and sorting strategy information; S42: Submit the full-text search request to the full-text library. The system matches relevant documents in the specified field based on the inverted index mechanism and returns the matching results. The matching results include entity attributes and record identifiers of the results. S43: Performing association search: Selecting a matching result, obtaining the entity ID of the selected result as the node ID in the graph database, querying the graph database through the specified node ID, locating the target node, and performing adjacency search based on the outgoing and incoming edges of the node to obtain the relationship information associated with the node; S44: After locating the associated node, by obtaining the target associated node ID, the node ID is used as a record identifier of the full-text library to initiate a request to the full-text library to obtain the attribute results of the associated node.

8. A method for entity extraction, storage, and retrieval based on LLM and multiple databases according to claim 7, characterized in that The process of encapsulating keywords into full-text search requests in step S41 is: converting keyword input into a full-text search request structure including field range, matching strategy and sorting configuration, and calling the standardized query interface of the full-text library through an HTTP request.

9. A method for entity extraction, storage, and retrieval based on LLM and multiple databases according to claim 7, characterized in that, Step S4 provides a data retrieval service through a unified retrieval entry. When performing entity attribute retrieval, it supports the use of logical operators, wildcards, and fuzzy search for accurate retrieval, thereby improving the efficiency of data retrieval. For association retrieval, it supports multiple graph algorithms including path discovery and clustering to deeply mine the associations and patterns hidden in the data.

Images