Multi-source heterogeneous data entity and relation extraction method and device based on large model

Abstract

The invention provides a multi-source heterogeneous data entity and relation extraction method and device based on a large model, and belongs to the technical field of data processing. The method comprises the following steps: automatically identifying and analyzing a plurality of data formats, and standardizing field names; based on the structured data, directly identifying an entity by adopting a regular expression or a rule; based on the unstructured data, splitting the unstructured data into text units through a text segmentation technology, and performing entity extraction and classification by using the pre-training model; capturing local and global relationship characteristics among entities by adopting a graph neural network technology, and tracking and predicting change characteristics of the entities and the relationships thereof along with time by adopting a time sequence graph neural network; the identified entities and the relation between the entities are converted into nodes and edges, and the nodes and the edges are stored in a graph database. The graph neural network technology and the time sequence graph neural network are comprehensively utilized, so that comprehensive capture and dynamic tracking of entities and relationships thereof are realized.

Description

Technical Field

[0001] The present application belongs to the field of data processing technology, and specifically relates to a method and device for extracting multi-source heterogeneous data entities and relationships based on a large model. Background Art

[0002] In today's data-driven era, the processing and analysis of multi-source heterogeneous data has become particularly important. Data from different sources usually exist in multiple formats, such as Excel, Word, PDF, and SQL databases. These data formats not only have different structures, but also contain a large amount of unstructured information.

[0003] Traditional data processing methods often have difficulty coping with this diversity and complexity, especially when it is necessary to extract entities and their relationships from massive data. Manual processing is not only time-consuming and laborious, but also prone to errors. Therefore, developing a method that can automatically identify, parse and standardize multiple data formats and efficiently extract entities and relationships from them has become an urgent problem to be solved. Summary of the invention

[0004] In order to solve at least one technical problem existing in the background technology, the present application provides a method for extracting entities and relationships from multi-source heterogeneous data based on a large model. By comprehensively utilizing graph neural network technology and time-series graph neural network, comprehensive capture and dynamic tracking of entities and their relationships are achieved.

[0005] The technical solution adopted in this application is: The first aspect of the present application provides a method for extracting entities and relationships from multi-source heterogeneous data based on a large model, including: Automatically identify and parse multiple data formats and perform standardization of field names; Based on structured data, regular expressions or rules are used to directly identify entities; based on unstructured data, the unstructured data is split into text units through text segmentation technology, and pre-trained models are used to extract and classify entities; Graph neural network technology is used to capture local and global relationship features between entities, and time-series graph neural network is used to track and predict the changing characteristics of entities and their relationships over time; The identified entities and the relationships between them are converted into nodes and edges and stored in a graph database.

[0006] According to an embodiment of the present application, the automatic identification and parsing of multiple data formats and the standardization of field names are specifically as follows: Automatic identification is achieved by checking file extensions, file header information or preset data structures, wherein the data formats include Excel, Word, PDF or SQL databases; For Excel files, Apache POI library is used for parsing; for Word files, Apache POIXWPF is used for parsing; for PDF files, Apache PDFBox or Tesseract OCR technology is used for parsing; for SQL databases, JDBC is used for parsing.

[0007] According to an embodiment of the present application, based on structured data, regular expressions or rules are used to directly identify entities; based on unstructured data, the unstructured data is split into text units by text segmentation technology, and entities are extracted and classified using a pre-trained model, specifically: Based on the specific application scenarios and data characteristics of structured data, define rules and regular expressions, wherein the rules and regular expressions cover all entity types; Based on unstructured data, natural language processing tools are used to perform text segmentation, and the segmented text units are input into the pre-trained model.

[0008] According to an embodiment of the present application, the graph neural network technology is used to capture the local and global relationship features between entities, and the time-series graph neural network is used to track and predict the change characteristics of entities and their relationships over time, specifically: After identifying entities and extracting relationships, the entities are represented as nodes in a graph, and the relationships between entities are represented as edges; Graph neural network technology updates node representation by aggregating information from neighboring nodes to capture information in the local graph structure; The graph attention mechanism is introduced to assign a weight to each edge and weight the information of neighbor nodes by calculating the attention coefficient.

[0009] According to an embodiment of the present application, the use of graph neural network technology to capture local and global relationship features between entities, and the use of time-series graph neural network to track and predict the changing features of entities and their relationships over time, also includes: Based on data with time series characteristics, the time dimension is introduced into the graph structure to record the creation timestamp or modification timestamp of each node and edge to track the changes of entities and their relationships over time; Aggregate node and edge information within a preset time window to capture short-term dynamic changes; Use recursive neural networks to process time series data and combine them with time-series graph neural networks to perform dynamic modeling of graph structures; The time information is embedded into the feature vectors of nodes and edges so that the model can capture the changing characteristics in the time dimension.

[0010] According to an embodiment of the present application, the identified entities and the relationships therebetween are converted into nodes and edges and stored in a graph database, specifically: The node includes an entity name, a category and a timestamp, and the edge includes a relationship type, a weight and a timestamp; Use the API or query language provided by the graph database to import the identified entities and relationships into the graph database.

[0011] According to an embodiment of the present application, the unstructured data is split into text units by using a text segmentation technology, specifically: Split long text into paragraph units, split each paragraph unit into sentence units, and split each sentence unit into phrases or words.

[0012] The second aspect of the present application provides a device for extracting entities and relationships from multi-source heterogeneous data based on a large model, including: Data recognition module, suitable for automatically identifying and parsing various data formats and standardizing field names; The entity recognition module is suitable for directly identifying entities based on structured data using regular expressions or rules; based on unstructured data, the unstructured data is split into text units using text segmentation technology, and entity extraction and classification are performed using a pre-trained model; The feature acquisition module is suitable for using graph neural network technology to capture local and global relationship features between entities, and using time-series graph neural network to track and predict the changing characteristics of entities and their relationships over time; The storage module is adapted to convert the identified entities and the relationships between them into nodes and edges, and store them in a graph database.

[0013] An embodiment of the third aspect of the present application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the method for extracting multi-source heterogeneous data entities and relationships based on a large model as described in any embodiment of the first aspect described above is implemented.

[0014] The present application also provides a non-volatile computer storage medium having computer executable instructions stored thereon, and when the computer program is executed by a processor, the method for extracting multi-source heterogeneous data entities and relationships based on a large model as described in any embodiment of the first aspect described above is implemented.

[0015] Due to the adoption of the above technical solution, the beneficial effects achieved by this application are as follows: According to the multi-source heterogeneous data entity and relationship extraction method based on a large model provided by the embodiment of the present application, first, the multiple data formats are automatically identified and parsed and the field names are standardized, ensuring that data from different sources and formats can be processed uniformly, greatly improving the efficiency and consistency of data processing. Then, regular expressions or rules are used to directly identify entities based on structured data, and text segmentation technology and pre-trained models are used to extract and classify entities from unstructured data. This method not only covers a wide range of entity types, but also significantly improves the accuracy and comprehensiveness of entity recognition. Then, graph neural network technology is used to capture the local and global relationship features between entities, and combined with the time-series graph neural network to track and predict the changes of these relationships over time, so that the system can dynamically model complex relationship networks and enhance the understanding and prediction capabilities of time-sensitive data. Finally, the identified entities and their relationships are converted into nodes and edges and stored in the graph database, which provides strong support for the subsequent knowledge graph construction and intelligent applications. This flexible and efficient storage method not only optimizes data management, but also provides a solid foundation for complex queries and reasoning. In summary, this method achieves comprehensive, accurate and dynamic processing of multi-source heterogeneous data by integrating multiple advanced technologies, significantly improving the intelligence level and application value of data processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings: Figure 1 A flowchart of a method for extracting entities and relationships from multi-source heterogeneous data based on a large model provided in an embodiment of the present application; Figure 2 A schematic diagram of the structure of a device for extracting entities and relationships from multi-source heterogeneous data based on a large model provided in an embodiment of the present application; Figure 3 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application.

[0017] Reference numerals: 110, data identification module; 120, entity identification module; 130, feature acquisition module; 140, storage module; 810, processor; 820, communication interface; 830, memory; 840, communication bus. DETAILED DESCRIPTION

[0018] In order to more clearly illustrate the overall concept of the present application, a detailed description is given below in an illustrative manner in conjunction with the accompanying drawings.

[0019] In the following description, many specific details are set forth to facilitate a full understanding of the present application. However, the present application may also be implemented in other ways different from those described herein. Therefore, the protection scope of the present application is not limited by the specific embodiments disclosed below. It should be noted that the embodiments of the present application and the features in each embodiment may be combined with each other without conflict.

[0020] In the present application, unless otherwise clearly specified and limited, a first feature "above" or "below" a second feature may be that the first and second features are in direct contact, or the first and second features are in indirect contact through an intermediate medium. In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in an appropriate manner in any one or more embodiments or examples.

[0021] like Figure 1 As shown, the first embodiment of the present application provides a method for extracting entities and relationships from multi-source heterogeneous data based on a large model, comprising: Step 100: Automatically identify and parse multiple data formats and perform standardization of field names.

[0022] Step 200: Based on structured data, regular expressions or rules are used to directly identify entities. Based on unstructured data, text segmentation technology is used to split unstructured data into text units, and a pre-trained model is used to extract and classify entities.

[0023] Step 300: Use graph neural network technology to capture local and global relationship features between entities, and use time-series graph neural network to track and predict the changing characteristics of entities and their relationships over time.

[0024] Step 400: Convert the identified entities and the relationships between them into nodes and edges, and store them in a graph database.

[0025] In step 100, the step first automatically identifies different data formats (such as Excel, Word, PDF, and SQL database) by checking the file extension, file header information, or preset data structure. For each data format, the system uses a corresponding parser for parsing. For example, the Apache POI library is used to parse Excel files, the Apache POI XWPF is used to parse Word files, the Apache PDFBox or Tesseract OCR technology is used to parse PDF files, and the JDBC is used to parse SQL databases. During the parsing process, the system removes noise data and converts all texts into a unified encoding format (such as UTF-8) to ensure data consistency and operability.

[0026] When parsing multiple data formats, the system's automated recognition capabilities significantly reduce the need for manual intervention and improve data processing efficiency. In addition, the standardized processing of field names ensures that data from different sources can be consistently understood and processed, avoiding errors caused by inconsistent naming. This standardization not only improves the quality of the data, but also lays a solid foundation for subsequent entity recognition and relationship modeling. In this way, the system can efficiently process large-scale heterogeneous data, greatly simplifying data management in complex data environments.

[0027] In step 200, for structured data, the system defines a series of rules and regular expressions based on specific application scenarios and data characteristics for direct identification and classification of entities. These rules cover all possible entity types, ensuring high-precision recognition results. For unstructured data, the system first uses natural language processing tools (such as NLTK or spaCy) to split it into smaller text units such as paragraphs, sentences, or phrases, and then inputs these units into pre-trained large models (such as BERT or GPT) to automatically extract and classify entities through contextual understanding capabilities. This approach combines the advantages of rules and deep learning to provide comprehensive and accurate entity recognition.

[0028] By combining regular expressions and pre-trained models, the system is able to achieve efficient entity recognition and classification on different data types. For structured data, the application of rules and regular expressions ensures fast and accurate recognition; while for unstructured data, the application of text segmentation technology and pre-trained models enhances the ability to understand complex texts. This multi-level approach not only improves the accuracy of entity recognition, but also expands the scope of application of the system, enabling it to handle a variety of complex data scenarios. Ultimately, this comprehensive approach provides a high-quality data foundation for subsequent relationship modeling and knowledge graph construction.

[0029] In step 300, after identifying the entities and their relationships, the system represents these entities as nodes in the graph and the relationships between entities as edges. Through the graph neural network (GNN) technology, the system aggregates the information of neighboring nodes to update the node representation, thereby capturing local and global graph structure information. To further enhance the accuracy of relationship modeling, the system introduces a graph attention mechanism (GAT) to assign weights to each edge and weight the information of neighboring nodes by calculating the attention coefficient. For data with time series characteristics, the system also uses a temporal graph neural network (Temporal GNN) to dynamically model and predict the changes of entities and their relationships over time.

[0030] The application of graph neural network technology enables the system to effectively capture complex relationships between entities, and both local and global relationship features can be fully expressed. The introduction of the graph attention mechanism further optimizes the importance assessment of relationships and ensures that key relationships receive more attention. The time-series graph neural network provides strong support for processing time-sensitive data. It can dynamically track and predict changes in entities and their relationships, and is suitable for scenarios that require long-term dependency modeling. This multi-level relationship modeling method not only improves the intelligence level of the system, but also provides reliable technical support for decision support in complex data environments.

[0031] In step 400, after completing entity recognition and relationship extraction, the system converts these entities and relationships into nodes and edges in the graph database. Each node contains attributes such as entity name, category and timestamp, and each edge contains attributes such as relationship type, weight and timestamp. Through the API or query language provided by the graph database (such as Neo4j's Cypher), the system imports the identified entities and relationships into the graph database to facilitate subsequent query and reasoning. This storage method not only optimizes data management, but also provides a solid foundation for complex query and reasoning.

[0032] Storing entities and relationships in a graph database enables the system to efficiently manage and query complex relationship networks. The flexibility and scalability of the graph database allows the system to easily cope with the increasing amount and complexity of data. In addition, by indexing and optimizing queries, the system can significantly improve query efficiency and meet the needs of real-time data analysis. This efficient storage and query mechanism not only improves the level of intelligent data processing, but also provides strong support for the construction of knowledge graphs and intelligent applications, greatly enhancing the practicality and value of the system.

[0033] According to the multi-source heterogeneous data entity and relationship extraction method based on a large model provided by the embodiment of the present application, first, the multiple data formats are automatically identified and parsed and the field names are standardized, ensuring that data from different sources and formats can be processed uniformly, greatly improving the efficiency and consistency of data processing. Then, regular expressions or rules are used to directly identify entities based on structured data, and text segmentation technology and pre-trained models are used to extract and classify entities from unstructured data. This method not only covers a wide range of entity types, but also significantly improves the accuracy and comprehensiveness of entity recognition. Then, graph neural network technology is used to capture the local and global relationship features between entities, and combined with the time-series graph neural network to track and predict the changes of these relationships over time, so that the system can dynamically model complex relationship networks and enhance the understanding and prediction capabilities of time-sensitive data. Finally, the identified entities and their relationships are converted into nodes and edges and stored in the graph database, which provides strong support for the subsequent knowledge graph construction and intelligent applications. This flexible and efficient storage method not only optimizes data management, but also provides a solid foundation for complex queries and reasoning. In summary, this method achieves comprehensive, accurate and dynamic processing of multi-source heterogeneous data by integrating multiple advanced technologies, significantly improving the intelligence level and application value of data processing.

[0034] In some embodiments of the present application, multiple data formats are automatically identified and parsed and field names are standardized, specifically: Automatically identify files by checking file extensions, file header information, or pre-set data structures, where data formats include Excel, Word, PDF, or SQL databases; For Excel files, Apache POI library is used for parsing; for Word files, Apache POIXWPF is used for parsing; for PDF files, Apache PDFBox or Tesseract OCR technology is used for parsing; for SQL databases, JDBC is used for parsing. This step first automatically identifies different data formats (such as Excel, Word, PDF, and SQL database) by checking the file extension, file header information, or preset data structure. Specifically: Excel files: Parsed using the Apache POI library. Apache POI is a powerful Java library that supports reading and writing Microsoft Office document formats, especially Excel files (.xls and .xlsx).

[0035] Word files: Parsed using Apache POI XWPF. XWPF is a submodule of Apache POI that is specifically designed to process XML-based Word documents (.docx).

[0036] PDF files: Parsed using Apache PDFBox or Tesseract OCR technology. Apache PDFBox is an open source Java tool that can read and manipulate PDF documents; while Tesseract OCR is an optical character recognition tool suitable for extracting text content from scanned PDF files.

[0037] SQL database: Parsed using JDBC (Java Database Connectivity). JDBC provides a standard API that allows Java applications to interact with various database management systems and perform query and update operations.

[0038] During the parsing process, the system removes noise data and converts all text into a unified encoding format (such as UTF-8) to ensure data consistency and operability. In addition, the system standardizes field names, such as converting field names to lowercase or unified capitalization rules, using predefined mapping tables to replace common abbreviations or aliases, and removing unnecessary prefixes or suffixes.

[0039] By automatically identifying and parsing multiple data formats, the system can process data from different sources without human intervention, greatly improving the efficiency and consistency of data processing. Each data format has its own unique parsing method, and choosing the right tools and techniques is essential to ensure the accuracy and integrity of data parsing. For example, Apache POI and XWPF can be used to efficiently process Office documents, while Apache PDFBox and Tesseract OCR provide powerful PDF file parsing capabilities. For SQL databases, JDBC provides a flexible and powerful interface that can easily connect and operate various types of databases.

[0040] In some embodiments of the present application, regular expressions or rules are used to directly identify entities based on structured data; based on unstructured data, text segmentation technology is used to split unstructured data into text units, and pre-trained models are used to extract and classify entities, specifically: Define rules and regular expressions based on the specific application scenarios and data characteristics of structured data, where the rules and regular expressions cover all entity types; Based on unstructured data, natural language processing tools are used to perform text segmentation, and the segmented text units are input into the pre-trained model.

[0041] Specifically, when processing structured data, the system defines a series of rules and regular expressions to directly identify and classify entities based on specific application scenarios and data characteristics. These rules and regular expressions are pre-designed based on known data patterns and entity types, and can cover all possible entity types. For example, in financial statements, regular expressions can be defined to match data fields in specific formats such as date formats (such as YYYY-MM-DD) and currency amounts. In this way, the system can efficiently and accurately extract the required entity information from structured data.

[0042] For unstructured data (such as text in Word documents and PDF files), the system first uses natural language processing (NLP) tools to segment the text and split it into smaller text units (such as paragraphs, sentences, or phrases). Common NLP tools include NLTK, spaCy, and Transformers. The segmented text units are input into pre-trained large models (such as BERT, RoBERTa, or GPT), which use the powerful contextual understanding capabilities of these models to automatically extract and classify entities in the text. For example, the BERT model can identify proper nouns in the text (such as names of people, organizations, places, etc.) and classify them into corresponding entity types.

[0043] The method that combines rules and regular expressions based on structured data and text segmentation and pre-trained models based on unstructured data not only improves the accuracy and efficiency of entity recognition and classification, but also enhances the adaptability and flexibility of the system, providing strong support for intelligent applications in complex data environments. This multi-level approach significantly improves the intelligence level and application value of data processing.

[0044] In some embodiments of the present application, graph neural network technology is used to capture local and global relationship features between entities, and a time-series graph neural network is used to track and predict the changing features of entities and their relationships over time, specifically: After identifying entities and extracting relationships, the entities are represented as nodes in a graph, and the relationships between entities are represented as edges; Graph neural network technology updates node representation by aggregating information from neighboring nodes to capture information in the local graph structure; The graph attention mechanism is introduced to assign a weight to each edge and weight the information of neighbor nodes by calculating the attention coefficient.

[0045] Specifically, after completing entity recognition and relationship extraction, the system represents these entities as nodes in the graph, and the relationships between entities as edges. Each node contains the relevant attributes of the entity (such as name, category, timestamp, etc.), and each edge contains information such as the type, weight, and timestamp of the relationship. This graph structure can not only clearly show the relationship between entities, but also provide a basis for subsequent relationship modeling and analysis.

[0046] Graph Neural Networks (GNNs) update node representations by aggregating information from neighboring nodes to capture information in local graph structures. Specifically, the GNN model iteratively obtains information from direct neighbor nodes and fuses it into the representation of the current node. This process can gradually expand the receptive field through multiple layers of GNNs, from the local neighborhood to a wider graph structure, thereby capturing global information. For example, in a social network, GNNs can capture the relationship between a user and his friends, and further expand it to friends of friends, ultimately forming an understanding of the entire social circle.

[0047] To further enhance the accuracy of relationship modeling, the system introduces the Graph Attention Networks (GAT). GAT distinguishes the importance of different relationships by assigning a weight to each edge and weighting the information of neighboring nodes by calculating attention coefficients. Specifically, GAT calculates the attention coefficient of each neighboring node during each message passing process, which determines the degree of influence of the neighboring node information on the current node. This approach enables the model to handle complex relationship networks more flexibly, especially when some relationships are more important than others.

[0048] The use of graph neural network technology and graph attention mechanism not only significantly improves the system's ability to model relationships between entities, but also enhances its application value in complex data environments. In this way, the system can efficiently and accurately capture and understand the relationship characteristics between entities, providing strong support for subsequent knowledge graph construction, intelligent recommendation, risk prediction and other applications. The flexibility and scalability of this method enable it to be widely used in various fields and scenarios, greatly improving the level of intelligent data processing.

[0049] In some embodiments of the present application, graph neural network technology is used to capture local and global relationship features between entities, and a time-series graph neural network is used to track and predict the changing features of entities and their relationships over time, and further includes: Based on data with time series characteristics, the time dimension is introduced into the graph structure to record the creation timestamp or modification timestamp of each node and edge to track the changes of entities and their relationships over time; Aggregate node and edge information within a preset time window to capture short-term dynamic changes; Use recursive neural networks to process time series data and combine them with time-series graph neural networks to perform dynamic modeling of graph structures; The time information is embedded into the feature vectors of nodes and edges so that the model can capture the changing characteristics in the time dimension.

[0050] Specifically, when processing data with time series characteristics, the system introduces a time dimension into the graph structure. Specifically, each node and edge records its creation timestamp or modification timestamp. These timestamps not only help track the changes of entities and their relationships over time, but can also be used for subsequent time series analysis. For example, in social networks, the interaction history between users can be tracked through timestamps; in financial transaction networks, the time changes of transactions can be tracked.

[0051] In order to capture short-term dynamic changes, the system can aggregate node and edge information within a preset time window. This method can help identify short-term relationship changes and patterns. For example, in a news dissemination network, by setting a time window of one day, the changes in information dissemination paths and key nodes within each day can be analyzed. Capturing such short-term dynamic changes is crucial to understanding the real-time status of the system and responding quickly.

[0052] Recurrent Neural Networks (RNN) and their variants (such as LSTM, GRU) are good at processing time series data. The system can use RNN to process time series data and combine it with Temporal Graph Neural Networks (Temporal GNN) to dynamically model the graph structure. Specifically, RNN can capture long-term dependencies in time series, while Temporal GNN can dynamically update the representation of nodes and edges in the graph structure, thereby modeling the changes of the graph structure over time. The combination of the two can better understand and predict the dynamic changes in the graph structure.

[0053] To further enhance the model's understanding of the time dimension, the system embeds time information into the feature vectors of nodes and edges. Specific methods include directly using the timestamp as part of the feature, or encoding the time difference (such as the time interval of the last interaction) as part of the feature vector. This method enables the model to more comprehensively capture the changing characteristics of the time dimension, thereby improving the understanding and prediction capabilities of dynamic graph structures. For example, when predicting future relationship changes, the model can make more accurate judgments based on past time information.

[0054] The use of graph neural network technology and time-series graph neural network methods not only significantly improves the system's ability to model relationships between entities, but also enhances its application value in time series data processing. In this way, the system can efficiently and accurately capture and understand the relationships between entities and their changing characteristics over time, providing strong support for subsequent knowledge graph construction, intelligent recommendations, risk prediction and other applications. The flexibility and scalability of this method enable it to be widely used in various fields and scenarios, greatly improving the level of intelligent data processing.

[0055] In some embodiments of the present application, the identified entities and the relationships between them are converted into nodes and edges and stored in a graph database, specifically: Nodes include entity name, category and timestamp, and edges include relationship type, weight and timestamp; Use the API or query language provided by the graph database to import the identified entities and relationships into the graph database.

[0056] Specifically, when converting the identified entities and their relationships into nodes and edges in the graph database, the system creates a node for each entity and an edge for each relationship between two entities. Each node contains the following key attributes: Entity Name: A unique name or identifier that identifies this entity.

[0057] Category: Describes the category or type to which the entity belongs (such as "person", "organization", "place", etc.).

[0058] Timestamp: records the creation time or last modification time of the entity, used to track changes in the entity over time.

[0059] Each edge has the following key properties: Relationship Type: Describes the type of relationship between two entities (e.g., "works at," "is located at," "friends," etc.).

[0060] Weight: Indicates the importance or strength of the relationship, which can be defined according to the specific application scenario (for example, it can represent the frequency of interaction in a social network).

[0061] Timestamp: records the creation time or last modification time of the relationship, used to track changes in the relationship over time.

[0062] These attributes not only help to fully describe entities and relationships, but also provide a rich information basis for subsequent queries and analysis.

[0063] In order to store the identified entities and relationships in a graph database, the system usually uses the API or query language provided by the graph database. Common graph databases include Neo4j, ArangoDB, and JanusGraph. Taking Neo4j as an example, it provides the Cypher query language to perform CRUD operations (create, read, update, delete). The following are the specific implementation steps: Connecting to a graph database: First, the system needs to establish a connection with the database through the API provided by the graph database. For example, for Neo4j, you can use Neo4j's Java driver or other supported language drivers to connect to the database.

[0064] Create nodes and edges: Use the API or query language provided by the graph database to convert the identified entities and relationships into nodes and edges and store them in the database. For example, in Neo4j, you can use Cypher statements to create nodes and edges: Indexing and optimization: To improve query efficiency, you can create indexes for commonly used attribute fields in the graph database. For example, in Neo4j, you can create an index for the name field of the node: Dynamic update and maintenance: The system needs to regularly check and update the data in the graph database to reflect the latest changes. The automatic update mechanism can be triggered by monitoring changes in the data source (such as file updates, database changes). When new entities or relationships are identified, the system should add them to the graph database; when existing entities or relationships change, the system should update the corresponding node and edge attributes in a timely manner.

[0065] The method of converting the identified entities and their relationships into nodes and edges and storing them in a graph database not only improves the efficiency and consistency of data processing, but also enhances the system's query capabilities and analysis accuracy. The flexibility and scalability of this method enable it to be widely used in various fields and scenarios, greatly improving the level of intelligent data processing. In this way, the system can efficiently and accurately capture and understand the relationships between entities and their changing characteristics over time, providing strong support for subsequent knowledge graph construction, intelligent recommendations, risk prediction and other applications.

[0066] In some embodiments of the present application, unstructured data is split into text units by using text segmentation technology, specifically: Split long text into paragraph units, split each paragraph unit into sentence units, and split each sentence unit into phrases or words.

[0067] Specifically, when processing unstructured data (such as text in Word documents and PDF files), the system first uses natural language processing (NLP) tools to perform multi-level segmentation, gradually splitting long text into smaller text units. This process usually includes the following steps: Paragraph segmentation: Split a long text into paragraph units. Paragraphs are usually identified based on visual delimiters (such as double line breaks) or specific markers (such as HTML tags). For example, in a news article, each paragraph usually discusses a separate topic or point of view.

[0068] Sentence segmentation: Each paragraph unit is further split into sentence units. Sentences are the basic units of text and usually end with punctuation marks such as a period (.), question mark (?), or exclamation mark (!). Accurate sentence segmentation is crucial for subsequent entity recognition and relationship extraction. Commonly used NLP libraries (such as NLTK, spaCy) provide built-in sentence segmenters to accomplish this task.

[0069] Phrase or word segmentation: Each sentence unit is further split into phrases or words. A phrase is a meaningful language unit composed of multiple words (such as "New York City", "artificial intelligence"), while a word is the smallest unit of language. Phrase and word level segmentation can be achieved through techniques such as POS tagging and dependency parsing. For example, spaCy provides a powerful word segmentation function that can split sentences into words or phrases.

[0070] By using text segmentation technology to gradually split unstructured data into paragraph units, sentence units, and phrases or words, it not only improves the system's text understanding ability, but also enhances the accuracy of entity recognition and relationship extraction. The flexibility and scalability of this method enable it to be widely used in various fields and scenarios, greatly improving the level of intelligent data processing. In this way, the system can efficiently and accurately process complex data environments, providing strong support for subsequent knowledge graph construction, intelligent recommendations, risk prediction and other applications.

[0071] like Figure 2 As shown, the second embodiment of the present application provides a multi-source heterogeneous data entity and relationship extraction device based on a large model, including: The data identification module 110 is adapted to automatically identify and parse various data formats and perform standardization of field names; The entity recognition module 120 is suitable for directly identifying entities based on structured data by using regular expressions or rules; based on unstructured data, the unstructured data is split into text units by using text segmentation technology, and entity extraction and classification are performed using a pre-trained model; A feature acquisition module 130 is adapted to use graph neural network technology to capture local and global relationship features between entities, and to use a temporal graph neural network to track and predict the changing features of entities and their relationships over time; The storage module 140 is adapted to convert the identified entities and the relationships between them into nodes and edges, and store them in a graph database.

[0072] The multi-source heterogeneous data entity and relationship extraction device based on a big model provided in the second aspect embodiment of the present application can implement the multi-source heterogeneous data entity and relationship extraction method based on a big model in any embodiment of the above-mentioned first aspect, and thus can achieve any technical effect of the above-mentioned multi-source heterogeneous data entity and relationship extraction method based on a big model, which will not be repeated here.

[0073] The third aspect of the present application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the method for extracting multi-source heterogeneous data entities and relationships based on a large model in any embodiment of the first aspect is implemented.

[0074] Figure 3 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 3 As shown, the electronic device may include: a processor 810, a communication interface 820, a memory 830 and a communication bus 840, wherein the processor 810, the communication interface 820 and the memory 830 communicate with each other through the communication bus 840. The processor 810 may call the logic instructions in the memory 830 to execute the multi-source heterogeneous data entity and relationship extraction method based on a large model in any embodiment of the first aspect, the method comprising: Step 100: Automatically identify and parse multiple data formats and perform standardization of field names.

[0075] Step 200: Based on structured data, regular expressions or rules are used to directly identify entities. Based on unstructured data, text segmentation technology is used to split unstructured data into text units, and a pre-trained model is used to extract and classify entities.

[0076] Step 300: Use graph neural network technology to capture local and global relationship features between entities, and use time-series graph neural network to track and predict the changing characteristics of entities and their relationships over time.

[0077] Step 400: Convert the identified entities and the relationships between them into nodes and edges, and store them in a graph database.

[0078] In addition, the logic instructions in the above-mentioned memory 830 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0079] On the other hand, the present invention further provides a computer program product, the computer program product comprising a computer program, the computer program can be stored in a non-transitory computer-readable storage medium, when the computer program is executed by a processor, the computer can execute the multi-source heterogeneous data entity and relationship extraction method based on a large model provided by the above methods, the method comprising: Step 100: Automatically identify and parse multiple data formats and perform standardization of field names.

[0080] Step 200: Based on structured data, regular expressions or rules are used to directly identify entities. Based on unstructured data, text segmentation technology is used to split unstructured data into text units, and a pre-trained model is used to extract and classify entities.

[0081] Step 300: Use graph neural network technology to capture local and global relationship features between entities, and use time-series graph neural network to track and predict the changing characteristics of entities and their relationships over time.

[0082] Step 400: Convert the identified entities and the relationships between them into nodes and edges, and store them in a graph database.

[0083] In another aspect, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, which is implemented when the computer program is executed by a processor to perform the multi-source heterogeneous data entity and relationship extraction method based on a large model provided by the above methods, the method comprising: Step 100: Automatically identify and parse multiple data formats and perform standardization of field names.

[0084] Step 200: Based on structured data, regular expressions or rules are used to directly identify entities. Based on unstructured data, text segmentation technology is used to split unstructured data into text units, and a pre-trained model is used to extract and classify entities.

[0085] Step 300: Use graph neural network technology to capture local and global relationship features between entities, and use time-series graph neural network to track and predict the changing characteristics of entities and their relationships over time.

[0086] Step 400: Convert the identified entities and the relationships between them into nodes and edges, and store them in a graph database.

[0087] Finally, the present invention also provides a non-volatile computer storage medium having computer executable instructions stored thereon. When the computer program is executed by a processor, the method for extracting entities and relationships from multi-source heterogeneous data based on a large model provided by the above methods is implemented. The method includes: Step 100: Automatically identify and parse multiple data formats and perform standardization of field names.

[0088] Step 200: Based on structured data, regular expressions or rules are used to directly identify entities. Based on unstructured data, text segmentation technology is used to split unstructured data into text units, and a pre-trained model is used to extract and classify entities.

[0089] Step 300: Use graph neural network technology to capture local and global relationship features between entities, and use time-series graph neural network to track and predict the changing characteristics of entities and their relationships over time.

[0090] Step 400: Convert the identified entities and the relationships between them into nodes and edges, and store them in a graph database.

[0091] Anything not described in this application can be achieved by adopting or drawing on existing technologies.

[0092] The various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referenced to each other, and each embodiment focuses on the differences from other embodiments.

[0093] The above is only the embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the protection scope of the present application.

Claims

1. A method for extracting entities and relationships from multi-source heterogeneous data based on a large model, characterized in that: include: Automatically identify and parse multiple data formats and perform standardization of field names; Based on structured data, regular expressions or rules are used to directly identify entities; based on unstructured data, the unstructured data is split into text units through text segmentation technology, and pre-trained models are used to extract and classify entities; Graph neural network technology is used to capture local and global relationship features between entities, and time-series graph neural network is used to track and predict the changing characteristics of entities and their relationships over time; The identified entities and the relationships between them are converted into nodes and edges and stored in a graph database.

2. The method for extracting entities and relationships from multi-source heterogeneous data based on a large model according to claim 1 is characterized in that: The automatic identification and parsing of multiple data formats and the standardization of field names are specifically as follows: Automatic identification is achieved by checking file extensions, file header information or preset data structures, wherein the data formats include Excel, Word, PDF or SQL databases; Based on Excel files, Apache POI library is used for parsing; based on Word files, Apache POI XWPF is used for parsing; based on PDF files, Apache PDFBox or Tesseract OCR technology is used for parsing; based on SQL databases, JDBC is used for parsing.

3. The method for extracting entities and relationships from multi-source heterogeneous data based on a large model according to claim 1 is characterized in that: Based on structured data, regular expressions or rules are used to directly identify entities; based on unstructured data, the unstructured data is split into text units through text segmentation technology, and the pre-trained model is used to extract and classify entities, specifically: Based on the specific application scenarios and data characteristics of structured data, define rules and regular expressions, wherein the rules and regular expressions cover all entity types; Based on unstructured data, natural language processing tools are used to perform text segmentation, and the segmented text units are input into the pre-trained model.

4. The method for extracting entities and relationships from multi-source heterogeneous data based on a large model according to claim 1, characterized in that: The graph neural network technology is used to capture the local and global relationship features between entities, and the time-series graph neural network is used to track and predict the changing characteristics of entities and their relationships over time, specifically: After identifying entities and extracting relationships, the entities are represented as nodes in a graph, and the relationships between entities are represented as edges; Graph neural network technology updates node representation by aggregating information from neighboring nodes to capture information in the local graph structure; The graph attention mechanism is introduced to assign a weight to each edge and weight the information of neighbor nodes by calculating the attention coefficient.

5. The method for extracting entities and relationships from multi-source heterogeneous data based on a large model according to claim 4 is characterized in that: The use of graph neural network technology to capture local and global relationship features between entities and the use of time-series graph neural network to track and predict the changing features of entities and their relationships over time also include: Based on data with time series characteristics, the time dimension is introduced into the graph structure to record the creation timestamp or modification timestamp of each node and edge to track the changes of entities and their relationships over time; Aggregate node and edge information within a preset time window to capture short-term dynamic changes; Use recursive neural networks to process time series data and combine them with time-series graph neural networks to perform dynamic modeling of graph structures; The time information is embedded into the feature vectors of nodes and edges so that the model can capture the changing characteristics in the time dimension.

6. The method for extracting entities and relationships from multi-source heterogeneous data based on a large model according to claim 4 is characterized in that: The identified entities and the relationships between them are converted into nodes and edges and stored in a graph database, specifically: The node includes an entity name, a category and a timestamp, and the edge includes a relationship type, a weight and a timestamp; Use the API or query language provided by the graph database to import the identified entities and relationships into the graph database.

7. The method for extracting entities and relationships from multi-source heterogeneous data based on a large model according to any one of claims 1 to 6, characterized in that: The unstructured data is split into text units by using text segmentation technology, specifically: Split long text into paragraph units, split each paragraph unit into sentence units, and split each sentence unit into phrases or words.

8. A device for extracting entities and relationships from multi-source heterogeneous data based on a large model, characterized in that: include: Data recognition module, suitable for automatically identifying and parsing various data formats and standardizing field names; The entity recognition module is suitable for directly identifying entities based on structured data using regular expressions or rules; based on unstructured data, the unstructured data is split into text units using text segmentation technology, and entity extraction and classification are performed using a pre-trained model; The feature acquisition module is suitable for using graph neural network technology to capture local and global relationship features between entities, and using time-series graph neural network to track and predict the changing characteristics of entities and their relationships over time; The storage module is adapted to convert the identified entities and the relationships between them into nodes and edges, and store them in a graph database.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the method for extracting multi-source heterogeneous data entities and relationships based on a large model as described in any one of claims 1 to 7 is implemented.

10. A non-volatile computer storage medium having computer executable instructions stored thereon, characterized in that: When the computer program is executed by a processor, the method for extracting entities and relationships from multi-source heterogeneous data based on a large model as described in any one of claims 1 to 7 is implemented.

Images