Method and system for constructing AI knowledge base based on unstructured text and structured information

Abstract

The invention discloses a method and system for constructing an AI knowledge base based on an unstructured text and structured information. The method comprises the following steps: acquiring original data from an unstructured text data source and a structured data source; preprocessing the unstructured text information and the structured information, and extracting an entity data set of the unstructured text information and the structured information through named entity recognition; the entity data of the unstructured text and the entity data of the structured information are converted into vector representations, data analysis is carried out on the vector representations, the similarity and the correlation index of the vector representations of the unstructured text and the structured information are calculated, and the correlation between the entity data is determined; combining the entity data with the relevance; and constructing a knowledge graph. According to the scheme, intelligent fusion of the unstructured text and the structured information is realized.

Description

Technical Field

[0001] The present invention belongs to the technical field of knowledge bases, and in particular relates to a method and system for building an AI knowledge base based on unstructured text and structured information. Background Art

[0002] With the rapid development of big data and artificial intelligence technologies, enterprises have accumulated a large amount of unstructured text (such as documents, emails, social media content, etc.) and structured information (such as database records, table data, etc.). How to effectively integrate these heterogeneous data and build a high-quality AI knowledge base has become a bottleneck restricting the further development of AI technology. Traditional knowledge base construction methods often focus on a single data source, such as using only unstructured text or structured data. However, unstructured text is rich in semantic information but difficult to structure directly, while structured data has high accuracy but limited information. This single perspective limits the breadth and depth of the knowledge base, makes it difficult to fully utilize the complementary advantages of multi-source data, and ignores the diversity and breadth of data sources. Summary of the invention

[0003] In order to solve the deficiencies in the prior art, the present invention provides a method and system for constructing an AI knowledge base based on unstructured text and structured information, so as to solve the technical problem that the knowledge base construction method focuses on a single data source.

[0004] In order to solve the above technical problems, the present invention adopts the following technical solutions.

[0005] The present invention first discloses a method for constructing an AI knowledge base based on unstructured text and structured information, the method comprising the following steps:

[0006] Step 1: Obtaining raw data from unstructured text data sources and structured data sources to obtain unstructured text information and structured information;

[0007] Step 2: preprocessing the unstructured text information and the structured information, and extracting entity data sets of the unstructured text and entity data sets of the structured information respectively through named entity recognition;

[0008] Step 3: converting the entity data of the unstructured text and the entity data of the structured information into vector representations, performing data analysis on the vector representations, calculating the similarity between the vector representations of the unstructured text and the structured information, calculating the association index between the unstructured text and the structured information based on the similarity, and determining the association between the entity data based on the similarity and the association index;

[0009] Step 4: Combine related entity data to form an entity relationship network;

[0010] Step 5: Construct a knowledge graph based on the entity relationship network.

[0011] The present invention further includes the following preferred embodiments:

[0012] The step of obtaining raw data from the unstructured text data source and the structured data source further includes:

[0013] Data information is obtained from unstructured text data sources and structured data sources through crawler technology, API calls or data interface docking. The unstructured text data sources include web page content, news reports and academic papers, and the structured data sources include databases and API interfaces.

[0014] The preprocessing of the unstructured text information and the structured information further includes:

[0015] The unstructured text information is subjected to part-of-speech tagging, named entities, entity relations, and type recognition and extraction processing, and the structured information is extracted, cleaned, noise data is removed, and spelling errors are corrected. At the same time, data standardization and normalization methods are adopted to unify data formats and dimensions.

[0016] The calculating the similarity between the vector representation of the unstructured text and the structured information further includes:

[0017] Calculate the similarity Sim between unstructured text and structured information vector:

[0018]

[0019] Among them, A q and B q Represent the values ​​of vectors A and B on the qth dimension or feature, respectively. A and B are vectors of unstructured text and structured information, respectively. and They are the square of the modulus of vector A and B, that is, the square of the vector length, and N is the dimension of the vector.

[0020] The calculating the association index between the unstructured text and the structured information based on the similarity further includes:

[0021] Calculate the correlation index μ between the vectors of unstructured text and structured information:

[0022]

[0023] Among them, n is the entity type of unstructured text and structured information, Sim(et i , em j ) represents the entity et in unstructured text iand entity em in structured information j The similarity between them, S represents the similarity between the unstructured text entity data set E t and structured information entity data set E m The subset selected from , k is the conversion coefficient, Sim is the similarity between the unstructured text and the structured information vector, f tp is the number of times the p-th entity appears in the unstructured text, f mp is the number of times the p-th entity appears in the structured information.

[0024] The determining the association between entity data based on the similarity and the association index further includes:

[0025] The joint index δ of the unstructured text and structured information vector is calculated.

[0026]

[0027] If satisfied:

[0028] δ∈[δ e -1, δ e +1]

[0029] It is determined that the entity data is related, where n is the entity type of unstructured text and structured information, |X r | is the relation vector modulus of the r-th entity in the unstructured text, Y r is the modulus of the relation vector of the r-th entity in the structured information, δ e is the preset threshold parameter of the joint index, and e is a mathematical constant.

[0030] The database is a Neo4j graph database, a relational database or a NoSQL database.

[0031] The present invention also discloses a system for constructing an AI knowledge base based on unstructured text and structured information using the above-mentioned method for constructing an AI knowledge base based on unstructured text and structured information, comprising:

[0032] A data source acquisition module is used to acquire raw data from unstructured text data sources and structured data sources to obtain unstructured text information and structured information;

[0033] An entity data extraction module, used to pre-process the unstructured text information and the structured information, and respectively extract an entity data set of the unstructured text and an entity data set of the structured information through named entity recognition;

[0034] A data analysis module, used to convert entity data of unstructured text and entity data of structured information into vector representations, perform data analysis on the vector representations, calculate the similarity between the vector representations of the unstructured text and the structured information, calculate the association index between the unstructured text and the structured information based on the similarity, and determine the association between the entity data based on the similarity and the association index;

[0035] The data union module is used to unite the entity data with association to form an entity relationship network;

[0036] A knowledge graph construction module is used to construct a knowledge graph based on the entity relationship network.

[0037] Accordingly, the present application also discloses a terminal, including a processor and a storage medium;

[0038] The storage medium is used to store instructions;

[0039] The processor is used to operate according to the instructions to execute the steps of the method for building an AI knowledge base based on unstructured text and structured information.

[0040] Correspondingly, the present application also discloses a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the steps of the aforementioned method for building an AI knowledge base based on unstructured text and structured information are implemented.

[0041] The beneficial effect of the present invention is that, compared with the prior art, the present invention provides a method and system for building an AI knowledge base based on unstructured text and structured information, by converting the unstructured text and structured information into vector representations, and inputting the data information represented by the vectors into a processing and analysis device, and using formula algorithms for analysis, a scientific judgment is made on the relevance of entities in the unstructured text and structured information, so that the associated entity data information can be integrated, thereby realizing the intelligent fusion of unstructured text and structured information, making full use of the complementary advantages of multi-source data, and enriching the data content of the knowledge base. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 It is a flowchart of the method for building an AI knowledge base based on unstructured text and structured information in the present invention. DETAILED DESCRIPTION

[0043] In order to make the purpose, technical solution and advantages of the present invention more clear, the technical solution of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention.

[0044] The embodiments described in this application are only some embodiments of the present invention, not all embodiments. Based on the spirit of the present invention, other embodiments obtained by ordinary technicians in this field without creative work are all within the protection scope of the present invention.

[0045] In view of the deficiencies of the prior art, the present invention proposes a method and system for building an AI knowledge base based on unstructured text and structured information, and obtains data information from unstructured text data sources and structured data sources through crawler technology, API calls, and data interface docking. The acquired unstructured text data is subjected to part-of-speech tagging, named entities, entity relationships, and type recognition and extraction processing, and the structured data information is extracted, cleaned, noise data is removed, and spelling errors are corrected. By converting unstructured text and structured information into vector representations, and inputting the data information represented by the vectors into a processing and analysis device, and analyzing it using formula algorithms, the relevance of entities in unstructured text and structured information is scientifically judged, so that the associated entity data information can be integrated, and the intelligent fusion of unstructured text and structured information is realized.

[0046] See also Figure 1 As shown, the method for building an AI knowledge base based on unstructured text and structured information disclosed in the present invention includes the following steps:

[0047] Step 1: Obtain raw data from unstructured text data sources and structured data sources to obtain unstructured text information and structured information.

[0048] First, data information needs to be acquired. Through crawler technology, API calls, and data interface docking, data information can be obtained from unstructured text data sources (such as web page content, news reports, academic papers) and structured data sources (such as databases, API interfaces).

[0049] In a preferred embodiment, during the process of acquiring data information, the reliability and authority of the source of the acquired data information are reviewed, and priority is given to data information from well-known institutions, official channels or peer-reviewed data information.

[0050] Step 2: Preprocess the unstructured text information and structured information, and respectively extract entity data sets of the unstructured text and entity data sets of the structured information through named entity recognition.

[0051] Preprocess the data information, that is, perform part-of-speech tagging, named entity, entity relationship, and type identification and extraction on the unstructured text data obtained in step 1, and extract, clean, remove noise data, and correct spelling errors from the structured information. At the same time, use data standardization and normalization methods to unify the data format and dimension.

[0052] In the process of extracting unstructured text and structured information, desensitization technology is used to desensitize sensitive and private data information. At the same time, the entity data set E in the unstructured text is extracted. t ={et 1 , et 2 ,…,et n}, and entity data collection in structured information

[0053] E m ={em 1 , em 2 ,…,em n}.

[0054] Step 3: Convert the entity data of the unstructured text and the entity data of the structured information into vector representations, perform data analysis on the vector representations, calculate the similarity between the vector representations of the unstructured text and the structured information, calculate the association index between the unstructured text and the structured information based on the similarity, and determine the association between the entity data based on the similarity and the association index.

[0055] Specifically, natural language processing (NLP) and machine learning (ML) technologies are used to convert the text data processed in step 2 into representations in a high-dimensional vector space (such as word embedding, TF-IDF), and the data information represented by the vector is input into a processing and analysis device, and calculation and analysis are performed according to a predefined formula algorithm to determine whether there is a correlation between entity data.

[0056] According to a preferred embodiment, in the process of data analysis, based on the unstructured text vector and the vector data information of the structured information, the similarity Sim between the unstructured text and the structured information vector is obtained by comprehensive calculation:

[0057]

[0058] Among them, A and B are vectors of unstructured text and structured information respectively. q and B q Represent the values ​​of vectors A and B in the qth dimension (or feature), respectively. and are the squares of the modulus of vectors A and B (i.e., the squares of the vector lengths), and N is the dimension of the vector, which is determined by the pre-trained word embedding model.

[0059] For example, if the Word2Vec model is used, the dimension of the vector is preset, such as 300 dimensions, which means that each word is represented as a 300-dimensional vector.

[0060] According to a preferred embodiment, during the data analysis process, the correlation index μ between the unstructured text and the structured information vector is comprehensively calculated based on the entity type, number of occurrences and entity vector data information of the unstructured text vector and the structured information:

[0061]

[0062] Among them, n is the entity type of unstructured text and structured information, Sim(et i , em j ) represents the entity et in unstructured text i and entity em in structured information j The similarity between them, S represents the similarity between the unstructured text entity data set E t and structured information entity data set E m The subset selected from , k is the conversion coefficient, Sim is the similarity between the unstructured text and the structured information vector, f tp is the number of times the p-th entity appears in the unstructured text, f mp is the number of times the p-th entity appears in the structured information.

[0063] According to a preferred embodiment, during the data analysis process, the joint index δ of the unstructured text and structured information vectors is comprehensively calculated based on the similarity, association index, entity type, and entity relationship vector data information of the unstructured text vector and the structured information.

[0064]

[0065] If satisfied:

[0066] δ∈[δ e -1, δ e +1]

[0067] This means that the entity data is associated with each other, and data information can be integrated to combine the entity data information in the unstructured text and structured information. Where n is the entity type of the unstructured text and structured information, |X r | is the relation vector modulus of the r-th entity in the unstructured text, Y r is the modulus of the relation vector of the r-th entity in the structured information, δ e is the preset threshold parameter of the joint index, and e is a mathematical constant.

[0068] Step 4: Combine related entity data to form an entity relationship network.

[0069] The related entity information data are combined, and the entities extracted from unstructured texts are combined with the entities in structured information through ontology matching and entity alignment technology to build a unified entity relationship network.

[0070] Through F(G i , R j , G k ) are combined, where F(x) is the combined function, G i and G k For entity, R j A relationship between two entities.

[0071] Step 5: Construct a knowledge graph based on the entity relationship network.

[0072] Specifically, the combined data is converted into a structured triple form, a knowledge graph containing "entity-relationship-entity" triples is constructed, and the constructed knowledge graph is stored in an efficient database in a structured form. For example, the efficient storage technology of graph databases (such as Neo4j), relational databases or NoSQL databases is used to achieve rapid construction and flexible query of knowledge graphs. At the same time, a variety of management functions are provided, including knowledge update, version control and permission management, to ensure the timeliness and security of the knowledge graph.

[0073] The beneficial effect of the present invention is that, compared with the prior art, the present invention provides a method and system for building an AI knowledge base based on unstructured text and structured information, by converting the unstructured text and structured information into vector representations, and inputting the data information represented by the vectors into a processing and analysis device, and using formula algorithms for analysis, a scientific judgment is made on the relevance of entities in the unstructured text and structured information, so that the associated entity data information can be integrated, thereby realizing the intelligent fusion of unstructured text and structured information, making full use of the complementary advantages of multi-source data, and enriching the data content of the knowledge base.

[0074] The present invention may be a system, a method and / or a computer program product. The present invention also discloses a system for constructing an AI knowledge base based on unstructured text and structured information based on the aforementioned method for constructing an AI knowledge base based on unstructured text and structured information, comprising:

[0075] A data source acquisition module is used to acquire raw data from unstructured text data sources and structured data sources to obtain unstructured text information and structured information;

[0076] An entity data extraction module, used to pre-process the unstructured text information and the structured information, and respectively extract an entity data set of the unstructured text and an entity data set of the structured information through named entity recognition;

[0077] A data analysis module, used to convert entity data of unstructured text and entity data of structured information into vector representations, perform data analysis on the vector representations, calculate the similarity between the vector representations of the unstructured text and the structured information, calculate the association index between the unstructured text and the structured information based on the similarity, and determine the association between the entity data based on the similarity and the association index;

[0078] The data union module is used to unite the entity data with association to form an entity relationship network;

[0079] A knowledge graph construction module is used to construct a knowledge graph based on the entity relationship network.

[0080] Based on the spirit of the present invention, those skilled in the art can easily think that a computer program product can be obtained based on the aforementioned method of building an AI knowledge base based on unstructured text and structured information. The computer program product may include a computer-readable storage medium, which carries computer-readable program instructions for enabling a processor to implement various aspects of the present disclosure. That is, the present application also includes a terminal, including a processor and a storage medium; the storage medium is used to store instructions; the processor is used to operate according to the instructions to execute the steps of the aforementioned method of building an AI knowledge base based on unstructured text and structured information.

[0081] Computer readable storage medium can be a tangible device that can keep and store the instructions used by the instruction execution device. Computer readable storage medium can be, for example, - but not limited to - electrical storage device, magnetic storage device, optical storage device, electromagnetic storage device, semiconductor storage device or any suitable combination of the above. More specific examples (non-exhaustive list) of computer readable storage medium include: portable computer disk, hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disk read-only memory (CD-ROM), digital versatile disk (DVD), memory stick, floppy disk, mechanical encoding device, for example, punch card or groove protrusion structure with instructions stored thereon and any suitable combination of the above. Computer readable storage medium used here is not interpreted as instantaneous signal itself, such as radio wave or other free propagating electromagnetic wave, electromagnetic wave propagated by waveguide or other transmission medium (for example, light pulse by optical fiber cable) or electrical signal transmitted by wire.

[0082] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing / processing device.

[0083] The computer program instructions for performing the operation of the present disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages-such as Smalltalk, C++, etc., and conventional procedural programming languages-such as "C" language or similar programming languages. Computer-readable program instructions may be executed completely on a user's computer, partially on a user's computer, executed as an independent software package, partially on a user's computer, partially on a remote computer, or completely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network-including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., using an Internet service provider to connect via the Internet). In some embodiments, an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA) may be personalized by utilizing the state information of a computer-readable program instruction, and the electronic circuit may execute a computer-readable program instruction, thereby realizing various aspects of the present disclosure.

[0084] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in the relevant field should understand that the specific implementation methods of the present invention can still be modified or replaced by equivalents, and any modifications or equivalent replacements that do not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A method for constructing an AI knowledge base based on unstructured text and structured information, characterized in that: The following steps are involved: Step 1: Obtaining raw data from unstructured text data sources and structured data sources to obtain unstructured text information and structured information; Step 2: preprocessing the unstructured text information and the structured information, and extracting entity data sets of the unstructured text and entity data sets of the structured information respectively through named entity recognition; Step 3: converting the entity data of the unstructured text and the entity data of the structured information into vector representations, performing data analysis on the vector representations, calculating the similarity between the vector representations of the unstructured text and the structured information, calculating the association index between the unstructured text and the structured information based on the similarity, and determining the association between the entity data based on the similarity and the association index; Step 4: Combine related entity data to form an entity relationship network; Step 5: Construct a knowledge graph based on the entity relationship network.

2. The method for constructing an AI knowledge base based on unstructured text and structured information according to claim 1, characterized in that: The step of obtaining raw data from the unstructured text data source and the structured data source further includes: Data information is obtained from unstructured text data sources and structured data sources through crawler technology, API calls or data interface docking. The unstructured text data sources include web page content, news reports and academic papers, and the structured data sources include databases and API interfaces.

3. The method for constructing an AI knowledge base based on unstructured text and structured information according to claim 2, characterized in that: The preprocessing of the unstructured text information and the structured information further includes: The unstructured text information is subjected to part-of-speech tagging, named entities, entity relations, and type recognition and extraction processing, and the structured information is extracted, cleaned, noise data is removed, and spelling errors are corrected. At the same time, data standardization and normalization methods are adopted to unify data formats and dimensions.

4. The method for constructing an AI knowledge base based on unstructured text and structured information according to claim 3, characterized in that: The calculating the similarity between the vector representation of the unstructured text and the structured information further includes: Calculate the similarity Sim between unstructured text and structured information vector: Among them, A q and B q Represent the values ​​of vectors A and B on the qth dimension or feature, respectively. A and B are vectors of unstructured text and structured information, respectively. and They are the square of the modulus of vector A and B, that is, the square of the vector length, and N is the dimension of the vector.

5. The method for constructing an AI knowledge base based on unstructured text and structured information according to claim 4, characterized in that: The calculating the association index between the unstructured text and the structured information based on the similarity further includes: Calculate the correlation index μ between the vectors of unstructured text and structured information: Among them, n is the entity type of unstructured text and structured information, Sim(et i , em j ) represents the entity et in unstructured text i and entity em in structured information j The similarity between them, S represents the similarity between the unstructured text entity data set E t and structured information entity data set E m The subset selected from , k is the conversion coefficient, Sim is the similarity between the unstructured text and the structured information vector, f tp is the number of times the p-th entity appears in the unstructured text, f mp is the number of times the p-th entity appears in the structured information.

6. The method for constructing an AI knowledge base based on unstructured text and structured information according to claim 5, characterized in that: The determining the association between entity data based on the similarity and the association index further includes: The joint index δ of the unstructured text and structured information vector is calculated. If satisfied: δ∈[δ e -1, d e +1] It is determined that the entity data is related, where n is the entity type of unstructured text and structured information, |X r | is the relation vector modulus of the r-th entity in the unstructured text, Y r is the modulus of the relation vector of the r-th entity in the structured information, δ e is the preset threshold parameter of the joint index, and e is a mathematical constant.

7. The method for constructing an AI knowledge base based on unstructured text and structured information according to claim 6, characterized in that: The database is a Neo4j graph database, a relational database or a NoSQL database.

8. A system for building an AI knowledge base based on unstructured text and structured information, characterized in that: include: A data source acquisition module is used to acquire raw data from unstructured text data sources and structured data sources to obtain unstructured text information and structured information; An entity data extraction module, used to pre-process the unstructured text information and the structured information, and respectively extract an entity data set of the unstructured text and an entity data set of the structured information through named entity recognition; A data analysis module, used to convert entity data of unstructured text and entity data of structured information into vector representations, perform data analysis on the vector representations, calculate the similarity between the vector representations of the unstructured text and the structured information, calculate the association index between the unstructured text and the structured information based on the similarity, and determine the association between the entity data based on the similarity and the association index; The data union module is used to unite the entity data with association to form an entity relationship network; A knowledge graph construction module is used to construct a knowledge graph based on the entity relationship network.

9. A terminal comprising a processor and a storage medium; characterized in that: The storage medium is used to store instructions; The processor is used to operate according to the instructions to execute the steps of the method for building an AI knowledge base based on unstructured text and structured information according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method for building an AI knowledge base based on unstructured text and structured information as described in any one of claims 1 to 7 are implemented.

Images