Automatic acquisition and processing method for AGI large model knowledge graph data

Abstract

The invention relates to the technical field of artificial intelligence, and discloses an AGI large model knowledge graph data automatic acquisition and processing method, which comprises a data automatic acquisition step and a data processing step. The data automatic collection step comprises the steps that S1, a data source is determined, the data source is determined through multiple channels, and for AGI knowledge graph construction in the scientific field, IEEE Xplore can provide a large amount of frontier research paper data; s2, web crawler setting: constructing an intelligent web crawler, compiling a crawler program, using a Python Scrapy framework and setting a reasonable crawling frequency according to a webpage structure and a rule of a target data source so as to avoid being forbidden, and meanwhile, setting a proper depth limit; s3, data format standardization: in a collection process, converting data from different sources into a unified format, unifying text codes into UTF-8, and converting collected semi-structured data into a JSON format for subsequent processing; and S4, performing incremental acquisition, periodically re-scanning the data source, and detecting new data.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and specifically to a method for automatically collecting and processing knowledge graph data for an AGI large model. Background Art

[0002] As artificial intelligence develops towards artificial general intelligence (AGI), building a comprehensive and accurate knowledge graph becomes the key. AGI aims to simulate human intelligence and can handle various types of tasks, while knowledge graphs, as a structured form of knowledge representation, provide AGI with a rich knowledge reserve. However, the existing knowledge graph construction has many challenges in data collection and processing.

[0003] Data collection challenges, data source diversity, knowledge exists in multiple different types of data sources, including academic, news, encyclopedia, etc. Traditional data collection methods often target a single or a few data sources and cannot fully cover different fields and types of knowledge. For example, collecting data only from academic databases will miss real-time knowledge in the news and basic knowledge in encyclopedias.

[0004] Data formats are heterogeneous, and the data formats of different data sources vary greatly. Academic papers may be in PDF format, with complex typesetting and specific citation formats; news websites are in HTML format with a large number of style tags; and the data structure of encyclopedias is unique. This format heterogeneity makes data integration difficult and difficult to directly use to build a unified knowledge graph.

[0005] Data processing challenges: data quality varies, and the collected data contains a lot of noise, such as HTML tags, special characters, etc. At the same time, the reliability of different data sources varies, and there are cases of data duplication, inaccuracy, or even errors. For example, news reports may contain some information that has not been strictly verified, which contrasts with the rigorous data in academic research.

[0006] Entity relationship extraction is complex. Accurately identifying entities and extracting relationships from natural language text is one of the core tasks of building a knowledge graph, but the complexity of natural language (such as semantic ambiguity, polysemy, etc.) makes this process challenging. Traditional rule-based methods are difficult to cope with large-scale and diverse text data. Summary of the invention

[0007] 1. Technical issues to be solved

[0008] In view of the shortcomings of the prior art, the present invention provides a method for automatically collecting and processing knowledge graph data for AGI large models, which has the advantages of being able to achieve comprehensive knowledge coverage and solves the problem of uneven data quality.

[0009] (II) Technical solution

[0010] To achieve the above-mentioned purpose, the present invention provides the following technical solutions: a method for automatically collecting and processing AGI large model knowledge graph data, including an automatic data collection step and a data processing step;

[0011] The steps of automatic data collection include, S1: determining the data source, determining the data source through multiple channels. For the construction of AGI knowledge graph in the scientific field, IEEE Xplore can provide a large amount of cutting-edge research paper data;

[0012] S2 web crawler settings, build intelligent web crawlers, write crawler programs according to the web page structure and rules of the target data source, use Python's Scrapy framework, set a reasonable crawling frequency to avoid being banned, and set appropriate depth limits;

[0013] S3 data format is standardized. During the collection process, data from different sources are converted to a unified format, and the text encoding is unified to UTF-8. The collected semi-structured data is converted to JSON format for subsequent processing;

[0014] S4 incremental collection, regularly rescans data sources to detect new data, and only collects new or updated data by recording information such as the timestamp or version number of the last collection. For example, for news websites, newly published articles are checked and collected every day.

[0015] Preferably, the web crawler is set to crawl at a frequency of 1-2 requests per second for common websites and 3-5 requests per second for large data sources with high load capacity.

[0016] Preferably, the crawling depth is limited to: 3-5 layers of link depth if the data source structure is relatively complex and has many related content links.

[0017] Preferably, the incremental collection has a time period of: for data sources with a lower update frequency, checking is performed once a week.

[0018] Preferably, the data processing step includes Y1 data cleaning, removing duplicate data, and identifying and deleting duplicate records by calculating the hash value of the data or comparing key features;

[0019] Process noisy data, such as removing HTML tags, special characters (except necessary punctuation marks), and converting full-width characters to half-width characters;

[0020] Y2 entity recognition and relationship extraction, using natural language processing techniques, such as entity recognition based on pre-trained models;

[0021] At the same time, relationship extraction is performed to determine the relationship between entities. This can be achieved by building a rule engine or a relationship extraction model based on deep learning.

[0022] Y3 knowledge fusion, which integrates information about the same entity or relationship from different data sources;

[0023] Y4 Knowledge verification and correction: verify the knowledge in the knowledge graph through cross-validation (comparing the same information from multiple data sources), find errors or inconsistencies, and make corrections;

[0024] Y5 knowledge graph construction and storage, build a knowledge graph based on the processed data, and use a graph database for storage, with entities as nodes and relationships as edges, and import data into the graph database according to a certain pattern for subsequent query and reasoning operations.

[0025] Preferably, in the data cleaning, duplicate data judgment threshold: when calculating the hash value, a hash algorithm such as MD5 can be used, and when the hash values ​​are exactly the same, they can be determined as duplicate data.

[0026] Preferably, for the entity recognition and relationship extraction, the pre-trained model selects relevant values: when using a pre-trained model such as BERT for fine-tuning, the training batch size can usually be set at 16-64, and the learning rate can be from 1e-5 to 5e-5.

[0027] Preferably, the knowledge fusion integrates credibility weights: when integrating information from different data sources, it is necessary to set credibility weights for the different data sources.

[0028] (III) Beneficial effects

[0029] Compared with the prior art, the present invention provides a method for automatically collecting and processing knowledge graph data for AGI large models, which has the following beneficial effects:

[0030] 1. The method for automatically collecting and processing knowledge graph data for AGI large models. The automatic data collection scheme can achieve comprehensive knowledge coverage. By determining multi-channel data sources and collecting data, it can cover a wide range of knowledge fields, such as obtaining professional knowledge from academic databases, real-time dynamic knowledge from news websites, and basic knowledge from encyclopedias, thereby providing AGI with a comprehensive knowledge base and helping AGI to handle various types of tasks.

[0031] 2. The method for automatic data collection and processing of the knowledge graph of the AGI large model realizes efficient data integration through automatic data collection. The data format standardization and incremental collection mechanism make the collected data easy to integrate. The standardized data format facilitates subsequent data processing and storage, while incremental collection avoids repeated collection of large amounts of data, improves collection efficiency, and saves time and resources.

[0032] 3. The method for automatically collecting and processing knowledge graph data for the AGI large model. The data processing of this solution realizes a high-quality knowledge graph. The duplicate and noisy data are removed through the data cleaning step, which improves the data quality. Entity recognition and relationship extraction can accurately construct the nodes and edges of the knowledge graph. Knowledge fusion and verification and correction further ensure the accuracy and completeness of the knowledge in the knowledge graph, thereby constructing a high-quality knowledge graph and providing a reliable knowledge basis for AGI.

[0033] 4. The method for automatically collecting and processing knowledge graph data for AGI large models enhances the performance of AGI. High-quality knowledge graphs can improve the performance of AGI in tasks such as reasoning and question answering. AGI can acquire knowledge more accurately and make more reasonable inferences, thereby better simulating human intelligence and providing better services in various application scenarios (such as intelligent customer service, intelligent assistants, etc.). DETAILED DESCRIPTION

[0034] The following will be combined with the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0035] The present invention provides a technical solution, specifically, a method for automatically collecting and processing data of an AGI large model knowledge graph, including an automatic data collection step and a data processing step. The automatic data collection step includes: S1: determining a data source, determining the data source through multiple channels. For the construction of an AGI knowledge graph in the scientific field, IEEE Xplore can provide a large amount of cutting-edge research paper data;

[0036] S2 web crawler settings, build intelligent web crawlers, write crawler programs according to the web page structure and rules of the target data source, use Python's Scrapy framework, set a reasonable crawling frequency to avoid being banned, and set appropriate depth limits;

[0037] S3 data format is standardized. During the collection process, data from different sources are converted to a unified format, and the text encoding is unified to UTF-8. The collected semi-structured data is converted to JSON format for subsequent processing;

[0038] S4 incremental collection: regularly rescan the data source to detect new data. By recording the timestamp or version number of the last collection, only new or updated data is collected. For example, for a news website, newly published articles are checked and collected every day.

[0039] Among them, the web crawler settings, crawling frequency: ordinary websites, 1-2 requests per second, large, high-load capacity data sources, can be appropriately increased to 3-5 times per second;

[0040] Crawling depth limit: If the data source structure is complex and has many related content links, 3-5 layers of link depth;

[0041] Among them, incremental collection, time cycle: for data sources with low update frequency, check once a week;

[0042] Further, the data processing step,includes Y1 data cleaning, removing duplicate data, and identifying and deleting duplicate records by calculating the hash value of the data or comparing key features;

[0043] Process noisy data, such as removing HTML tags, special characters (except necessary punctuation marks), and converting full-width characters to half-width characters;

[0044] Y2 entity recognition and relationship extraction, using natural language processing techniques, such as entity recognition based on pre-trained models;

[0045] For example, in the sentence “Einstein proposed the theory of relativity,” “Einstein” and “theory of relativity” are identified as entities;

[0046] At the same time, relationship extraction is performed to determine the relationship between entities. This can be achieved by building a rule engine or a relationship extraction model based on deep learning.

[0047] Y3 knowledge fusion, which integrates information about the same entity or relationship from different data sources;

[0048] Y4 Knowledge verification and correction: verify the knowledge in the knowledge graph through cross-validation (comparing the same information from multiple data sources), find errors or inconsistencies, and make corrections;

[0049] Y5 knowledge graph construction and storage, build a knowledge graph based on the processed data, and use a graph database for storage, with entities as nodes and relationships as edges, and import data into the graph database according to a certain pattern for subsequent query and reasoning operations.

[0050] Among them, data cleaning, duplicate data judgment threshold: when calculating the hash value, you can use the MD5 hash algorithm, when the hash value is exactly the same, it can be judged as duplicate data.

[0051] Among them, entity recognition and relationship extraction, pre-trained model selection related values: When using pre-trained models such as BERT for fine-tuning, the training batch size can usually be set between 16-64, and the learning rate can be from 1e-5 to 5e-5.

[0052] Among them, knowledge fusion, fusion credibility weight: when fusing information from different data sources, it is necessary to set credibility weights for different data sources.

[0053] Furthermore, the automatic data collection of this solution can achieve comprehensive knowledge coverage. By determining multi-channel data sources and collecting data, it can cover a wide range of knowledge fields, such as obtaining professional knowledge from academic databases, real-time dynamic knowledge from news websites, and basic knowledge from encyclopedias, thereby providing AGI with a comprehensive knowledge base and helping AGI handle various types of tasks;

[0054] Furthermore, automatic data collection realizes efficient data integration, data format standardization and incremental collection mechanism make the collected data easy to integrate, standardized data format facilitates subsequent data processing and storage, and incremental collection avoids repeated collection of large amounts of data, improves collection efficiency, and saves time and resources.

[0055] Furthermore, the data processing of this solution realizes a high-quality knowledge graph. The data cleaning step removes duplicate and noisy data, improving the data quality. Entity recognition and relationship extraction can accurately construct nodes and edges of the knowledge graph. Knowledge fusion and verification correction further ensure the accuracy and completeness of the knowledge in the knowledge graph, thereby constructing a high-quality knowledge graph and providing a reliable knowledge basis for AGI.

[0056] This solution enhances the performance of AGI. High-quality knowledge graphs can improve the performance of AGI in tasks such as reasoning and question answering. AGI can acquire knowledge more accurately and make more reasonable inferences, thereby better simulating human intelligence and providing better services in various application scenarios (such as intelligent customer service, intelligent assistants, etc.).

[0057] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A method for automatically collecting and processing knowledge graph data for AGI large models, characterized by: It includes automatic data collection steps and data processing steps; The steps of automatic data collection include, S1: determining the data source, determining the data source through multiple channels. For the construction of AGI knowledge graph in the scientific field, IEEE Xplore can provide a large amount of cutting-edge research paper data; S2 web crawler settings, build intelligent web crawlers, write crawler programs according to the web page structure and rules of the target data source, use Python's Scrapy framework, set a reasonable crawling frequency to avoid being banned, and set appropriate depth limits; S3 data format is standardized. During the collection process, data from different sources are converted to a unified format, and the text encoding is unified to UTF-8. The collected semi-structured data is converted to JSON format for subsequent processing; S4 incremental collection, regularly rescans data sources to detect new data, and only collects new or updated data by recording information such as the timestamp or version number of the last collection. For example, for news websites, newly published articles are checked and collected every day.

2. According to claim 1, a method for automatically collecting and processing AGI large model knowledge graph data is characterized by: The web crawler settings and crawling frequency are as follows: for ordinary websites, 1-2 requests per second; for large, high-load capacity data sources, the frequency can be appropriately increased to 3-5 times per second.

3. The method for automatically collecting and processing AGI large model knowledge graph data according to claim 1 is characterized in that: The crawling depth limit is: if the data source structure is complex and has many related content links, the link depth is 3-5 layers.

4. The method for automatically collecting and processing AGI large model knowledge graph data according to claim 1 is characterized in that: The incremental collection time period is: for data sources with low update frequency, check once a week.

5. The method for automatically collecting and processing AGI large model knowledge graph data according to claim 1 is characterized in that: The data processing step includes Y1 data cleaning, removing duplicate data, and identifying and deleting duplicate records by calculating the hash value of the data or comparing key features; Process noisy data, such as removing HTML tags, special characters (except necessary punctuation marks), and converting full-width characters to half-width characters; Y2 entity recognition and relationship extraction, using natural language processing techniques, such as entity recognition based on pre-trained models; At the same time, relationship extraction is performed to determine the relationship between entities. This can be achieved by building a rule engine or a relationship extraction model based on deep learning. Y3 knowledge fusion, which integrates information about the same entity or relationship from different data sources; Y4 Knowledge verification and correction: verify the knowledge in the knowledge graph through cross-validation (comparing the same information from multiple data sources), find errors or inconsistencies, and make corrections; Y5 knowledge graph construction and storage, build a knowledge graph based on the processed data, and use a graph database for storage, with entities as nodes and relationships as edges, and import data into the graph database according to a certain pattern for subsequent query and reasoning operations.

6. The method for automatically collecting and processing AGI large model knowledge graph data according to claim 1 is characterized in that: The data cleaning and duplicate data judgment threshold: when calculating the hash value, an MD5 hash algorithm may be used, and when the hash values ​​are completely the same, they may be judged as duplicate data.

7. The method for automatically collecting and processing AGI large model knowledge graph data according to claim 1 is characterized in that: For entity recognition and relationship extraction, the pre-trained model selects relevant values: When using a pre-trained model such as BERT for fine-tuning, the training batch size can usually be set to 16-64, and the learning rate can be from 1e-5 to 5e-5.

8. The method for automatically collecting and processing AGI large model knowledge graph data according to claim 1 is characterized in that: The knowledge fusion, fusion credibility weight: when fusing information from different data sources, it is necessary to set credibility weights for different data sources.