Automatic collection and update method and system of agricultural product information based on knowledge graph
Through the automatic collection and update method based on knowledge graph, combined with deep learning and blockchain technology, the problems of low efficiency and insufficient accuracy of traditional agricultural information collection are solved, and efficient, reliable and flexible collection of agricultural information is achieved. Update to adapt to market changes.
Patent Information
- Application Number
- CN202410726188.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-06-06
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2044-06-06
AI Technical Summary
Traditional agricultural information collection methods rely on manual labor, are inefficient and error-prone, and cannot meet the high requirements of modern agriculture for information accuracy and timeliness. The existing technology similarity algorithm is affected by algorithm limitations and thresholds, and lacks the overall agricultural product market. and supply chain globality and flexibility.
Design an automatic collection and update method of agricultural product information based on knowledge graphs, introduce field ontology and professional knowledge base, combine deep learning models to automatically extract entities, attributes and relationships, use data fusion algorithms and rule engines to discover implicit relationships, and adopt a stream processing framework Carry out incremental updates, and ensure data credibility through blockchain technology, and set up intelligent monitoring and early warning systems.
It improves the accuracy and timeliness of agricultural product information, adapts to rapid market changes and regional specificity, reduces the cumbersomeness and error rate of manual operations, ensures data security and credibility, and optimizes the quality and production efficiency of knowledge graphs.
Smart Images

Figure CN118296164B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of agricultural information technology, and specifically to a method and system for automatically collecting and updating agricultural product information based on a knowledge graph. Background Art
[0002] With the growth of global population and the acceleration of urbanization, agricultural production is facing unprecedented challenges, including improving production efficiency, optimizing resource allocation, ensuring food safety and environmental protection. In this context, intelligence and informatization have become important development directions in the agricultural field. As an important branch of artificial intelligence, knowledge graph technology provides strong support for agricultural informatization with its powerful entity relationship modeling and reasoning capabilities. As an important basis for agricultural production and management, the accuracy and timeliness of agricultural information have a significant impact on agricultural production decisions. However, traditional agricultural information collection methods often rely on manual labor, which is inefficient and error-prone, and cannot meet the high requirements of modern agriculture for information accuracy and timeliness. Therefore, it is particularly important to develop a method that can automatically collect and update agricultural information.
[0003] For example, a Chinese patent with the authorization announcement number CN111582303B discloses a method and system for correcting agricultural product information collection, including: obtaining the current moment information collection table; collecting agricultural product information according to the current moment information collection table collection element, and determining the set of active collection elements; obtaining the current moment collection element and agricultural product information collection ratio set; using the similarity algorithm to determine the similarity of different collection elements for the same agricultural product information collection; determining the collection characteristics of the agricultural product information of the next moment collection element according to the similarity of the same agricultural product information collection by different collection elements; updating the current moment information collection table according to the collection characteristics of the agricultural product information of the next moment collection element, and the collection element collects the agricultural product information of the next moment according to the updated information collection table. The above method of the technical solution solves the tedious work of manually setting which data the collection element needs to collect, thereby improving work efficiency.
[0004] The above patents all have the problems raised by this background technology: 1) The use of similarity algorithms to determine the similarity of different collectors in collecting similar agricultural product information is affected by the limitations of the algorithm and the threshold setting; 2) They mainly focus on the similarity between a single collector and the agricultural product information collection, and lack the overall perspective of the overall agricultural product market and supply chain; 3) They rely on preset collectors and collection ratios, and lack sufficient flexibility to adapt to the rapid changes in the agricultural product market and the specificity of different regions. Summary of the invention
[0005] In view of the shortcomings of the prior art, the present invention proposes an automatic collection and update method and system for agricultural product information based on knowledge graph, designs a knowledge graph architecture, introduces domain ontology and professional knowledge base to enrich semantic information; uses data capture technology to capture agricultural product information in the target data source, combines deep learning models to automatically extract entities, attributes and relationships, and matches them with existing data in the knowledge graph; uses data fusion algorithms to integrate multi-version information, combines rule engines and spatiotemporal data analysis to discover implicit relationships and dynamic changes; sets regular collection tasks, uses a stream processing framework for incremental updates, and uses blockchain technology to ensure data credibility; at the same time, sets a user feedback mechanism and an intelligent monitoring and early warning system to optimize the quality of the knowledge graph and respond to abnormal changes in a timely manner, thereby improving production efficiency and output, ensuring the security and credibility of agricultural product information, and preventing data from being tampered with or forged.
[0006] To achieve the above object, the present invention provides the following technical solutions:
[0007] The method for automatic collection and update of agricultural information based on knowledge graph includes:
[0008] Step S1: Identify the types of agricultural information that need to be collected and updated, design the knowledge graph architecture based on the demand analysis results, plan the top-level structure of the knowledge graph, define the core entities and attributes, and at the same time, introduce domain ontology and agricultural professional knowledge base to enrich the semantic information of the knowledge graph;
[0009] Step S2: determine the target data source for collecting agricultural product information, capture the agricultural product information from the target data source and pre-process it, automatically extract entities, attributes and relationships in the agricultural product information based on the named entity recognition method and combined with the deep learning model, and match the extracted entities with the existing entities and relationships in the knowledge graph;
[0010] Step S3: Select a graph database system, and import the processed entity, attribute, and relationship data into the selected graph database to build a complete agricultural knowledge graph. Combined with the characteristics of the agricultural field, graph embedding technology and index optimization strategies are used to optimize the query and index performance of the knowledge graph.
[0011] Step S4: Integrate agricultural information from different data sources and the same entity through data fusion algorithms, use rule engine reasoning models to discover implicit relationships, causal effects, and trend predictions, and introduce spatiotemporal data analysis methods, using GIS technology and time series analysis to capture the dynamic changes and regional characteristics of agricultural information;
[0012] Step S5: Set up regular collection tasks, use the stream processing framework to incrementally update the knowledge graph based on the newly collected data, set up a change detection mechanism, monitor changes in data sources, trigger an instant update process, use blockchain technology to build a data storage mechanism, and regularly conduct data quality assessments. Set up a user feedback mechanism to optimize the quality of the knowledge graph based on feedback. At the same time, introduce an intelligent monitoring and early warning system to promptly detect and warn of abnormal changes in agricultural information.
[0013] Specifically, the specific steps of step S2 include:
[0014] S201: Design the structure, access frequency, and concurrency control of the data crawling program, set the data crawling target URL and crawling depth, start the data crawling program to crawl agricultural product information after the settings are completed, and pre-process the captured agricultural product information;
[0015] S202: using a named entity recognition method to label the agricultural product information, and using the labeled agricultural product information to train a deep learning model. After the training is completed, the trained deep learning model is used to perform entity recognition on the pre-processed agricultural product information. According to the output of the deep learning model, words labeled as entities are extracted, and the entity category is determined. By analyzing the contextual relationship in the text, the relationship between the entities is identified using a relationship extraction technology, and the entity attributes are extracted.
[0016] S203: Compare the extracted entity with the existing entities in the knowledge graph. If the same entity exists, merge the entity with the entity in the knowledge graph and update the entity attributes and relationships. If the same entity does not exist, add the entity as a new entity to the knowledge graph. At the same time, analyze the extracted entity relationship to determine whether it is consistent with the existing relationship in the knowledge graph. If it is consistent, connect the extracted entity relationship with the corresponding entity. If it is not consistent, adjust it according to business needs.
[0017] S204: Use Talend Data Quality to evaluate the accuracy and completeness of the extraction results, and make adjustments and optimizations based on the evaluation results.
[0018] Specifically, the specific steps of step S3 include:
[0019] S301: using the Neo4j graph database system, collecting the adjusted entity, attribute and relationship data, cleaning and formatting them, writing a data import script, and importing the adjusted entity, attribute and relationship data into the Neo4j graph database by executing the import script;
[0020] S302: Define the types and relationships of entities, attributes and relationships in the Neo4j graph database, and use the graph query language to build the structure and content of the knowledge graph;
[0021] S303: Analyze the structure and characteristics of the knowledge graph, use a graph embedding strategy to embed the knowledge graph, convert nodes and relationships into low-dimensional vector representations, and use a vector similarity query method to optimize query performance;
[0022] S304: Create indexes for attributes or relationships, monitor query performance, use composite indexes to regularly adjust and optimize index strategies, and perform performance tests on the knowledge graph, and make adjustments and optimizations based on the test results.
[0023] Specifically, the specific steps of S4 include:
[0024] S401: Collect agricultural product information of the same entity from different data sources ,in, Represents the nth agricultural information data value in the mth data source, n represents the number of agricultural information of the same entity in the same data source, and m represents the type of data source. A weight is assigned to each data source based on the credibility, accuracy, and timeliness of the data source. , use the weighted average fusion algorithm to fuse the agricultural information data values from different data sources according to their respective weights. For each data point, calculate the weighted average value, and use the weighted average value as the fused agricultural information data value, and output the fused agricultural information data set. The formula is:
[0025]
[0026] in, It represents the nth agricultural information data value of the same entity after fusion in the mth data source. represents the adjustment factor of the ith data source, represents the credibility factor of the i-th data source, Represents the timeliness factor of the i-th data source;
[0027] S402: define logical rules, build a reasoning model using logical reasoning methods, preprocess the integrated agricultural product information data set, configure the defined logical rules into the rule engine, set trigger conditions and execution logic, and input the preprocessed agricultural product information data into the rule engine. The rule engine performs conditional judgment, logical reasoning and data analysis on the input preprocessed agricultural product information data according to the configured rules, and outputs the implicit relationship and causal effect between the preprocessed agricultural product information data. According to actual feedback and new agricultural product information data, optimize the configuration of the logical reasoning model and the rule engine;
[0028] S403: Combine the pre-processed agricultural information data with GIS technology to perform spatial analysis and visualization, and use time series analysis to perform trend prediction;
[0029] S404: Use the goodness of fit evaluation method to evaluate the time series analysis results, and optimize the rule engine reasoning model according to the evaluation results. The formula is:
[0030]
[0031] in, represents the goodness-of-fit index, represents the nth agricultural information data value after fusion in the i-th data source, express The mean of It represents the mean value of the nth agricultural information data in the data source, and a and b represent index parameters.
[0032] Specifically, the specific steps of step S5 include:
[0033] S501: Setting the collection frequency and collection content, and using the data capture method to regularly capture agricultural information data from the data source;
[0034] S502: Select a stream processing framework, set a stream processing task, transmit the newly collected agricultural information data to the stream processing framework in real time, process the stream data according to the update rules and logic of the knowledge graph, and update the knowledge graph;
[0035] S503: Using the agricultural product information data change detection algorithm to detect data change events in real time, triggering the instant update process when data changes are detected, and updating the knowledge graph according to the changed content;
[0036] S504: Use blockchain technology to build a data storage platform, upload records of each agricultural product information data change to the chain, regularly perform agricultural product information data quality assessment tasks, design user feedback channels and feedback collection mechanisms, and optimize the structure and content of the knowledge graph based on user feedback and agricultural product information data quality assessment results, set up configuration monitoring and early warning rules, and conduct real-time monitoring and early warning of abnormal changes in agricultural product information.
[0037] An automatic collection and update system for agricultural information based on knowledge graph, the system comprising: an information collection module, a knowledge graph construction module, a data processing and analysis module, an update and maintenance module, and a user feedback and optimization module;
[0038] The information collection module is responsible for capturing agricultural product information from various data sources and performing preliminary preprocessing; the knowledge graph construction module is used to construct a knowledge graph based on the preprocessed agricultural product information and optimize its query and indexing performance; the data processing and analysis module is used to integrate agricultural product information from multiple sources and perform data fusion and reasoning analysis; the update and maintenance module is used to regularly collect new data, incrementally update the knowledge graph, and monitor data quality and system performance; the user feedback and optimization module is used to collect user feedback and optimize the quality and performance of the knowledge graph based on the feedback.
[0039] Specifically, the information collection module includes: a data source management unit, a data capture unit, a data parsing unit, and a preprocessing unit; the knowledge graph construction module includes: an architecture design unit, an entity and relationship extraction unit, a graph model design unit, a graph database import unit, and a performance optimization unit;
[0040] The data source management unit is used to determine the target data source from which information needs to be collected; the data capture unit is used to capture agricultural product information from the target data source in real time using data capture technology; the data parsing unit is used to convert the captured data in different formats into a unified format; the preprocessing unit is used to clean, standardize, and fuse the captured agricultural product information; the architecture design unit is used to design the overall architecture and top-level structure of the knowledge graph; the graph model design unit is used to define the entity types, attributes, relationship types and constraints of the knowledge graph to form a meta-model of the knowledge graph; the entity and relationship extraction unit is used to automatically extract entities, attributes and relationships in agricultural product information using named entity recognition methods and deep learning models; the graph database import unit is used to import the extracted entity, attribute and relationship data into a graph database to construct a knowledge graph; the performance optimization unit is used to improve the query and indexing efficiency of the knowledge graph through graph embedding technology and index optimization strategies.
[0041] Specifically, the data processing and analysis module includes: a data fusion unit, an inference analysis unit, and a spatiotemporal data analysis unit, and the update and maintenance module includes: a data stream processing unit, an incremental update unit, a change detection and triggering unit, and a data quality assessment unit;
[0042] The knowledge integration unit is used to merge and resolve conflicts of different descriptions of the same entity to form a unified knowledge representation; the reasoning and analysis unit is used to perform logical reasoning, causal analysis, trend prediction, etc. based on known facts and rules to generate new implicit knowledge; the spatiotemporal data analysis unit is used to combine GIS technology and time series analysis to capture the dynamic changes and regional characteristics of agricultural information; the data stream processing unit is used to connect to real-time data streams and update the knowledge graph in real time; the incremental update unit is used to use a stream processing framework to automatically execute data collection, processing, and knowledge update processes at preset time intervals, and perform real-time or periodic incremental updates to the knowledge graph; the change detection and triggering unit is used to monitor changes in data sources and trigger an instant update process once a change is detected; the data quality assessment unit is used to use blockchain technology to build a data storage mechanism and regularly perform quality assessments on data.
[0043] Compared with the prior art, the present invention has the following beneficial effects:
[0044] 1. The present invention proposes an automatic collection and update system for agricultural product information based on knowledge graph, and optimizes and improves the architecture, operation steps and processes. The system has the advantages of simple process, low investment and operation costs, and low production work costs.
[0045] 2. The present invention proposes an automatic collection and update method for agricultural product information based on knowledge graph. By designing the knowledge graph architecture and introducing the deep learning model, the automatic collection, extraction, matching and updating of agricultural product information are realized, which reduces the tediousness and error rate of manual operation and improves work efficiency. At the same time, the rule engine reasoning model is used to automatically discover implicit relationships, causal effects and trend predictions, provide intelligent support, and combine spatiotemporal data analysis methods to capture the dynamic changes and regional characteristics of agricultural product information; agricultural product information from different data sources is integrated through data fusion algorithms to ensure the comprehensiveness and accuracy of the information; blockchain technology is used to build a data notarization mechanism to ensure the security and credibility of the data; regular data quality assessments and user feedback mechanisms are performed to further optimize the quality and accuracy of the knowledge graph.
[0046] 3. The present invention proposes an automatic collection and update method for agricultural product information based on knowledge graph, which is dynamically updated and expanded according to new data and knowledge, adapting to the rapid changes in the market and the specificity of different regions, and introducing an intelligent monitoring and early warning system to timely detect and warn of abnormal changes in agricultural product information, which can cope with market fluctuations and risks and reduce potential losses. BRIEF DESCRIPTION OF THE DRAWINGS
[0047] Figure 1 This is a flow chart of the method for automatically collecting and updating agricultural product information based on knowledge graph of the present invention;
[0048] Figure 2 It is a flowchart of automatic collection and preprocessing of agricultural product information according to the method for automatic collection and update of agricultural product information based on knowledge graph of the present invention;
[0049] Figure 3 This is the architecture diagram of the automatic collection and update system of agricultural product information based on knowledge graph of the present invention. DETAILED DESCRIPTION
[0050] Example 1
[0051] See also Figure 1-Figure 2 , an embodiment provided by the present invention: a method for automatically collecting and updating agricultural product information based on a knowledge graph, comprising the following steps:
[0052] Step S1: Identify the types of agricultural information that need to be collected and updated, design the knowledge graph architecture based on the demand analysis results, plan the top-level structure of the knowledge graph, define the core entities and attributes, and at the same time, introduce domain ontology and agricultural professional knowledge base to enrich the semantic information of the knowledge graph;
[0053] Knowledge graph, also known as scientific knowledge graph, is called knowledge domain visualization or knowledge domain mapping map in the library and information industry. It is a graph-based data structure composed of nodes and edges. Each node represents an "entity" and each edge represents the "relationship" between entities. The core idea of knowledge graph is to transform information in the real world into a graph, where nodes represent entities and edges represent the relationship between entities. This structure enables knowledge graph to describe the knowledge development process and structural relationship, use visualization technology to describe knowledge resources and their carriers, and mine, analyze, construct, draw and display knowledge and their mutual connections.
[0054] Step S2: determine the target data source for collecting agricultural product information, capture the agricultural product information from the target data source and pre-process it, automatically extract entities, attributes and relationships in the agricultural product information based on the named entity recognition method and combined with the deep learning model, and match the extracted entities with the existing entities and relationships in the knowledge graph;
[0055] The target data sources of agricultural product information include government departments, industry associations, e-commerce platforms, and agricultural websites.
[0056] Data crawling technology, also known as web data crawling, web spider or web robot, is an automated web data crawling technology that uses programming languages and tools, such as BeautifulSoup and Scrapy, to simulate human operations on web pages, automatically crawl web page content on the Internet according to rules, including text, pictures, audio, and video, and convert these data into a structured format.
[0057] Step S3: Select a graph database system, and import the processed entity, attribute, and relationship data into the selected graph database to build a complete agricultural knowledge graph. Combined with the characteristics of the agricultural field, graph embedding technology and index optimization strategies are used to optimize the query and index performance of the knowledge graph.
[0058] Step S4: Integrate agricultural information from different data sources and the same entity through data fusion algorithms, use rule engine reasoning models to discover implicit relationships, causal effects, and trend predictions, and introduce spatiotemporal data analysis methods, using GIS technology and time series analysis to capture the dynamic changes and regional characteristics of agricultural information;
[0059] Step S5: Set up regular collection tasks, use the stream processing framework to incrementally update the knowledge graph based on the newly collected data, set up a change detection mechanism, monitor changes in data sources, trigger an instant update process, use blockchain technology to build a data storage mechanism, and regularly conduct data quality assessments. Set up a user feedback mechanism to optimize the quality of the knowledge graph based on feedback. At the same time, introduce an intelligent monitoring and early warning system to promptly detect and warn of abnormal changes in agricultural information.
[0060] Agricultural product information types include: types of agricultural products, yields, origins, planting conditions, and pest and disease conditions.
[0061] The specific steps of step S2 include:
[0062] S201: Design the structure, access frequency, and concurrency control of the data crawling program, set the data crawling target URL and crawling depth, start the data crawling program to crawl agricultural product information after the settings are completed, and pre-process the captured agricultural product information;
[0063] Preprocessing includes: data cleaning, data formatting, and data storage.
[0064] S202: using a named entity recognition method to label the agricultural product information, and using the labeled agricultural product information to train a deep learning model. After the training is completed, the trained deep learning model is used to perform entity recognition on the pre-processed agricultural product information. According to the output of the deep learning model, words labeled as entities are extracted, and the entity category is determined. By analyzing the contextual relationship in the text, the relationship between the entities is identified using a relationship extraction technology, and the entity attributes are extracted.
[0065] The specific steps of labeling agricultural product information using named entity recognition methods include:
[0066] (1) receiving the preprocessed agricultural product information, extracting entity features such as vocabulary features, part-of-speech features, and context features according to the characteristics of the agricultural product information, and converting the text into a feature vector or structured representation;
[0067] (2) Select a named entity recognition model for conditional random fields. If there is labeled data, use this data to train the named entity recognition model for conditional random fields. If there is no labeled data or the data is insufficient, use the transfer learning method to train the named entity recognition model for conditional random fields.
[0068] (3) Optimize the performance of the conditional random field named entity recognition model by adjusting the parameters and architecture of the conditional random field named entity recognition model;
[0069] (4) Use the validation set or test set to evaluate the recognition effect of the conditional random field named entity recognition model, and use performance indicators such as accuracy, recall, and F1 value to measure the annotation accuracy;
[0070] (5) Post-process the annotation results output by the conditional random field named entity recognition model, such as removing duplicate entities, merging adjacent entities of the same type, and outputting the annotation results in a structured form, such as JSON or XML;
[0071] (6) Iterate and optimize the named entity recognition model of conditional random fields based on actual application needs and effect feedback.
[0072] The present invention uses a named entity recognition method to annotate agricultural product information, which can automatically identify and annotate specific entities in the text, such as the name of the agricultural product, place of origin, and production date, and quickly extract information related to the agricultural product from a large amount of text data, thereby improving the efficiency of information collection, and can accurately identify the entity boundaries and types in the text. Compared with manual annotation, automatic annotation can reduce human errors and improve the accuracy of information.
[0073] The specific steps of using relationship extraction technology to identify the relationship between entities include:
[0074] (1) Load the preprocessed agricultural information dataset and the trained conditional random field named entity recognition model;
[0075] (2) Each piece of information in the preprocessed agricultural information dataset is input into the named entity recognition model of the conditional random field. The named entity recognition model of the conditional random field parses the input information according to the learned knowledge, outputs the label of each word, that is, the entity category to which it belongs, identifies the entity boundaries, and gives the entity category to which each identified entity belongs;
[0076] (3) Extract the recognized entities and their categories from the results of named entity recognition. For each entity, further analyze its context information and extract the attributes related to the entity. The attributes can be the specific features, values, and states of the entity. The extracted entities and attributes are saved in a structured form.
[0077] (4) Verify the recognition results of the named entity recognition model of the conditional random field to ensure the accuracy of the entity and the integrity of the attributes. If any recognition errors or omissions are found, manual correction or auxiliary correction can be performed;
[0078] (5) The extracted entities and attributes are used to construct or enrich the agricultural knowledge graph, where the entities are used as nodes in the knowledge graph and the attributes are used as the connection relationships between nodes, thereby constructing a complete agricultural knowledge network.
[0079] S203: Compare the extracted entity with the existing entities in the knowledge graph. If the same entity exists, merge the entity with the entity in the knowledge graph and update the entity attributes and relationships. If the same entity does not exist, add the entity as a new entity to the knowledge graph. At the same time, analyze the extracted entity relationship to determine whether it is consistent with the existing relationship in the knowledge graph. If it is consistent, connect the extracted entity relationship with the corresponding entity. If it is not consistent, adjust it according to business needs.
[0080] S204: Use Talend Data Quality to evaluate the accuracy and completeness of the extraction results, and make adjustments and optimizations based on the evaluation results.
[0081] Specific steps to evaluate the accuracy and completeness of the extraction results include:
[0082] (1) Randomly select a portion of samples from the extraction results, compare the samples with the original data one by one, calculate the proportion of samples that are consistent with the original data, and obtain the accuracy index;
[0083] (2) When the amount of data is small, all extraction results are compared, and the overall accuracy assessment is obtained by calculating the number of completely consistent, partially consistent, or inconsistent extraction results;
[0084] (3) Analyze the accuracy assessment results, identify problems or deficiencies in the extraction process, and adjust and optimize the data extraction methods, rules or tools based on the analysis results, such as repeated evaluation and cross-validation.
[0085] Common data quality assessment tools include: 1) Talend Data Quality; 2) Informatica DataQuality; 3) Data Wrangler; 4) Trifacta; 5) OpenRefine; 6) IBM InfoSphere InformationAnalyzer; 7) Apache NiFi; 8) StreamSets. The present invention uses the Talend Data Quality data quality assessment method to conduct a comprehensive quality assessment of the data, covering multiple dimensions such as data accuracy, completeness, and consistency, ensuring the reliability and validity of the data, and provides a rich data quality rule library and templates. Users can select appropriate rules and templates as needed to quickly conduct data quality assessment and monitoring, thereby improving work efficiency.
[0086] The specific steps of step S3 include:
[0087] S301: using the Neo4j graph database system, collecting the adjusted entity, attribute and relationship data, cleaning and formatting them, writing a data import script, and importing the adjusted entity, attribute and relationship data into the Neo4j graph database by executing the import script;
[0088] Database systems include: Neo4j, OrientDB, JanusGraph, and Dgraph.
[0089] S302: Define the types and relationships of entities, attributes and relationships in the Neo4j graph database, and use the graph query language to build the structure and content of the knowledge graph;
[0090] The specific steps of using graph query language to build the structure and content of knowledge graph include: 1) determine the types of entities, attributes and relationships; 2) create nodes and relationships; 3) define indexes and constraints; 4) query and update knowledge graph; 5) import data; 6) optimize and monitor; 7) visualize. Among them, the above steps all use Neo4j or Cypher language to build knowledge graph.
[0091] S303: Analyze the structure and characteristics of the knowledge graph, use a graph embedding strategy to embed the knowledge graph, convert nodes and relationships into low-dimensional vector representations, and use a vector similarity query method to optimize query performance;
[0092] S304: Create indexes for attributes or relationships, monitor query performance, use composite indexes to regularly adjust and optimize index strategies, and perform performance tests on the knowledge graph, and make adjustments and optimizations based on the test results.
[0093] The specific steps of the index optimization strategy include:
[0094] (1) Identify the frequently queried attributes or relationships in the knowledge graph, analyze the complexity of the query, and select the bitmap indexing method based on the query requirements and database type;
[0095] (2) Create a bitmap index on the selected attributes or relations and set the parameters of the bitmap index, such as the index granularity and update strategy;
[0096] (3) Collect query performance data through query logs or performance monitoring tools, and analyze query response time and throughput indicators;
[0097] (4) Compare query performance data before and after establishing the bitmap index to identify query types with improved or degraded performance;
[0098] (5) Based on the evaluation results, adjust the parameters or type of the bitmap index and use composite indexes to optimize queries involving multiple attributes or relationships;
[0099] (6) Regularly check the status of bitmap and composite indexes, such as fragmentation and expiration, and rebuild or optimize the bitmap and composite indexes.
[0100] The specific steps of S4 include:
[0101] S401: Collect agricultural product information of the same entity from different data sources ,in, Represents the nth agricultural information data value in the mth data source, n represents the number of agricultural information of the same entity in the same data source, and m represents the type of data source. A weight is assigned to each data source based on the credibility, accuracy, and timeliness of the data source. , use the weighted average fusion algorithm to fuse the agricultural information data values from different data sources according to their respective weights. For each data point, calculate the weighted average value, and use the weighted average value as the fused agricultural information data value, and output the fused agricultural information data set. The formula is:
[0102]
[0103] in, It represents the nth agricultural information data value of the same entity after fusion in the mth data source. represents the adjustment factor of the ith data source, represents the credibility factor of the i-th data source, Represents the timeliness factor of the i-th data source;
[0104] S402: define logical rules, build a reasoning model using logical reasoning methods, preprocess the fused agricultural information data set, such as data cleaning, data standardization, and feature extraction, configure the defined logical rules into the rule engine, set trigger conditions and execution logic, and input the preprocessed agricultural information data into the rule engine. The rule engine performs conditional judgment, logical reasoning, and data analysis on the input preprocessed agricultural information data according to the configured rules, and outputs the implicit relationship and causal effect between the preprocessed agricultural information data. According to actual feedback and new agricultural information data, optimize the configuration of the logical reasoning model and the rule engine;
[0105] The specific steps of building a reasoning model using logical reasoning methods include:
[0106] (1) Receive agricultural entity information and concepts, define the relationships between concepts, such as causal relationships, conditional relationships, and inclusion relationships, and formulate logical rules;
[0107] (2) to deduce particular cases from general principles and generalize general laws from individual facts, and to infer the nature of unknown things based on the similarities of known things;
[0108] (3) Organize the defined logical rules into a structured knowledge base, design the reasoning mechanism based on the selected logical reasoning method, including rule matching, condition judgment, and weight calculation, and use programming language to implement the reasoning algorithm so that the model can automatically perform logical reasoning;
[0109] (4) Preprocess the received agricultural entity information and integrate the information from multiple data sources into a unified data set;
[0110] (5) Design the interface between the inference model and the dataset, map the entities and attributes in the dataset to the corresponding concepts in the inference model, and use the inference model to perform logical reasoning on the preprocessed dataset.
[0111] S403: combining the pre-processed agricultural information data with GIS technology to perform spatial analysis and visualization, and using time series analysis to perform trend prediction, wherein the time series analysis method is a prior art content in this field, is not an inventive solution of the present application, and is not described in detail here;
[0112] S404: Use the goodness of fit evaluation method to evaluate the time series analysis results, and optimize the rule engine reasoning model according to the evaluation results. The formula is:
[0113]
[0114] in, represents the goodness-of-fit index, represents the nth agricultural information data value after fusion in the i-th data source, express The mean of It represents the mean value of the nth agricultural information data in the data source, and a and b represent index parameters.
[0115] The specific steps of step S5 include:
[0116] S501: Setting the collection frequency and collection content, and using the data capture method to regularly capture agricultural information data from the data source;
[0117] S502: Select a stream processing framework, set a stream processing task, transmit the newly collected agricultural information data to the stream processing framework in real time, process the stream data according to the update rules and logic of the knowledge graph, and update the knowledge graph;
[0118] S503: Using the agricultural product information data change detection algorithm to detect data change events in real time, triggering the instant update process when data changes are detected, and updating the knowledge graph according to the changed content;
[0119] S504: Use blockchain technology to build a data storage platform, upload records of each agricultural product information data change to the chain, regularly perform agricultural product information data quality assessment tasks, design user feedback channels and feedback collection mechanisms, and optimize the structure and content of the knowledge graph based on user feedback and agricultural product information data quality assessment results, set up configuration monitoring and early warning rules, and conduct real-time monitoring and early warning of abnormal changes in agricultural product information.
[0120] Example 2
[0121] See also Figure 3 Another embodiment provided by the present invention is an automatic collection and update system of agricultural information based on knowledge graph, the system comprising:
[0122] Information collection module, knowledge graph construction module, data processing and analysis module, update and maintenance module, user feedback and optimization module;
[0123] The information collection module is responsible for capturing agricultural product information from various data sources and performing preliminary preprocessing; the knowledge graph construction module is used to construct a knowledge graph based on the preprocessed agricultural product information and optimize its query and indexing performance; the data processing and analysis module is used to integrate agricultural product information from multiple sources and perform data fusion and reasoning analysis; the update and maintenance module is used to regularly collect new data, incrementally update the knowledge graph, and monitor data quality and system performance; the user feedback and optimization module is used to collect user feedback and optimize the quality and performance of the knowledge graph based on the feedback.
[0124] The information collection module includes: a data source management unit, a data capture unit, a data parsing unit, and a preprocessing unit. The knowledge graph construction module includes: an architecture design unit, an entity and relationship extraction unit, a graph model design unit, a graph database import unit, and a performance optimization unit.
[0125] The data source management unit is used to determine the target data source from which information needs to be collected; the data capture unit is used to capture agricultural product information from the target data source in real time using data capture technology; the data parsing unit is used to convert the captured data in different formats into a unified format; the preprocessing unit is used to clean, standardize, and fuse the captured agricultural product information; the architecture design unit is used to design the overall architecture and top-level structure of the knowledge graph; the graph model design unit is used to define the entity types, attributes, relationship types and constraints of the knowledge graph to form a meta-model of the knowledge graph; the entity and relationship extraction unit is used to automatically extract entities, attributes and relationships in agricultural product information using named entity recognition methods and deep learning models; the graph database import unit is used to import the extracted entity, attribute and relationship data into a graph database to construct a knowledge graph; the performance optimization unit is used to improve the query and indexing efficiency of the knowledge graph through graph embedding technology and index optimization strategies.
[0126] The data processing and analysis module includes: a data fusion unit, an inference analysis unit, and a spatiotemporal data analysis unit; the update and maintenance module includes: a data stream processing unit, an incremental update unit, a change detection and triggering unit, and a data quality assessment unit;
[0127] The knowledge integration unit is used to merge and resolve conflicts of different descriptions of the same entity to form a unified knowledge representation; the reasoning and analysis unit is used to perform logical reasoning, causal analysis, trend prediction, etc. based on known facts and rules to generate new implicit knowledge; the spatiotemporal data analysis unit is used to combine GIS technology and time series analysis to capture the dynamic changes and regional characteristics of agricultural information; the data stream processing unit is used to connect to real-time data streams and update the knowledge graph in real time; the incremental update unit is used to use a stream processing framework to automatically execute data collection, processing, and knowledge update processes at preset time intervals, and perform real-time or periodic incremental updates to the knowledge graph; the change detection and triggering unit is used to monitor changes in data sources and trigger an instant update process once a change is detected; the data quality assessment unit is used to use blockchain technology to build a data storage mechanism and regularly perform quality assessments on data.
[0128] The embodiments of the present invention are described above in conjunction with the accompanying drawings, but the present invention is not limited to the above-mentioned specific implementation modes, which are merely illustrative rather than restrictive. Under the guidance of the present invention, ordinary technicians in the field may also change, modify, replace and modify the above-mentioned embodiments without departing from the scope of protection of the purpose of the present invention and the claims, and all of these are within the protection of the present invention.
Claims
1. The method for automatic collection and update of agricultural product information based on knowledge graph is characterized by: include: Step S1: Identify the types of agricultural information that need to be collected and updated, design the knowledge graph architecture based on the demand analysis results, plan the top-level structure of the knowledge graph, define the core entities and attributes, and at the same time, introduce domain ontology and agricultural professional knowledge base to enrich the semantic information of the knowledge graph; Step S2: determine the target data source for collecting agricultural product information, capture the agricultural product information from the target data source and pre-process it, automatically extract entities, attributes and relationships in the agricultural product information based on the named entity recognition method and combined with the deep learning model, and match the extracted entities with the existing entities and relationships in the knowledge graph; Step S3: Select a graph database system, and import the processed entity, attribute, and relationship data into the selected graph database to build a complete agricultural knowledge graph. Combined with the characteristics of the agricultural field, graph embedding technology and index optimization strategies are used to optimize the query and index performance of the knowledge graph. Step S4: Integrate agricultural information from different data sources and the same entity through data fusion algorithms, use rule engine reasoning models to discover implicit relationships, causal effects, and trend predictions, and introduce spatiotemporal data analysis methods, using GIS technology and time series analysis to capture the dynamic changes and regional characteristics of agricultural information; Step S5: Set up regular collection tasks, use the stream processing framework to incrementally update the knowledge graph based on the newly collected data, set up a change detection mechanism, monitor changes in data sources, trigger an instant update process, use blockchain technology to build a data storage mechanism, and regularly conduct data quality assessments, set up a user feedback mechanism, and optimize the quality of the knowledge graph based on feedback. At the same time, introduce an intelligent monitoring and early warning system to promptly detect and warn of abnormal changes in agricultural information; The specific steps of S4 include: S401: Collect agricultural product information of the same entity from different data sources ,in, Represents the nth agricultural information data value in the mth data source, n represents the number of agricultural information of the same entity in the same data source, and m represents the type of data source. A weight is assigned to each data source based on the credibility, accuracy, and timeliness of the data source. , use the weighted average fusion algorithm to fuse the agricultural information data values from different data sources according to their respective weights. For each data point, calculate the weighted average value, and use the weighted average value as the fused agricultural information data value, and output the fused agricultural information data set. The formula is: ; in, It represents the nth agricultural information data value of the same entity after fusion in the mth data source. represents the adjustment factor of the ith data source, represents the credibility factor of the i-th data source, Represents the timeliness factor of the i-th data source; S402: define logical rules, build a reasoning model using logical reasoning methods, preprocess the integrated agricultural product information data set, configure the defined logical rules into the rule engine, set trigger conditions and execution logic, and input the preprocessed agricultural product information data into the rule engine. The rule engine performs conditional judgment, logical reasoning and data analysis on the input preprocessed agricultural product information data according to the configured rules, and outputs the implicit relationship and causal effect between the preprocessed agricultural product information data. According to actual feedback and new agricultural product information data, optimize the configuration of the logical reasoning model and the rule engine; S403: Combine the pre-processed agricultural information data with GIS technology to perform spatial analysis and visualization, and use time series analysis to perform trend prediction; S404: Use the goodness of fit evaluation method to evaluate the time series analysis results, and optimize the rule engine reasoning model according to the evaluation results. The formula is: ; in, represents the goodness-of-fit index, represents the nth agricultural information data value after fusion in the i-th data source, express The mean of It represents the mean value of the nth agricultural information data in the data source, and a and b represent index parameters.
2. The method for automatically collecting and updating agricultural product information based on knowledge graph according to claim 1, characterized in that: The specific steps of step S2 include: S201: Design the structure, access frequency, and concurrency control of the data crawling program, set the data crawling target URL and crawling depth, start the data crawling program to crawl agricultural product information after the settings are completed, and pre-process the captured agricultural product information; S202: using a named entity recognition method to label the agricultural product information, and using the labeled agricultural product information to train a deep learning model. After the training is completed, the trained deep learning model is used to perform entity recognition on the pre-processed agricultural product information. According to the output of the deep learning model, words labeled as entities are extracted, and the entity category is determined. By analyzing the contextual relationship in the text, the relationship between the entities is identified using a relationship extraction technology, and the entity attributes are extracted. S203: Compare the extracted entity with the existing entities in the knowledge graph. If the same entity exists, merge the entity with the entity in the knowledge graph and update the entity attributes and relationships. If the same entity does not exist, add the entity as a new entity to the knowledge graph. At the same time, analyze the extracted entity relationship to determine whether it is consistent with the existing relationship in the knowledge graph. If it is consistent, connect the extracted entity relationship with the corresponding entity. If it is not consistent, adjust it according to business needs. S204: Use Talend Data Quality to evaluate the accuracy and completeness of the extraction results, and make adjustments and optimizations based on the evaluation results.
3. The method for automatically collecting and updating agricultural product information based on knowledge graph according to claim 2 is characterized in that: The specific steps of step S3 include: S301: using the Neo4j graph database system, collecting the adjusted entity, attribute and relationship data, cleaning and formatting them, writing a data import script, and importing the adjusted entity, attribute and relationship data into the Neo4j graph database by executing the import script; S302: Define the types and relationships of entities, attributes and relationships in the Neo4j graph database, and use the graph query language to build the structure and content of the knowledge graph; S303: Analyze the structure and characteristics of the knowledge graph, use a graph embedding strategy to embed the knowledge graph, convert nodes and relationships into low-dimensional vector representations, and use a vector similarity query method to optimize query performance; S304: Create indexes for attributes or relationships, monitor query performance, use composite indexes to regularly adjust and optimize index strategies, and perform performance tests on the knowledge graph, and make adjustments and optimizations based on the test results.
4. The method for automatically collecting and updating agricultural product information based on knowledge graph according to claim 3 is characterized in that: The specific steps of step S5 include: S501: Setting the collection frequency and collection content, and using the data capture method to regularly capture agricultural information data from the data source; S502: Select a stream processing framework, set a stream processing task, transmit the newly collected agricultural information data to the stream processing framework in real time, process the stream data according to the update rules and logic of the knowledge graph, and update the knowledge graph; S503: Using the agricultural product information data change detection algorithm to detect data change events in real time, triggering the instant update process when data changes are detected, and updating the knowledge graph according to the changed content; S504: Use blockchain technology to build a data storage platform, upload records of each agricultural product information data change to the chain, regularly perform agricultural product information data quality assessment tasks, design user feedback channels and feedback collection mechanisms, and optimize the structure and content of the knowledge graph based on user feedback and agricultural product information data quality assessment results, set up configuration monitoring and early warning rules, and conduct real-time monitoring and early warning of abnormal changes in agricultural product information.
5. A system for automatically collecting and updating agricultural product information based on a knowledge graph, which is implemented based on the method for automatically collecting and updating agricultural product information based on a knowledge graph as described in any one of claims 1 to 4, characterized in that: The system includes: an information collection module, a knowledge graph construction module, a data processing and analysis module, an update and maintenance module, and a user feedback and optimization module; The information collection module is responsible for capturing agricultural product information from various data sources and performing preliminary preprocessing; the knowledge graph construction module is used to construct a knowledge graph based on the preprocessed agricultural product information and optimize its query and indexing performance; the data processing and analysis module is used to integrate agricultural product information from multiple sources and perform data fusion and reasoning analysis; the update and maintenance module is used to regularly collect new data, incrementally update the knowledge graph, and monitor data quality and system performance; the user feedback and optimization module is used to collect user feedback and optimize the quality and performance of the knowledge graph based on the feedback.
6. The automatic collection and update system of agricultural information based on knowledge graph according to claim 5 is characterized in that: The information collection module includes: a data source management unit, a data capture unit, a data parsing unit, and a preprocessing unit. The knowledge graph construction module includes: an architecture design unit, an entity and relationship extraction unit, a graph model design unit, a graph database import unit, and a performance optimization unit. The data source management unit is used to determine the target data source from which information needs to be collected; the data capture unit is used to capture agricultural product information from the target data source in real time using data capture technology; the data parsing unit is used to convert the captured data in different formats into a unified format; the preprocessing unit is used to clean, standardize, and fuse the captured agricultural product information; the architecture design unit is used to design the overall architecture and top-level structure of the knowledge graph; the graph model design unit is used to define the entity types, attributes, relationship types and constraints of the knowledge graph to form a meta-model of the knowledge graph; the entity and relationship extraction unit is used to automatically extract entities, attributes and relationships in agricultural product information using named entity recognition methods and deep learning models; the graph database import unit is used to import the extracted entity, attribute and relationship data into a graph database to construct a knowledge graph; the performance optimization unit is used to improve the query and indexing efficiency of the knowledge graph through graph embedding technology and index optimization strategies.
7. The automatic agricultural product information collection and updating system based on knowledge graph according to claim 6 is characterized in that: The data processing and analysis module includes: a data fusion unit, an inference analysis unit, and a spatiotemporal data analysis unit; the update and maintenance module includes: a data stream processing unit, an incremental update unit, a change detection and triggering unit, and a data quality assessment unit; The data fusion unit is used to merge and resolve conflicts of different descriptions of the same entity to form a unified knowledge representation; the reasoning and analysis unit is used to perform logical reasoning, causal analysis, trend prediction, etc. based on known facts and rules to generate new implicit knowledge; the spatiotemporal data analysis unit is used to combine GIS technology and time series analysis to capture the dynamic changes and regional characteristics of agricultural information; the data stream processing unit is used to connect to real-time data streams and update the knowledge graph in real time; the incremental update unit is used to use a stream processing framework to automatically execute data collection, processing, and knowledge update processes at preset time intervals, and perform real-time or periodic incremental updates to the knowledge graph; the change detection and triggering unit is used to monitor changes in data sources and trigger an instant update process once a change is detected; the data quality assessment unit is used to use blockchain technology to build a data storage mechanism and regularly perform quality assessments on data.
