A method and system for identifying a farm based on geospatial big data

Abstract

The application provides a breeding farm identification method and system based on geospatial big data, and relates to the technical fields of animal epidemic disease monitoring and spatial information processing. First, the name, latitude and longitude, province and country, etc. of the global breeding farm data are collected, the fields are merged and the format is unified, and a structured data set is constructed. Combined with global land boundaries, multi-layer administrative divisions and land use data, abnormal coordinates are removed and sampled for review to generate a breeding farm spatial distribution map. The application improves the automation and spatial accuracy of identification by integrating multi-source data and large language models.

Description

Technical Field

[0001] This invention relates to the field of animal disease monitoring and spatial information processing technology, and in particular to a method and system for identifying farms based on geospatial big data. Background Technology

[0002] Existing research mostly focuses on single hosts or local countries or regions, and there is still a lack of systematic assessment of the spatial distribution patterns of interaction interfaces in global agricultural ecosystems and their potential risks.

[0003] Current research on the spatial distribution of global livestock farms still suffers from significant data gaps. Most studies rely solely on national-level estimates or indirect indicators (such as agricultural statistics, land cover data, and agricultural yearbooks), lacking precise location information, which leads to considerable uncertainty in the results at a spatial scale.

[0004] Traditional methods typically rely on open-source geospatial big data, involving manual data downloading, filtering, analysis, and geocoding to obtain the surface features and spatial distribution of a specific country or region. However, this approach is labor-intensive and time-consuming, inefficient, and may lack the analytical capabilities for complex data structures. Furthermore, when preparing global surface feature datasets, which involve data sources from different countries and require extensive textual and language processing, traditional methods demand a massive amount of data analysis and language processing work. Summary of the Invention

[0005] To overcome the shortcomings of existing technologies, the purpose of this invention is to provide a method and system for identifying key locations (farms) based on big data. Compared with traditional location identification methods that rely on a single source and rule matching, this invention achieves automation, semantic enhancement, and spatial accuracy optimization of key location identification by introducing a large language model and fusing multi-source data.

[0006] To achieve the above objectives, the present invention provides the following solution:

[0007] A method for identifying livestock farms based on geospatial big data includes:

[0008] The system utilizes public knowledge bases, map APIs, and open-source geographic information databases to collect raw data from farms worldwide; the raw data includes at least the name, latitude and longitude, province, and country.

[0009] The original data is then processed by merging fields and standardizing formats to generate a structured dataset.

[0010] Data points in the structured dataset are filtered based on global land boundaries, multi-level administrative divisions, and global land use and land cover types. Coordinate points with abnormal spatial locations are removed, and finally, random sampling is performed for verification to form the final spatial distribution map of the farms.

[0011] Preferably, raw data from farms worldwide is collected by utilizing public knowledge bases, map APIs, and open-source geographic information databases, including:

[0012] Construct a SPARQL query template in the public knowledge base Wikidata, limit the entity type to "farm", and retrieve the corresponding links and metadata page by page;

[0013] The obtained links and metadata are crawled using a master-slave distributed collection architecture. The master node of the master-slave distributed collection architecture splits the URL queue and issues sub-tasks. The slave nodes complete the crawling and return the results according to the random delay of 0.5s-2s and the IP rotation strategy.

[0014] The Deepseek large language model is used to perform text completion, language unification and location category filtering on the returned results, and the first original dataset is output.

[0015] The Gaode Map Open Platform Point of Interest Data Retrieval Interface is called. Based on the retrieval interface, each provincial-level administrative region and its subordinate cities, districts and counties are loaded. A tree-structured collection queue is constructed according to the order of province-city-district and county. A Python script is developed to realize automatic pagination collection loop and batch obtain JSON format data of farms in each administrative unit to obtain the second original dataset.

[0016] The Overpass API is used to obtain spatial features containing farms worldwide from the open-source geospatial dataset OpenStreetMap. First, the geographic coordinate range of the seven continents is determined, the bounding boxes of each continent are divided into grids, the grids are traversed in turn, the tag features related to farms are retrieved one by one, and an HTTP request is sent. The returned results are parsed to obtain the third original dataset.

[0017] The first original dataset, the second original dataset, and the third original dataset are merged to construct the original data of the farm.

[0018] Preferably, the Deepseek large language model is used to perform text completion, language unification, and location category filtering on the returned results, outputting the first original dataset, including:

[0019] Based on the returned results, AI prompts are constructed according to the preset field order and calibration instructions, and encoded into a text input sequence that the model can read.

[0020] The AI ​​prompts are input into the Deepseek large language model, which uses its omission detection and garbled character recognition capabilities to output corrected text.

[0021] The Deepseek large language model performs a unified translation of the corrected text into English, and intelligently completes it based on the context, while deleting useless information to obtain the completed text;

[0022] Using the semantic generalization and keyword extraction functions of the Deepseek large language model, core features including main business categories and geographical locations are extracted from the completed text;

[0023] The output format is set by the AI ​​prompt words, and records categorized as "Farm" or "Animal Market" are filtered according to the matching rules of the core features to output the first raw data.

[0024] Preferably, the JSON format data includes latitude and longitude, province and city, and detailed address information; the latitude and longitude are used to generate usable coordinates through extraction and parsing processes.

[0025] Preferably, the original data is subjected to field merging, format unification, and deduplication to generate a structured location dataset, including:

[0026] Import the raw data into the Python data processing environment;

[0027] The field names from different data sources in the original data are mapped and merged to be unified into standard fields of "number, name, address, country, latitude, longitude";

[0028] The latitude and longitude fields are formatted according to a unified coordinate system and preset decimal precision to maintain consistent coordinate representation;

[0029] Records with the same location name and corresponding coordinate differences not exceeding a set threshold are identified as duplicate entries and deleted to obtain unique location records.

[0030] The unique records after deduplication are output in CSV format to generate the structured location dataset.

[0031] Preferably, the data points of the structured dataset are filtered according to global land boundaries, multi-level administrative divisions, and global land use and land cover types, and coordinate points with abnormal spatial locations are removed. Finally, random sampling is performed for verification to form the final spatial distribution map of the aquaculture farms, including:

[0032] The structured dataset is imported into the geographic information system environment and standardized to a preset coordinate system;

[0033] Based on the latitude and longitude range of the structured dataset, spatial intersection analysis is performed using global land boundary vector data to delete points falling outside the land boundaries and generate the first filter set.

[0034] Call the global multi-level administrative division vector boundary, perform spatial intersection analysis on the first filter set, delete records with inconsistent administrative divisions, and generate the second filter set;

[0035] Call the current global land use and land cover raster data with a spatial resolution of 10 meters, perform point-based raster value extraction on the second filter set, assign a land use and land cover type to each point in the second filter set; delete records that fall under land use and land cover types such as open water bodies, ice and snow, and bare land, and generate a third filter set;

[0036] Samples are randomly selected from the third screening set according to a preset ratio, and visual interpretation and verification are performed in conjunction with the GIS visualization interface. Samples with questionable positioning are removed, and the dataset that passes the verification is output.

[0037] The verified dataset is loaded into the GIS platform as point features to generate a spatial distribution map.

[0038] A farm identification system based on geospatial big data includes:

[0039] The raw data acquisition unit is used to call public knowledge bases, map APIs, and open-source geographic information databases to collect raw data from farms worldwide; the raw data includes at least the name, latitude and longitude, province, and country.

[0040] The structured processing unit is used to perform field merging and format unification on the original data to generate a structured dataset;

[0041] The spatial analysis and visualization unit is used to filter data points in the structured dataset based on global land boundaries, multi-level administrative divisions, and global land use and land cover types, remove coordinate points with abnormal spatial locations, and finally perform random sampling verification to form the final spatial distribution map of the aquaculture farm.

[0042] According to specific embodiments provided by the present invention, the present invention discloses the following technical effects:

[0043] This invention integrates geographic information data from multiple heterogeneous sources, including farms and agricultural markets. Through structured processing and large-scale model semantic calibration, it enhances the completeness and accuracy of the data. Combined with coordinate anomaly removal and spatial visualization techniques, it achieves high-precision identification of high-risk locations. Compared to traditional methods relying on manual annotation or single data sources, this invention offers significant advantages in data coverage, automation, and spatial identification accuracy, providing efficient and scalable data support for early warning and regional control of major animal diseases globally. Attached Figure Description

[0044] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0045] Figure 1 A flowchart of the method provided in an embodiment of the present invention;

[0046] Figure 2 This is a schematic diagram of the technical route provided for an embodiment of the present invention. Detailed Implementation

[0047] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0048] The purpose of this invention is to provide a method and system for identifying farms based on geospatial big data. It has significant advantages in terms of data coverage, automation of processing, and spatial identification accuracy, and provides efficient and scalable data support for early warning and regional prevention and control of major animal diseases worldwide.

[0049] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0050] Figure 1 The method flowchart provided in the embodiments of the present invention is as follows: Figure 1 As shown, this invention provides a method for identifying farms based on geospatial big data, including:

[0051] Step 100: Use public knowledge bases, map APIs, and open-source geographic information databases to collect raw data from farms worldwide; the raw data must include at least the name, latitude and longitude, province, and country.

[0052] Step 200: Perform field merging and format unification on the original data to generate a structured dataset;

[0053] Step 300: Filter the data points of the structured dataset according to global land boundaries, multi-level administrative divisions, and global land use / land cover types, remove coordinate points with abnormal spatial locations, and finally perform random sampling verification to form the final spatial distribution map of the aquaculture farm.

[0054] Preferably, step 100 includes:

[0055] Construct a SPARQL query template in the public knowledge base Wikidata, limit the entity type to "farm", and retrieve the corresponding links and metadata page by page;

[0056] A master-slave distributed data collection architecture is used to crawl the obtained links. The master node splits the URL queue and sends out sub-tasks. The slave nodes complete the crawling and send back the results according to a random delay of 0.5s-2s and an IP rotation strategy, thus obtaining the first raw dataset.

[0057] The system utilizes the POI search interface of the Gaode Map Open Platform to first load the administrative divisions of 34 provincial-level administrative regions across China and their subordinate cities, districts, and counties. Then, a tree-structured data collection queue is constructed based on the order of province-city-district / county. A Python script is developed to automatically paginate and loop through the data collection process, batch-collecting JSON-formatted data (including latitude and longitude, province / city, and detailed address) of farms within each administrative unit. The latitude and longitude information is extracted and parsed into usable coordinates. The entire data collection process strictly adheres to API call specifications to ensure data integrity and spatial distribution balance, resulting in a second raw dataset.

[0058] Using the Places API of Google Maps Platform, the world's space was first divided into regular grid cells with 20km as the smallest unit. Then, farms within each grid cell were retrieved one by one, and their names and coordinates were saved to obtain the third original dataset.

[0059] The first original dataset, the second original dataset, and the third original dataset are merged to construct the original data for candidate locations.

[0060] like Figure 2As shown, this embodiment uses the Wikidata Query Service to build a SPARQL query template, filtering wiki data and their corresponding links that meet the "farm" entity type globally. A distributed data collection architecture is adopted, sending requests based on wiki links and using IP rotation strategies and request interval control (0.5-2 second random delay) to circumvent anti-scraping mechanisms. The metadata of each farm (including farm information, location, and corresponding webpage information) is stored locally for subsequent analysis and processing. Based on the Gaode Map Open Platform POI search interface, the 34 provincial-level administrative divisions and their subordinate cities, districts, and counties are first loaded. Then, a tree-structured data collection queue is constructed according to the province-city-district / county order. A Python script is developed to automatically paginate and loop through the data collection, batch acquiring JSON format data (including latitude and longitude, province / city, and detailed address) of farms within each administrative unit. The latitude and longitude information is extracted and parsed into usable coordinates. Using Google Maps Platform's Places API, the global space was first divided into regular grid cells with a minimum unit of 20km. Then, farms within each grid cell were retrieved, and their names and coordinates were saved. Next, based on Python's data structuring process, the semi-structured data was processed and saved as structured information using a Python script. This involved integrating the names, addresses, countries, provinces, latitude and longitude, and detailed addresses of all farms and agricultural markets, and saving them in a local CSV file. Following this, this embodiment focuses on data processing and quality calibration based on a large language model. The obtained structured data was processed using the open-source large language model Deepseek 671B to detect missing text and garbled characters in the main descriptions. The text was then uniformly translated into English, and intelligent completion was performed based on context. Useless information was deleted. Based on this, the main descriptions were summarized to extract core features (such as main business categories and geographical locations). The output format was constrained using the Prompt project, and the types of desired targets were filtered to generate structured data. Finally, this embodiment visualizes the data space. To spatially visualize latitude and longitude data on a map, the local data is first manually validated using spatial logic, i.e., samples with obvious abnormal spatial coordinate distribution are identified. Then, the consistency analysis between spatial coordinates and administrative divisions is used to verify the rationality of the coordinates and filter out outliers. At the same time, random sampling and visual interpretation analysis are performed to improve the overall quality of the data, and finally, a global spatial distribution map of livestock farms is generated.

[0061] In the United States, particularly in the Midwest and East, livestock farms are densely distributed, which is closely related to the highly developed state of American agriculture and the vast amount of arable land. In Europe, the number of livestock farms is concentrated in Western European countries, including France, Germany, the United Kingdom, and the Netherlands, countries with long agricultural histories and advanced agricultural technologies.

[0062] Corresponding to the above method, this embodiment also provides a farm identification system based on geospatial big data, including:

[0063] The raw data acquisition unit is used to call public knowledge bases, map APIs, and open-source geographic information databases to collect raw data from farms worldwide; the raw data includes at least the name, latitude and longitude, province, and country.

[0064] The structured processing unit is used to perform field merging and format unification on the original data to generate a structured dataset;

[0065] The spatial analysis and visualization unit is used to filter data points in the structured dataset based on global land boundaries, multi-level administrative divisions, and global land use and land cover types, removing coordinate points with spatial anomalies, and finally performing random sampling verification to generate the final spatial distribution map of the aquaculture farms.

[0066] The beneficial effects of this invention are as follows:

[0067] (1) Cross-platform integration of international Wikidata and domestic and foreign map services, using a distributed architecture + IP rotation strategy to achieve efficient crawling, and solving the problem of hierarchical collection of administrative regions through a tree-like block request mechanism.

[0068] (2) Using Deepseek’s contextual understanding and multilingual processing capabilities, the web page content is intelligently completed and the main content is extracted. At the same time, the Prompt project is used to achieve accurate conversion of unstructured text into standardized data.

[0069] (3) Construct a dual verification model of coordinate anomaly detection + administrative region matching, establish a global geographic data logical correction paradigm, and achieve a systematic improvement in cross-language and cross-regional data quality.

[0070] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the systems disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the descriptions are relatively simple; relevant parts can be referred to the method section.

[0071] This document uses specific examples to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. Furthermore, those skilled in the art will recognize that, based on the ideas of the present invention, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A method for identifying livestock farms based on geospatial big data, characterized in that, include: It utilizes public knowledge bases, map APIs, and open-source geographic information databases to collect raw data from farms worldwide; The raw data shall include at least the name, latitude and longitude, province, and country. The original data is then processed by merging fields and standardizing formats to generate a structured dataset. Data points in the structured dataset are filtered based on global land boundaries, multi-level administrative divisions, and global land use / land cover types. Coordinate points with abnormal spatial locations are removed. Finally, random sampling is performed for verification to form the final spatial distribution map of the farms. Data points in the structured dataset are filtered based on global land boundaries, multi-level administrative divisions, and global land use and land cover types. Coordinate points with spatial anomalies are removed. Finally, random sampling is performed for verification to generate the final spatial distribution map of the aquaculture farms, including: The structured dataset is imported into the geographic information system environment and standardized to a preset coordinate system; Based on the latitude and longitude range of the structured dataset, spatial intersection analysis is performed using global land boundary vector data to delete points falling outside the land boundaries and generate the first filter set. Call the global multi-level administrative division vector boundary, perform spatial intersection analysis on the first filter set, delete records with inconsistent administrative divisions, and generate the second filter set; Call the current global land use and land cover raster data with a spatial resolution of 10 meters, perform point-based raster value extraction on the second filter set, assign a land use and land cover type to each point in the second filter set; delete the records that fall on open water bodies, ice and snow, and bare land, and generate the third filter set; Samples are randomly selected from the third screening set according to a preset ratio, and visual interpretation and verification are performed in conjunction with the GIS visualization interface. Samples with questionable positioning are removed, and the dataset that passes the verification is output. The verified dataset is loaded into the GIS platform as point features to generate a spatial distribution map.

2. The method for identifying farms based on geospatial big data according to claim 1, characterized in that, By utilizing public knowledge bases, map APIs, and open-source geographic information databases, raw data from farms worldwide is collected, including: Construct a SPARQL query template in the public knowledge base Wikidata, limit the entity type to "farm", and retrieve the corresponding links and metadata page by page; The obtained links and metadata are crawled using a master-slave distributed collection architecture. The master node of the master-slave distributed collection architecture splits the URL queue and issues sub-tasks. The slave nodes complete the crawling and return the results according to the random delay of 0.5s-2s and the IP rotation strategy. The Deepseek large language model is used to perform text completion, language unification and location category filtering on the returned results, and the first original dataset is output. The Gaode Map Open Platform Point of Interest Data Retrieval Interface is called. Based on the retrieval interface, each provincial-level administrative region and its subordinate cities, districts and counties are loaded. A tree-structured collection queue is constructed according to the order of province-city-district and county. A Python script is developed to realize automatic pagination collection loop and batch obtain JSON format data of farms in each administrative unit to obtain the second original dataset. The Overpass API is used to obtain spatial features containing farms worldwide from the open-source geospatial dataset OpenStreetMap. First, the geographic coordinate range of the seven continents is determined, and the bounding boxes of each continent are divided into grids. The grids are traversed in turn, and the tag features related to farms are retrieved one by one. An HTTP request is sent, and the returned results are parsed to obtain the third original dataset. The first original dataset, the second original dataset, and the third original dataset are merged to construct the original data of the farm.

3. The method for identifying farms based on geospatial big data according to claim 2, characterized in that, The Deepseek large language model is used to perform text completion, language unification, and location category filtering on the returned results, outputting the first original dataset, including: Based on the returned results, AI prompts are constructed according to the preset field order and calibration instructions, and encoded into a text input sequence that the model can read. The AI ​​prompts are input into the Deepseek large language model, which uses its omission detection and garbled character recognition capabilities to output corrected text. The Deepseek large language model performs a unified translation of the corrected text into English, and intelligently completes it based on the context, while deleting useless information to obtain the completed text; Using the semantic generalization and keyword extraction functions of the Deepseek large language model, core features including main business categories and geographical locations are extracted from the completed text; The output format is set by the AI ​​prompt words, and records categorized as "Farm" or "Animal Market" are filtered according to the matching rules of the core features to output the first raw data.

4. The method for identifying farms based on geospatial big data according to claim 2, characterized in that, The JSON format data contains latitude and longitude, province and city, and detailed address information; the latitude and longitude are used to generate usable coordinates through extraction and parsing processes.

5. The method for identifying farms based on geospatial big data according to claim 1, characterized in that, The original data is processed by field merging, format unification, and deduplication to generate a structured location dataset, including: Import the raw data into the Python data processing environment; The field names from different data sources in the original data are mapped and merged to be unified into standard fields of "number, name, address, country, latitude, longitude"; The latitude and longitude fields are formatted according to a unified coordinate system and preset decimal precision to maintain consistent coordinate representation; Records with the same location name and corresponding coordinate differences not exceeding a set threshold are identified as duplicate entries and deleted to obtain unique location records. The unique records after deduplication are output in CSV format to generate the structured location dataset.

6. A farm identification system based on geospatial big data, characterized in that, For implementing the farm identification method based on geospatial big data as described in any one of claims 1 to 5, the farm identification system comprises: The raw data acquisition unit is used to call public knowledge bases, map APIs, and open-source geographic information databases to collect raw data from farms worldwide; the raw data includes at least the name, latitude and longitude, province, and country. The structured processing unit is used to perform field merging and format unification on the original data to generate a structured dataset; The spatial analysis and visualization unit is used to filter data points in the structured dataset based on global land boundaries, multi-level administrative divisions, and global land use / land cover types, remove coordinate points with abnormal spatial locations, and finally perform random sampling verification to form the final spatial distribution map of the aquaculture farm.

Images