Method and system for identifying avian influenza virus overflowing farm based on geographic space big data

Abstract

The invention provides an avian influenza virus overflow farm identification method and system based on geographic space big data, and relates to the technical field of animal epidemic disease monitoring and spatial information processing. The method comprises the following steps of: firstly, acquiring data such as names, longitudes and latitudes, affiliated provinces and countries of global farms, performing field merging and format unification, and constructing a structured data set; and in combination with global land boundaries, multi-layer administrative division and land utilization data, abnormal coordinates are eliminated, sampling and re-checking are carried out, and a farm spatial distribution map is generated. According to the method, the multi-source data and the large language model are fused, so that the recognition automation and the space precision are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of animal epidemic monitoring and spatial information processing, in particular to an avian influenza virus overflow farm identification method and system based on geographic spatial big data. BACKGROUND

[0002] Existing researches mostly focus on a single host or a local country or region, and there is still a lack of systematic assessment of the spatial distribution and potential risks of the interaction interface in the global agricultural ecosystem.

[0003] Current research on the spatial distribution of global farms still has a significant data gap. Most researches only rely on national estimates or indirect indicators (such as agricultural statistics, land cover data, and agricultural yearbooks) to deduce, lacking precise point information, resulting in a large uncertainty in the spatial scale of the results.

[0004] Traditional methods usually rely on open-source geographic spatial big data to obtain surface elements and spatial distribution in a country or a single region through manual data download, screening, analysis, and geographic coding. However, the cost of manpower and time is high, the efficiency is low, and there may be a problem of insufficient analysis capability for complex structured data. In addition, when preparing a global surface element data set, different data sources from different countries are involved, and more text language processing is involved. The traditional method requires a large amount of data analysis and language processing. SUMMARY

[0005] In order to overcome the shortcomings of the prior art, the purpose of the present application is to provide an avian influenza virus overflow key place (farm) identification method and system based on big data. Compared with the traditional place identification method which relies on a single source and rule matching, the present application introduces a large language model and multi-source data fusion to realize the automation, semantic enhancement, and spatial precision optimization of avian influenza virus overflow key place identification.

[0006] To achieve the above purpose, the present application provides the following scheme:

[0007] An avian influenza virus overflow farm identification method based on geographic spatial big data, comprising:

[0008] Call public knowledge base, map API and open-source geographic information database to collect original data of farms in a global range; the original data at least includes name, longitude and latitude, belonging to province and belonging to country;

[0009] Perform field merging and format unification on the original data to generate a structured data set;

[0010] According to the global land boundary, multi-level administrative division, and global land use and land cover type, the data points of the structured data set are screened, the coordinate points with abnormal spatial positions are removed, and finally random sampling review is performed to form the final spatial distribution map of the farms.

[0011] Preferably, a public knowledge base, a map API, and an open source geographic information database are called to collect original data of the farms worldwide, including:

[0012] A SPARQL query template is constructed in the public knowledge base Wikidata, the entity type is limited to "farms", and the corresponding links and metadata are obtained by page;

[0013] The obtained links and metadata are executed by a master-slave distributed collection architecture, the master node of the master-slave distributed collection architecture splits the URL queue and issues sub-tasks, and the slave node completes the collection and returns the results according to the random delay of 0.5s-2s and the IP rotation strategy;

[0014] The returned results are text-completed, language-unified, and place-category-filtered by a Deepseek large language model to output a first original data set;

[0015] The POI data retrieval interface of the Gaode map open platform is called, each provincial administrative division and the subordinate cities, districts, and counties are loaded based on the retrieval interface, and a tree-shaped collection queue is constructed according to the order of province-city-district, and a Python script is developed to realize automatic paging collection cycle, and JSON format data of the farms in each administrative unit is batch acquired to obtain a second original data set;

[0016] The OverpassAPI is called to obtain spatial elements containing farms (farms) from the open source geographic spatial data set OpenStreetMap, the geographic coordinate ranges of the seven continents are determined, the boundary boxes of each continent are divided into grids, the grids are traversed in turn, the label elements related to the farms are retrieved one by one, HTTP requests are sent, the returned results are parsed, and a third original data set is obtained;

[0017] The first original data set, the second original data set, and the third original data set are merged to construct the original data of the farms.

[0018] Preferably, the returned results are text-completed, language-unified, and place-category-filtered by a Deepseek large language model to output a first original data set, including:

[0019] For the returned results, AI prompt words are constructed according to the preset field order and calibration instructions, and are encoded into text input sequences readable by the model;

[0020] inputting the AI prompt word into a Deepseek large language model, outputting a revised text using the missing detection and garbled code recognition capabilities;

[0021] performing unified translation of the revised text into English by the Deepseek large language model, intelligently completing based on context, and deleting useless information to obtain a completed text;

[0022] extracting core features including main business categories and geographic locations from the completed text using the semantic summarization and keyword extraction functions of the Deepseek large language model;

[0023] setting the output format through the AI prompt word, and filtering records with the category label "Farm" or "Animal Market" according to the matching rules of the core features, and outputting the first raw data.

[0024] Preferably, the JSON format data contains information such as latitude and longitude, province and city, and detailed address; the latitude and longitude are used to generate usable coordinates after extraction and analysis.

[0025] Preferably, field merging, format unification and deduplication are performed on the raw data to generate a structured place data set, including:

[0026] importing the raw data into a Python data processing environment;

[0027] Mapping and merging the field names of different data sources in the raw data into standard fields such as "number, name, address, country, latitude, and longitude";

[0028] Format the latitude and longitude fields according to a unified coordinate system and a preset decimal place accuracy to maintain consistent coordinate representation;

[0029] Determine a duplicate entry if the place names are the same and the corresponding coordinate difference does not exceed a set threshold, and delete the duplicate entry to obtain unique place records;

[0030] Output the deduplicated unique records in CSV format to generate the structured place data set.

[0031] Preferably, according to the global land boundary, multi-level administrative division, and global land use and land cover type, the data points of the structured data set are screened to remove spatially abnormal coordinate points, and finally a random sample review is performed to form the final spatial distribution map of the farm, including:

[0032] Import the structured data set into a geographic information system environment and unify it to a preset coordinate system;

[0033] According to the latitude and longitude value range of the structured data set, a global land boundary vector data is called to perform a spatial intersection analysis, points falling outside the land boundary are deleted, and a first screening set is generated;

[0034] A global multi-layer administrative division vector boundary is called to perform a spatial intersection analysis on the first screening set, records of inconsistent administrative divisions are deleted, and a second screening set is generated;

[0035] A global land use and land cover raster data with a current 10-meter spatial resolution is called to perform a point-based raster value extraction on the second screening set, and each point in the second screening set is assigned a land use and land cover type; records falling in open water, ice and snow, bare land and other land use and land cover types are deleted, and a third screening set is generated;

[0036] Samples are randomly extracted from the third screening set at a preset proportion, visual interpretation review is performed in combination with a GIS visualization interface, positioning doubtful samples are removed, and a data set passing the verification is output;

[0037] The data set passing the verification is loaded to a GIS platform in the form of a point element, and a spatial distribution map is generated.

[0038] An avian influenza virus overflow farm identification system based on geographic spatial big data, comprising:

[0039] An original data acquisition unit is configured to call a public knowledge base, a map API and an open source geographic information database to acquire original data of farms in a global range; the original data at least includes a name, a latitude and longitude, a province to which the farm belongs and a country to which the farm belongs;

[0040] A structured processing unit is configured to perform field merging and format unification on the original data to generate a structured data set;

[0041] A spatial analysis and visualization unit is configured to filter data points of the structured data set according to a global land boundary, a multi-layer administrative division and a global land use and land cover type, remove coordinate points with abnormal spatial positions, and finally perform random sampling review to form a final spatial distribution map of the farms.

[0042] According to the specific embodiments of the present application, the following technical effects are provided:

[0043] The application can fuse multi-source heterogeneous farm and agricultural market geographic information data, improve the integrity and accuracy of data through structured processing and large model semantic calibration, and realize high-precision identification of high-risk places by combining coordinate anomaly elimination and spatial visualization technology. Compared with the traditional method relying on manual annotation or single data source, the application has significant advantages in data coverage, processing automation and spatial recognition accuracy, and provides efficient and scalable data support means for early warning and regional prevention and control of global major animal diseases. BRIEF DESCRIPTION OF DRAWINGS

[0044] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed in the embodiments will be briefly introduced as follows. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0045] Figure 1 The method flowchart provided for the embodiments of the present application is shown in

[0046] Figure 2 The technical route schematic diagram provided for the embodiments of the present application is shown in DETAILED DESCRIPTION

[0047] The technical solutions in the embodiments of the present application will be described clearly and completely with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some embodiments of the present application, not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0048] The purpose of the present application is to provide an avian influenza virus overflow farm identification method and system based on geographic spatial big data, which has significant advantages in data coverage, processing automation and spatial recognition accuracy, and provides efficient and scalable data support means for early warning and regional prevention and control of global major animal diseases.

[0049] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and easy to understand, the present application will be further described in detail with reference to the drawings and specific embodiments.

[0050] Figure 1 The method flowchart provided for the embodiments of the present application is shown in Figure 1 As shown in the figure, the present application provides an avian influenza virus overflow farm identification method based on geographic spatial big data, which includes:

[0051] Step 100: Call public knowledge base, map API and open source geographic information database to collect raw data of farms worldwide; the raw data at least includes name, latitude and longitude, province and country;

[0052] Step 200: Perform field merging and format unification on the raw data to generate a structured data set;

[0053] Step 300: According to the global land boundary, multi-layer administrative division and global land use / land cover type, filter the data points of the structured data set, remove the coordinate points with abnormal spatial position, and finally perform random sampling review to form the final spatial distribution map of farms.

[0054] Preferably, step 100 comprises:

[0055] Construct a SPARQL query template in the public knowledge base Wikidata, limit the entity type to "farms", and get the corresponding links and metadata by page;

[0056] Use the master-slave distributed collection architecture to perform content crawling on the obtained links, the master node splits the URL queue and issues sub-tasks, and the slave node completes the crawling according to the random delay of 0.5s-2s and the IP rotation strategy and returns the results to obtain the first raw data set;

[0057] Call the POI retrieval interface of the Gaode map open platform, first load the 34 provincial administrative divisions and their subordinate cities and counties, then build a tree-shaped collection queue according to the province-city-county order, develop a Python script to realize automatic paging collection cycle, and batch obtain JSON format data of farms in each administrative unit (including latitude and longitude / province / city / detailed address, etc.), and extract the latitude and longitude information therein to parse into usable coordinates. The whole collection process strictly follows the API calling specification to ensure the integrity of the data and the balance of the spatial distribution, and obtains the second raw data set;

[0058] Use the Places api of Google Map Platform to first divide the space in the world into regular grid units with a minimum unit of 20km, then retrieve the farms in each grid range one by one, save their names and coordinates, and obtain the third raw data set.

[0059] Merge the first raw data set, the second raw data set and the third raw data set to construct the raw data of the candidate site.

[0060] As Figure 2As shown, this embodiment uses the Wikidata Query Service to build a SPARQL query template, filtering wiki data and their corresponding links that meet the "farm" entity type globally. A distributed data collection architecture is adopted, sending requests based on wiki links and using IP rotation strategies and request interval control (0.5-2 second random delay) to circumvent anti-scraping mechanisms. The metadata of each farm (including farm information, location, and corresponding webpage information) is stored locally for subsequent analysis and processing. Based on the Gaode Map Open Platform POI search interface, the 34 provincial-level administrative divisions and their subordinate cities, districts, and counties are first loaded. Then, a tree-structured data collection queue is constructed according to the order of province-city-district / county. A Python script is developed to automatically paginate and loop through the data collection, batch acquiring JSON format data (including latitude and longitude, province / city, and detailed address) of farms within each administrative unit. The latitude and longitude information is extracted and parsed into usable coordinates. Using Google Maps Platform's Places API, the global space was first divided into regular grid cells with a minimum unit of 20km. Then, farms within each grid cell were retrieved, and their names and coordinates were saved. Next, based on Python's data structuring process, the semi-structured data was processed and saved as structured information using a Python script. This involved integrating the names, addresses, countries, provinces, latitude and longitude, and detailed addresses of all farms and agricultural markets, and saving them in a local CSV file. Following this, this embodiment focuses on data processing and quality calibration based on a large language model. The obtained structured data was processed using the open-source large language model Deepseek 671B to detect missing text and garbled characters in the main descriptions. The text was then uniformly translated into English, and intelligent completion was performed based on context. Useless information was deleted. Based on this, the main descriptions were summarized to extract core features (such as main business categories and geographical locations). The output format was constrained using the Prompt tool, and the types of desired targets were filtered to generate structured data. Finally, this embodiment visualizes the data space. To spatially visualize latitude and longitude data on a map, the local data is first manually validated using spatial logic, i.e., samples with obvious abnormal spatial coordinate distribution are identified. Then, the consistency analysis between spatial coordinates and administrative divisions is used to verify the rationality of the coordinates and filter out outliers. At the same time, random sampling and visual interpretation analysis are performed to improve the overall quality of the data, and finally, a global spatial distribution map of livestock farms is generated.

[0061] In the United States, particularly in the Midwest and East, livestock farms are densely distributed, which is closely related to the highly developed state of American agriculture and the vast amount of arable land. In Europe, the number of livestock farms is concentrated in Western European countries, including France, Germany, the United Kingdom, and the Netherlands, countries with long agricultural histories and advanced agricultural technologies.

[0062] Corresponding to the above method, the embodiment also provides an avian influenza virus overflow farm identification system based on geospatial big data, comprising:

[0063] An original data acquisition unit is configured to call a public knowledge base, a map API and an open source geographic information database, and collect original data of farms in a global range; the original data at least includes names, longitude and latitude, provinces and countries to which the farms belong;

[0064] A structured processing unit is configured to perform field merging and format unification on the original data to generate a structured data set;

[0065] A spatial analysis and visualization unit is configured to filter data points of the structured data set according to global land boundaries, multi-layer administrative divisions and global land use and land cover types, remove coordinate points with abnormal spatial positions, and finally perform random sampling review to form a final spatial distribution map of the farms

[0066] The beneficial effects of the present application are as follows:

[0067] (1) International Wikidata and domestic and foreign map services are integrated across platforms, distributed architecture + IP rotation strategy is adopted to realize efficient crawling, and a tree-shaped block request mechanism is adopted to solve the problem of hierarchical collection of administrative regions.

[0068] (2) The context understanding ability and multilingual processing ability of Deepseek are adopted to intelligently complete and extract the main content of the web page content, and the Prompt engineering is adopted to realize accurate conversion of unstructured text to standardized data.

[0069] (3) A coordinate anomaly detection + administrative region matching double verification model is constructed, a global geographic data logical correction paradigm is established, and the systematic improvement of cross-language and cross-regional data quality is realized.

[0070] The embodiments in the specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments, and the same and similar parts of each embodiment can be referred to each other. For the system disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple, and the related parts can be referred to the method part.

[0071] The principles and implementation modes of the present application are described by applying specific examples in this paper, and the above embodiment description is only used to help understand the method and core idea of the present application; at the same time, for those skilled in the art, according to the idea of the present application, the specific implementation mode and application range will be changed. In view of the above, the content of the specification should not be understood as a limitation of the present application.

Claims

1. A method for identifying an avian influenza virus spill over from a farm based on geospatial big data, characterized in that, The method comprises the following steps: Calling public knowledge base, map API and open source geographic information database to collect original data of farms in a global range, wherein the original data at least includes name, latitude and longitude, province and country; Performing field merging and format unification on the original data to generate a structured data set; According to global land boundaries, multi-level administrative divisions and global land use / land cover types, screening data points of the structured data set, removing coordinate points with abnormal spatial positions, and finally performing random sampling review to form a final spatial distribution map of farms. The method comprises the following steps:

2. The geospatial big data based avian influenza virus spillage farm identification method according to claim 1, wherein, Building a SPARQL query template in the public knowledge base Wikidata, limiting the entity type to "farm", and obtaining corresponding links and metadata according to pages; Using a master-slave distributed collection architecture to perform content crawling on the obtained links and metadata, splitting the URL queue by the master node of the master-slave distributed collection architecture and issuing sub-tasks, and completing the crawling according to a random delay of 0.5s-2s and an IP rotation strategy by the slave node and returning the results; Using a Deepseek large language model to perform text completion, language unification and place category screening on the returned results to output a first original data set; Calling the POI data retrieval interface of the Gaode map open platform, loading each provincial administrative division and the subordinate cities and counties based on the retrieval interface, and building a tree-shaped collection queue according to the order of province-city-county, developing a Python script to realize automatic paging collection cycle, and batch obtaining JSON format data of farms in each administrative unit to obtain a second original data set; Calling OverpassAPI to obtain spatial elements containing farms from the open source geographic spatial data set OpenStreetMap, first determining the geographic coordinate range of the seven continents, dividing the boundary box of each continent into a grid, traversing the grid in turn, retrieving each label element related to farms, sending an HTTP request, parsing the returned results, and obtaining a third original data set; Merging the first, second and third original data sets to build the original data of farms. Using a Deepseek large language model to perform text completion, language unification and place category screening on the returned results to output a first original data set, comprising:

3. The geospatial big data based avian influenza virus spillage farm identification method according to claim 2, wherein, For the returned results, constructing an AI prompt word according to a preset field order and calibration instruction, and encoding it into a text input sequence readable by the model; Inputting the AI prompt word into the Deepseek large language model to output a corrected text using the missing detection and garbled code recognition capabilities; Performing unified translation of the corrected text into English by the Deepseek large language model, intelligently completing based on the context, deleting useless information, and obtaining a completed text; ​ The semantic summarization and keyword extraction function of the Deepseek large language model is used to extract core features including main business categories and geographic locations from the completed text; The output format is set by the AI prompt word, and records with category labels of "Farm" or "Animal Market" are filtered according to the matching rules of the core features, and the first original data is output.

4. The geospatial big data based avian influenza virus spill over farm identification method as claimed in claim 2 wherein, The JSON format data contains information such as latitude and longitude, province and city, and detailed address; the latitude and longitude are used to generate usable coordinates through extraction and analysis.

5. The geospatial big data based avian influenza virus spill over farm identification method as claimed in claim 1 wherein, The original data is subjected to field merging, format unification and deduplication to generate a structured site data set, including: Import the original data into the Python data processing environment; Map and merge the field names of different data sources in the original data to standard fields "number, name, address, country, latitude, longitude"; Format the latitude and longitude fields according to the unified coordinate system and the preset decimal place accuracy to keep the coordinate representation consistent; Determine duplicate entries based on the same site name and corresponding coordinates with a difference not exceeding the set threshold, and delete duplicate entries to obtain unique site records; Output the de-duplicated unique records in CSV format to generate the structured site data set.

6. The geospatial big data based avian influenza virus spill over farm identification method as claimed in claim 1 wherein, According to the global land boundary, multi-layer administrative division, and global land use and land cover type, the data points of the structured data set are screened to remove spatially abnormal coordinate points, and finally a random sample review is performed to form the final spatial distribution map of the farm, including: Import the structured data set into the geographic information system environment and unify it to the preset coordinate system; According to the latitude and longitude value range of the structured data set, call the global land boundary vector data to perform spatial intersection analysis, delete points outside the land boundary, and generate a first screening set; Call the global multi-layer administrative division vector boundary to perform spatial intersection analysis on the first screening set, delete records with inconsistent administrative divisions, and generate a second screening set; Call the current 10-meter spatial resolution global land use and land cover raster data to perform point-based raster value extraction on the second screening set, and assign a land use and land cover type to each point in the second screening set; delete records with land use and land cover types of open water, ice and snow, and bare land to generate a third screening set; Randomly sample the third screening set according to a preset proportion, combine with the GIS visualization interface for visual interpretation review, remove positioning questionable samples, and output the verified data set; Load the verified data set into the GIS platform in the form of point features to generate a spatial distribution map.

7. A system for identifying avian influenza virus spillover from farms based on geospatial big data, characterized in that, It includes: An original data acquisition unit for calling public knowledge bases, map APIs, and open-source geographic information databases to collect original data of farms worldwide; the original data at least includes name, latitude and longitude, province and country; A structured processing unit for performing field merging and format unification on the original data to generate a structured data set; The spatial analysis and visualization unit is used to filter the data points of the structured data set according to the global land boundary, multi-layer administrative division and global land use / land cover type, eliminate coordinate points with abnormal spatial positions, and finally perform random sampling review to form a final spatial distribution map of the farms.

Images