Construction method of agricultural and rural data warehouse based on dataworks

Abstract

The application is particularly a construction method of agricultural and rural data warehouse based on Dataworks, and relates to the fields of big data warehouse construction and agricultural digitization technology, and comprises: based on the best practice of DataWorks data warehouse construction, a five-layer data warehouse architecture is built to adapt to the characteristics of agricultural and rural data. In the application, in view of the unique characteristics of agricultural and rural data, such as multi-source heterogeneity, strong regional and periodic attributes, and wide business scene coverage, a dedicated data warehouse architecture, theme domain system and public dimension specification are designed, the full-warehouse index statistical caliber is unified, the barriers of cross-department and cross-business agricultural-related data are broken, the deep fusion and efficient reuse of multi-source agricultural-related data are realized, the technical threshold of agricultural and rural data warehouse construction is reduced, and the feasibility and generalizability of the scheme are improved, so that the scheme can adapt to the differentiated development needs of different regional digital rural construction.

Description

Technical Field

[0001] This invention relates to the fields of big data warehouse construction and agricultural digitalization technology, and in particular to a method for constructing an agricultural and rural data warehouse based on Dataworks. Background Technology

[0002] Data warehouses, as the core carrier for agricultural data resource integration, standardized governance, value mining, and business empowerment, are key infrastructure for the construction of agricultural digital systems. Alibaba Cloud DataWorks, a one-stop big data development and governance platform, has achieved large-scale application in the construction of data systems in multiple industries with its mature product capabilities covering the entire chain of data integration, development, governance, and services, providing a solid technical foundation for the standardized and regulated construction of agricultural and rural data warehouses.

[0003] Currently, the construction of data warehouses in the agricultural and rural sectors has undergone multi-dimensional practical exploration. Various solutions have been implemented extensively, focusing on the aggregation, centralized storage, and basic statistical analysis of agricultural data. This has initially achieved centralized management of agricultural data from multiple departments, providing data support for digital agricultural supervision and services. Agricultural and rural data possess unique characteristics such as multi-source heterogeneity and dispersion, strong regional and seasonal attributes, high privacy sensitivity, and wide coverage of business scenarios. These characteristics place higher industry demands on the data warehouse's adaptability to business scenarios, multi-source data fusion capabilities, full-process compliance control, and the linkage between construction and application.

[0004] Therefore, there is an urgent need for a solution that can be deeply integrated with the characteristics of agricultural and rural business, and form a standardized full-process construction path based on the capabilities of mature big data platforms, so as to meet the refined development needs of agricultural digital transformation.

[0005] Meanwhile, in the construction of agricultural and rural data warehouses, the core aspects that need to be improved in the current industry are: unified indicators across departments and businesses; standardized integration and governance of multi-source heterogeneous data; full-process control of data security and compliance; and the construction of an iterative mechanism from business needs to value realization. Based on this, how to leverage DataWorks' full-chain product capabilities to build a standardized, implementable, and continuously iterative data warehouse construction methodology that adapts to all business scenarios in agriculture and rural areas, and achieves standardized governance of agricultural data assets and efficient release of business value, has become an important research direction in the field of digital rural construction. Summary of the Invention

[0006] The purpose of this invention is to propose a method for constructing an agricultural and rural data warehouse based on Dataworks in order to solve the above-mentioned problems.

[0007] To achieve the above objectives, the present invention adopts the following technical solution:

[0008] The method for building an agricultural and rural data warehouse based on Dataworks includes: The entire process of the three core business lines of agriculture and rural affairs, namely government supervision, production and operation and public services, is sorted out. The multi-source heterogeneous data sources of agriculture and rural areas are inventoried and sensitive data are classified and marked. The business users are connected to break down the feasible analysis scenarios, construct a standardized three-level indicator system for agriculture and rural areas and unify the indicator caliber of the whole warehouse, and define the red line for data compliance management. Based on the best practices of DataWorks data warehouse construction, a five-layer data warehouse architecture adapted to the characteristics of agricultural and rural data is built. The five-layer architecture includes ODS operation data layer, DWD data detail layer, DWS data aggregation layer, DIM common dimension layer, and ADS application data layer. It divides the exclusive business theme domains for agriculture and rural areas, constructs a bus matrix that associates agricultural business processes with common dimensions, and formulates full-process data warehouse design and development specifications. Based on the DataWorks data integration module, the system adapts and securely connects to all types of agricultural and rural data sources. It formulates differentiated offline batch synchronization and real-time synchronization strategies for different agricultural data update frequencies and real-time requirements, configures a synchronization task stability guarantee mechanism, and standardizes and aggregates scattered multi-source heterogeneous agricultural data to the ODS layer. Based on the DataWorks data development module, the system completes the full-link layered data processing from the ODS layer to the ADS layer according to the five-layer data warehouse architecture. It achieves multi-source data fusion and characteristic processing specific to agricultural scenarios, configures a full-link automated task scheduling and operation and maintenance system, and implements full-process data governance capabilities based on DataWorks.

[0009] Preferably, the method further includes: Based on the DataWorks data service module, a standardized data service API system for agricultural scenarios is built and full lifecycle security management is completed. Through standardized APIs, the value of data warehouse business is realized by connecting with various business scenarios in agriculture and rural areas.

[0010] Preferably, the three-level sensitive data classification and labeling is performed on agricultural and rural data: The first-level sensitive data includes farmers' personal identity information, bank card information, and other private data, as well as surveying and mapping data such as the red line for arable land and the confidential coordinates of high-standard farmland. The second-level sensitive data includes unpublished agricultural statistics and agricultural enterprise business operation data. The third-level sensitive data consists of publicly available meteorological and industry statistics. The standardized three-level indicator system includes indivisible atomic indicators, derived indicators bound to dimensions and statistical periods, and business composite indicators combining multiple indicators. All indicators have clear and unified statistical logic, data sources, and calculation rules.

[0011] Preferably, in the five-layer data warehouse architecture, the ODS layer fully preserves the original agricultural data traces and adapts to the compliant separate storage of confidential spatial data; the DWD layer splits the fact table according to the minimum agricultural business process and completes data cleaning, desensitization, and standardization; the DWS layer completes light aggregation and pre-calculation of general indicators according to the high-frequency analysis dimensions of agriculture; the DIM layer constructs a unified public dimension specifically for agriculture and rural areas; and the ADS layer generates directly usable result data for specific agricultural business scenarios. The unified public dimension specifically for agriculture and rural areas includes the national standard five-level administrative division, unified land parcel coding, standard crop variety codes of the Ministry of Agriculture and Rural Affairs, unified credit codes for agricultural business entities, and agricultural solar term dimensions.

[0012] Preferably, the divided primary subject areas dedicated to agriculture and rural areas include five major subject areas: agricultural production, agricultural product circulation and quality safety, rural economy and business entities, arable land and resource environment, and agricultural government supervision. Each primary subject area is further divided into corresponding secondary sub-sub-themes. The bus matrix uses the minimum agricultural business process as the row and the unified public dimension as the column, clarifying the relationship between each business process and the public dimension, and realizing the unification of the entire warehouse dimension scope.

[0013] Preferably, the adapted agricultural and rural data sources include relational databases of government and production business systems, field soil moisture / insect / meteorological IoT devices, manual ledger documents, satellite / drone remote sensing geospatial data, and third-party meteorological / price / public opinion API interface data; The differentiated synchronization strategy employs daily incremental and monthly full offline batch synchronization for static low-frequency data, second-level FlinkCDC real-time synchronization for high-frequency real-time IoT monitoring and agricultural product price data, and a compliance strategy of separating attribute data from spatial subject data for the synchronization of classified spatial data.

[0014] Preferably, in the end-to-end layered data processing, the ODS layer to the DWD layer completes data deduplication, outlier filtering, null value filling, sensitive data anonymization, dimension standardization mapping, and unique primary key generation; the DWD layer to the DWS layer completes the association of multiple detailed fact tables and the construction of highly reusable summary wide tables; and the DWS layer to the ADS layer completes the calculation of scenario-based indicators and the implementation of complex business logic. All processing tasks are included in version management to support rapid rollback in abnormal scenarios.

[0015] Preferably, the agricultural-specific customized processing includes: By integrating agricultural records across the entire chain according to plot, crop, and growth cycle, a full-cycle agricultural growth archive is constructed. Through unified plot coding and management entity coding, multi-source fusion of remote sensing growth data, IoT monitoring data, agricultural operation records, and agricultural subsidy data is achieved. For core agricultural indicators, multi-dimensional drilling and processing of five-level administrative divisions (province-city-county-township-village) and industry types is completed, as well as the linkage and correlation processing of GIS spatial analysis results and business data.

[0016] Preferably, the automated task scheduling strictly follows the hierarchical order of the data warehouse to configure upstream and downstream dependencies, and configures differentiated scheduling cycles of daily, hourly, monthly, and trigger-based scheduling to meet the real-time needs of different agricultural business scenarios; the full-process data governance includes metadata management and full-link data lineage construction based on DataWorks data maps, field-level fine-grained permission control, data lifecycle management, and standardized data asset catalog construction.

[0017] Preferably, the standardized data service API system for agricultural scenarios includes four core interfaces: indicator query API, detailed data query API, real-time monitoring API, and statistical report API; the full lifecycle security management includes API access permission and IP whitelist management, traffic and call frequency management, automatic anonymization of returned data, HTTPS encrypted transmission, full call log retention, and operation auditing.

[0018] In summary, due to the adoption of the above technical solution, the beneficial effects of the present invention are: 1. This invention addresses the unique characteristics of agricultural and rural data, such as its multi-source heterogeneity, strong regional and cyclical attributes, and wide coverage of business scenarios. It designs a dedicated data warehouse architecture, subject domain system, and common dimension specifications, unifies the statistical standards for all warehouse indicators, breaks down barriers between agricultural data across departments and businesses, achieves deep integration and efficient reuse of multi-source agricultural data, lowers the technical threshold for building agricultural and rural data warehouses, and improves the feasibility and scalability of the solution, enabling it to adapt to the differentiated development needs of digital village construction in different regions.

[0019] 2. This invention achieves secure and compliant management of agricultural data throughout its entire lifecycle and ensures the continuous and efficient release of its business value. It proactively completes the hierarchical identification and compliance rule formulation of sensitive agricultural data, integrates data security management throughout the entire process of data collection, processing, and service, and fully adapts to relevant laws and regulations on data security and the requirements of agricultural and rural data management standards through end-to-end de-identification and encryption, refined access control, and end-to-end operation auditing mechanisms. This effectively protects the security of agricultural data, especially farmers' privacy data and confidential regulatory data. Attached Figure Description

[0020] Further details, features, and advantages of this application are disclosed in the following description of exemplary embodiments in conjunction with the accompanying drawings, in which: Figure 1 This is a flowchart of the method of the present invention. Detailed Implementation

[0021] Several embodiments of this application will now be described in more detail with reference to the accompanying drawings to enable those skilled in the art to implement this application. This application may be embodied in many different forms and for various purposes and should not be limited to the embodiments set forth herein. These embodiments are provided to make this application thorough and complete, and to fully convey the scope of this application to those skilled in the art. The embodiments described do not limit this application.

[0022] Unless otherwise defined, all terms used herein (including technical and scientific terms) shall have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains. It will be further understood that terms such as those defined in commonly used dictionaries shall be interpreted as having a meaning consistent with their meaning in the relevant field and / or the context of this specification, and shall not be interpreted in an idealized or overly formal sense unless expressly defined herein.

[0023] Example 1 Its specific implementation method is combined with the appendix Figure 1 Please provide a detailed explanation.

[0024] Appendix Figure 1 The flowchart of the method for constructing an agricultural and rural data warehouse based on Dataworks provided in this embodiment of the invention demonstrates the complete steps from the end-to-end review of agricultural and rural government supervision to the implementation of full-process data governance capabilities based on DataWorks.

[0025] In this embodiment, it includes: Step 1: Business Research and Needs Analysis This is the core steering wheel for data warehouse construction. The core objective is to anchor the boundaries, value anchors, and compliance red lines of data warehouse construction, and to solve the core pain points of agricultural data warehouses such as "building but not being able to use them" and "inconsistent interpretations among departments" from the source. It clarifies "what to build, for whom to build, how to use, and what the compliance bottom line is" for data warehouses.

[0026] First, a thorough review of the entire agricultural and rural business chain is needed, breaking down departmental and business barriers and comprehensively analyzing all process nodes of the three core agricultural and rural business lines. The government regulatory business line covers core functions such as the distribution of agricultural subsidies, high-standard farmland construction, arable land protection, agricultural product quality and safety supervision, and rural collective asset management, outlining the entire process of each business from approval and execution to supervision and auditing. The production and operation business line covers the entire agricultural industry chain, including pre-production agricultural input supply and seed registration, agricultural operations and technical services throughout the entire growth cycle of planting and breeding, and post-production processing, warehousing, logistics, and brand marketing, clearly defining the smallest operational actions such as sowing, fertilization, quality inspection, and harvesting. The public service business line covers meteorological services, pest and disease early warning, agricultural financial insurance, and rural living environment improvement, clarifying the triggering conditions and core logic of these services.

[0027] Secondly, a comprehensive inventory and hierarchical labeling of multiple data sources must be completed. Agricultural and rural data is complex and highly dispersed, requiring a systematic inventory of all available data sources. This includes structured ledger data from government and enterprise business systems, semi-structured message data reported by field IoT devices, unstructured data such as satellite remote sensing imagery and inspection data, as well as third-party external data such as meteorological, price, and public opinion data. During the inventory process, the ownership department, update frequency, quality status, and access feasibility of each type of data must be clearly defined. Simultaneously, the data must be hierarchically labeled, distinguishing between core production data, regulatory confidential data, personal privacy data, and publicly available statistical data, thus establishing rules for subsequent data access, processing, and access control in advance.

[0028] The next step is to complete the decomposition of business scenarios and the standardized definition of the indicator system. This is the core prerequisite for realizing the value of the data warehouse. It requires connecting with specific users such as government departments, cooperatives, farmers, and agricultural enterprises to clarify the applicable analysis scenarios, and then decompose them into an indicator system with completely unified output. First, break down the scenarios into three categories based on the user: government supervision, industrial services, and public services. Clarify the core needs, analysis dimensions, and data update requirements of each scenario. Then, build a standardized indicator dictionary according to basic, indivisible atomic indicators, derived indicators with dimensions and statistical periods, and composite indicators for business evaluation combining multiple indicators. The most crucial point is to unify the statistical logic, data sources, and calculation rules of each indicator. For example, clarify and unify the official verification standard for "sown area" to avoid inconsistencies in the statistical results of the same indicator by different departments from the source.

[0029] Finally, it is essential to pre-identify compliance and privacy risks. Agricultural and rural data contains a large amount of farmers' personal privacy, classified surveying and mapping data, and regulatory data. Risks must be identified in advance, and compliance bottom lines must be clearly defined. Sensitive data should be categorized and labeled, with the highest level including farmers' ID cards, bank cards, and classified farmland coordinates; the second level including unpublished agricultural statistics and agricultural enterprise operating data; and the third level including publicly available industry data. Simultaneously, in accordance with relevant national regulations on data security, personal information protection, and agricultural and rural data management, the de-identification rules, encryption requirements, access control scope, and cross-network transmission standards for different levels of data must be clearly defined, establishing inviolable compliance red lines for subsequent data processing throughout the entire process.

[0030] This step will be accompanied by DataWorks' data mapping function to explore the source system's metadata, to understand the current status of the data structure and quality in advance, to pre-configure sensitive data identification rules through the data security center, and finally deliver a business research white paper, a full list of data sources and access solutions, a standardized indicator system dictionary, and a sensitive data classification and compliance rule manual.

[0031] Step Two: Architecture and Data Model Design This is the core framework of data warehouse construction. The core objective is to build a layered data warehouse architecture and data model based on the DataWorks lake warehouse integrated architecture, which is adapted to the characteristics of agricultural and rural data, scalable, highly reusable, and with unified standards. This will completely solve the problems of data silos, redundant development, and poor reusability, and provide a standardized storage and circulation space for agricultural data.

[0032] First, we need to complete a data warehouse layered architecture design that conforms to DataWorks best practices, strictly follow the classic layered system of DataWorks offline data warehouses, and make special optimizations for the characteristics of agricultural and rural data. We need to clarify the positioning, processing rules and design boundaries of each layer to ensure that the data flows in an orderly manner, is traceable and reusable.

[0033] The bottom layer is the operational data layer, which is the first stop for data warehouse entry. It completely retains the original data that is consistent with the source system, only performs basic format adaptation, and does not perform any in-depth processing. According to the data update type, it is divided into three categories: incremental synchronization, full synchronization, and temporary buffer. It completely retains all traces of the original data to meet the needs of subsequent data auditing and backtracking. Confidential spatial data only synchronizes attribute fields, and the main data is stored in a compliant GIS engine.

[0034] Above that is the data detail layer, the core foundational layer of the data warehouse. Its core function is to standardize, clean, and de-identify the data. It breaks down factual data according to the smallest agricultural business processes, such as sowing, fertilizing, plant protection, and subsidy disbursement, each corresponding to an independent detailed data system. This layer performs full data cleaning, including deduplication, outlier filtering, null value filling, and format standardization. Simultaneously, it de-identifies and encrypts sensitive data according to pre-defined compliance rules, then associates it with unified public dimensions, mapping non-standard fields to standard codes, generating a unique identifier for each detailed data entry, and constructing a complete data flow chain to achieve end-to-end traceability.

[0035] Above that is the data aggregation layer, whose core purpose is to improve data reusability, reduce redundant calculations, and lower the query pressure on upper-layer applications. It performs light aggregation and pre-calculation based on high-frequency agricultural analysis dimensions, including time, administrative division, plot, crop variety, business entity, and agricultural seasons. This layer builds highly reusable wide aggregation tables, such as a wide table integrating the entire process of sowing, fertilization, plant protection, and harvesting for crop growth cycles at each plot level, and a monthly production and sales summary table for each county category. Simultaneously, it performs unified pre-calculation of universal basic indicators across the entire database, ensuring that all application scenarios use the same set of indicators. At the same time, it strictly controls the aggregation granularity to avoid excessive aggregation that leads to data loss and inability to support subsequent refined drill-down analysis.

[0036] Parallel to the detail layer and the summary layer is the dimension layer, which is the common language of the data warehouse. Its core is to unify the dimension definitions across business operations and completely break down data silos. This layer will construct a unified public dimension specific to agriculture and rural areas, including the national standard five-level administrative divisions, land parcels with unified land ownership codes, crop varieties according to the Ministry of Agriculture and Rural Affairs standards, business entities with unified credit codes, as well as core dimensions such as soil type, agricultural input categories, and time of year. At the same time, it will establish mapping relationships between different systems for the same dimension, solve the problem of inconsistent coding between different departments, and also manage dimension versions to cope with slow changes such as administrative division adjustments and crop variety updates.

[0037] The top layer is the application data layer, which directly connects to the upper-layer business applications. It completes the final indicator calculation for specific business scenarios, builds the result data system according to specific business scenarios, and fully meets the query needs of applications, such as the compliance audit results of agricultural subsidies and the indicator data of digital rural screens. It completes the final implementation of complex business indicators and prediction model calculation results, while adapting to the format requirements of external system interfaces, without requiring secondary processing on the application side.

[0038] After completing the layered architecture design, the next step is to divide the agricultural subject domains and design a bus matrix. This is crucial for avoiding siloed development and connecting all business data. First, based on the characteristics of agricultural and rural business, primary subject domains are defined, including five core themes: agricultural production, agricultural product circulation and quality safety, rural economy and business entities, arable land and resource environment, and agricultural government supervision. Each theme is further subdivided into corresponding sub-themes to ensure all data has a clear affiliation. Then, the bus matrix clarifies the relationship between each minimum business process and common dimensions. Rows in the matrix correspond to minimum business processes such as sowing, fertilization, and subsidy distribution, while columns correspond to unified common dimensions. Intersections mark the relationship between business processes and dimensions, ensuring all business processes use a unified dimension and completely resolving the problem of cross-business data incompatibility.

[0039] Finally, a unified data warehouse model design specification should be established for the entire team to ensure the maintainability and scalability of the data warehouse. This includes a unified naming convention for tables and fields, clear naming rules for layers and subject areas, and clear Chinese comments for all fields, clearly defining the meaning, statistical scope, and source of the data. The model design should adhere to the principles of high cohesion and low coupling, scalability, traceability, and consistency of data scope to avoid excessive data redundancy and duplicate calculations. At the same time, the storage lifecycle of different layer tables should be clearly defined, such as the retention period for raw data, the permanent retention rules for core business data, and the automatic cleanup cycle for temporary tables, so as to reasonably control storage costs.

[0040] This step will use DataWorks' data modeling module to complete dimensional modeling and table structure design, and use data maps to achieve unified management of table structure and metadata. The final deliverables include data warehouse layered architecture design documents, subject domains and bus matrices, dimensional model and table structure design specifications, and data warehouse development design specifications.

[0041] Step 3: Multi-source data acquisition and integration This is the core data entry point for data warehouse construction. The core objective is to achieve stable, efficient, secure, and compliant unified aggregation of multi-source heterogeneous agricultural and rural data based on the DataWorks data integration module. It standardizes and synchronizes data scattered in the fields, various departments, and different systems to the operational data layer, solving the core pain points of agricultural data being "unable to be collected, integrated, or transmitted stably".

[0042] First, adaptation and connection configuration for all types of data sources must be completed. DataWorks supports offline and real-time synchronization of dozens of data sources. For data sources specific to agricultural and rural scenarios, targeted adaptation and connection debugging are required. For various relational databases in government business systems and enterprise production management systems, corresponding JDBC connections must be configured, source system whitelists and network connectivity debugging must be completed, and only read-only account permissions for the source system must be requested to avoid synchronization tasks affecting the normal operation of the source business system. At the same time, the identifier fields for incremental data synchronization must be clearly defined, field mapping must be properly configured, and all information of the original fields must be completely preserved. For IoT data sources such as field soil moisture, insect infestation, weather stations, and integrated water and fertilizer equipment, direct connection to the Alibaba Cloud IoT platform is required. The message data reported by the devices is parsed, and dirty data such as offline, over-range, and duplicate reports are filtered in advance. Real-time data synchronization at the second to minute level is supported to meet the needs of real-time agricultural monitoring and early warning. For file-based data sources such as manually reported ledgers and statistical reports, corresponding file service connections must be configured, file parsing rules must be clearly defined, large file fragment synchronization is supported, and a mechanism for triggering synchronization by configuration file upload is implemented to adapt to the need for irregular updates of manually reported data. For spatial geographic data sources such as satellite remote sensing imagery, UAV aerial photography data, and land parcel boundary vector data, strict adherence to surveying and mapping data confidentiality regulations is maintained. Classified data is synchronized only on the government intranet, while spatial subject data is stored in a dedicated GIS engine. Simultaneously, corresponding attribute data such as land parcel codes, areas, and crop types are synchronized to the data warehouse via DataWorks, establishing a unique mapping between attribute data and spatial data to enable subsequent interconnection and integration. For third-party API data sources such as meteorological data, agricultural product prices, and public opinion data, corresponding interface connections are configured, interface authentication, parameter configuration, and data parsing are completed, and data update frequency and call frequency limits are clearly defined. Data copyright and usage scope are also clearly marked to ensure compliant use.

[0043] Secondly, differentiated strategies for synchronization tasks need to be formulated. These strategies should be tailored to the update frequency, real-time requirements, and data volume of different types of agricultural data, balancing synchronization efficiency and resource consumption. For static data and infrequently updated ledgers, statistical data, and dimension tables, an offline batch synchronization mode is adopted, combining full and incremental synchronization. Static dimension tables undergo monthly full synchronization, while business system data undergoes daily incremental synchronization, synchronizing only changed data to reduce resource consumption. Scheduling time is uniformly set during the early morning off-peak business period to avoid impacting the source system's operation. For high-frequency updated IoT monitoring data, agricultural product price data, and traceability barcode scanning data with high real-time requirements, a real-time synchronization mode is adopted, achieving second-level incremental synchronization of source data. Real-time cleaning, format conversion, and data filtering rules are configured to directly write the parsed standardized data into the data warehouse. A fault recovery mechanism is also configured to ensure no data loss or duplication. For offline collected data from remote areas and government intranet data crossing network gateways, corresponding offline file import and network gateway adaptation synchronization tasks are configured to ensure full and complete data access.

[0044] Finally, to ensure the stability and reliability of the synchronization tasks, a comprehensive stability assurance system is established to address common challenges in agricultural and rural data synchronization, such as network fluctuations, device offline, and source system instability. All synchronization tasks are enabled with breakpoint resumption by default. If a task is interrupted due to network fluctuations or source system failures, it can resume synchronization from the breakpoint without requiring a full resynchronization. A retry strategy is also configured to handle temporary failures. After synchronization is completed, a data consistency check is automatically performed, comparing the completeness and accuracy of the source and target data. If the check fails, an alarm is automatically triggered to ensure zero data loss and zero errors. A synchronization rate limiting strategy is configured for the source business system to control the synchronization speed and concurrency, preventing excessive resource consumption by the synchronization tasks. Elastic resources are configured for large-scale data synchronization to ensure stable operation. Simultaneously, full-link monitoring and alarm rules are configured in the DataWorks operations center to automatically trigger alarms for tasks that fail, synchronization delays exceeding thresholds, abnormal data fluctuations, and check failures, notifying operations personnel through designated channels to handle the faults promptly.

[0045] This step involves using DataWorks' data integration module to configure and maintain offline and real-time synchronization tasks, using the operations and maintenance center to monitor and configure alarms, using the data map to manage metadata and establish lineage for inbound data, and finally delivering a data source synchronization configuration manual, synchronization task design and strategy document, data consistency verification rules, and a list of synchronization task monitoring and alarm configurations.

[0046] Step 4: Data Processing and Scheduling Governance This is the core of the data warehouse's "data-to-gold" transformation. The core objective is to leverage DataWorks' data development and governance capabilities to complete the standardized processing of data across the entire chain, from the operational data layer to the application data layer. This includes data cleaning and desensitization, aggregation and computation, and quality control. Simultaneously, it involves building an automated scheduling and maintenance system and full-process data governance capabilities to transform raw agricultural data into high-value, high-quality, and reusable data assets.

[0047] First, the development of a layered data processing system across the entire data pipeline must be completed. This strictly adheres to the pre-designed layered architecture, processing data layer by layer from the operational data layer to the detail layer, summary layer, and application layer. Each layer has clearly defined processing boundaries and core actions to ensure standardized, traceable, and reusable data processing. Processing from the operational data layer to the detail layer forms the core of data quality control. The core task is to transform raw data into standardized, clean, and compliant detailed data. This mainly involves data cleaning, sensitive data desensitization and encryption, dimensional association and standardized mapping, fact table splitting and unique identifier generation, and building a complete data lineage to ensure the traceability of the source and processing chain of each data point. Processing from the detail layer to the summary layer focuses on improving data reusability and reducing redundant calculations. Light aggregation and pre-calculation are performed according to high-frequency agricultural analysis dimensions. Multiple detailed fact tables are linked and integrated into a highly reusable wide summary table. Simultaneously, unified pre-calculation of general basic indicators across the entire data warehouse is completed to ensure consistent indicator definitions. At the same time, the aggregation granularity is strictly controlled to retain data details to support subsequent refined analysis. The processing from the aggregation layer to the application layer is the final link that directly connects to business applications. Its core is to be oriented towards specific business scenarios, complete the final indicator calculation, implement complex models, generate directly usable result data, build scenario-based wide result tables that fit the application query requirements, complete complex logic such as composite indicator calculation, compliance cross-validation, and prediction model result calculation, and adapt to the format requirements of external systems without the need for secondary processing on the application side.

[0048] Secondly, it is essential to complete specialized processing tailored to agricultural scenarios. This involves in-depth processing of agricultural and rural data, addressing its unique characteristics and achieving deep integration of multi-source data. This is the core value that distinguishes agricultural data warehouses from general-purpose data warehouses. This includes building a complete agricultural lifecycle archive, integrating scattered agricultural records from different systems and stages into a comprehensive "from planting to harvest" archive, organized by plot, crop, and growth cycle. This achieves one full-cycle archive for each plot and crop, supporting full traceability and refined production management of agricultural products. It also includes multi-source data fusion and correlation. Through unified plot and subject coding, it connects remote sensing imagery growth data, field IoT monitoring data, farmer agricultural records, subsidy distribution data, and yield and quality inspection data, achieving the integration of "one plot, one map, and all data," completely breaking down data silos. This includes multi-dimensional drill-down processing of agricultural indicators, targeting core indicators such as farmer income and industry output value. It achieves hierarchical aggregation based on five levels of administrative divisions, industry type, business entity type, and agricultural seasons, supporting full-chain drill-down analysis from macro to micro levels to meet the regulatory and analytical needs of different levels. It also includes the linkage processing of spatial and business data, synchronizing spatial analysis results from the GIS engine back to the data warehouse and performing calculations in conjunction with business data to achieve deep linkage between spatial and business data.

[0049] The next step is to build an automated task scheduling and operation and maintenance system, which is the core guarantee for the stable operation of the data warehouse 24 / 7. First, the upstream and downstream dependencies of tasks must be configured strictly according to the data warehouse's layered order. This ensures that downstream processing tasks are executed only after upstream data synchronization is complete, avoiding logical errors such as "processing with outdated data." Simultaneously, cross-task dependencies must be configured to ensure that data updates in related tables remain synchronized. Then, differentiated periodic scheduling strategies must be configured to meet the real-time requirements of different agricultural scenarios. Most business processing tasks are configured for daily scheduling during the early morning off-peak period to calculate the full data of the previous day; monitoring and early warning tasks with high real-time requirements are configured for hourly or minute-level high-frequency scheduling; low-frequency statistical and inventory tasks are configured for weekly, monthly, or annual scheduling; and manually reported temporary data is configured for scheduling modes triggered manually or by file upload, flexibly adapting to temporary needs. Furthermore, a comprehensive end-to-end monitoring and alerting system integrating "task status, data quality, and resource usage" needs to be established. This system monitors anomalies such as task execution failures, timeouts, and failures to run on time. It also configures table-level and field-level data quality verification rules to ensure that processed data meets requirements. The system monitors the utilization of computing and storage resources to prevent task failures due to insufficient resources. Different notification channels and escalation rules are configured according to the severity of the anomalies to ensure timely handling. All processing tasks are also included in version management; each modification automatically generates a new version, allowing for rapid rollback to a stable historical version in case of anomalies, preventing end-to-end failures caused by modifications.

[0050] Finally, the implementation of a comprehensive data governance system is crucial for the continuous and healthy operation of the data warehouse and the realization of data assetization. DataWorks data mapping manages the metadata of all tables and fields within the warehouse, enhancing annotations for all tables and fields, clarifying data meaning, definitions, and sources, automatically constructing the entire data lineage, clearly showing the complete data flow from the source system to the application layer, supporting forward and reverse traceability, and meeting data auditing and problem-solving needs. Following the principle of least privilege, fine-grained data security and access control are configured. Sensitive and privacy fields are only authorized for viewing by designated personnel; ordinary users can only see anonymized data. Different access, editing, and calling permissions are assigned according to different roles such as development, operations, analysis, and supervision to prevent unauthorized operations. Simultaneously, all data access, modification, and export operations are fully recorded, ensuring traceability of all data behavior and meeting compliance auditing requirements. For different layers and types of tables, corresponding storage lifecycles are configured, automatically cleaning up expired temporary and invalid data, optimizing storage costs, and ensuring the permanent retention of core business data to meet traceability and auditing needs. Furthermore, all tables, metrics, and dimensions in the entire warehouse should be organized into a standardized data asset catalog, categorized and displayed by subject area and business scenario, so that business personnel can quickly find, understand, and use the data, achieving the goal of making data assets searchable, usable, and manageable.

[0051] This step involves using DataWorks' data development module to configure processing tasks, using the operations and maintenance center to configure scheduling and task operations, using the data quality module to configure quality rules, and using the data map and data security center to perform data governance. The final deliverables include a full-chain data processing script library, a task scheduling and dependency configuration manual, a data quality verification rule library, data governance and access control specifications, and a data asset catalog.

[0052] Step 5: Data Services and Scenario Applications This is the core value outlet of data warehouse construction. The core objective is to output the processed data assets in the data warehouse to upper-level business applications in a standardized, secure and controllable manner, so as to realize the business value of the data warehouse. At the same time, based on application feedback, the data warehouse is iterated and optimized in reverse to form a complete process of "construction-application-feedback-optimization", so that the data warehouse can continuously adapt to business development and continuously generate value.

[0053] First, we need to build standardized data service interfaces. Based on the DataWorks data service module, we can quickly build standardized, manageable, and highly available data service interfaces using low-code methods. This is the core channel for outputting the value of data assets. We designed four core interfaces for different application scenarios in agriculture and rural areas, covering the vast majority of business needs: Indicator query interfaces, which primarily connect to visual reports, dashboards, and decision-making dashboards, supporting queries of core indicators by dimension and time period, and supporting multi-dimensional combined queries; Detailed data query interfaces, which primarily connect to business management systems and traceability platforms, supporting precise queries of detailed data such as plot agricultural records, agricultural product traceability information, and business entity subsidy records by unique identifiers; Real-time monitoring interfaces, which primarily connect to field monitoring systems and early warning platforms, supporting high-frequency, low-latency real-time data queries; and Statistical report interfaces, which primarily connect to government office systems and statistical systems, supporting the generation of standardized statistical reports in fixed formats.

[0054] Simultaneously, a comprehensive security management system for the entire lifecycle of interfaces must be established to ensure compliant and controllable data output, addressing the privacy and confidentiality concerns of agricultural data. Each interface is configured with access permissions, allowing only authorized applications and accounts to call it. An IP whitelist is also configured, allowing access only from specified server IPs to prevent unauthorized calls. Traffic and frequency limits are configured for each interface to prevent malicious calls and protect data warehouse computing resources. Interface-returned data automatically undergoes de-identification rules, with sensitive fields returning de-identified results by default. Only high-privilege authorized accounts can request to view plaintext data. Encryption protocols are used throughout interface transmission to ensure data security. All interface call logs are fully recorded, including caller, call time, call parameters, and returned data volume, ensuring traceability of all data output activities and meeting compliance audit requirements. Furthermore, the entire process of interface development, testing, release, operation, and decommissioning must be managed within DataWorks. Interface modifications must undergo testing and approval before release to ensure stability. Standardized interface documentation is automatically generated to reduce integration costs with business systems.

[0055] Secondly, it is necessary to complete the implementation of multi-scenario business applications. Based on the data capabilities and standardized interfaces of the data warehouse, it is necessary to connect with various business scenarios in agriculture and rural areas, so that the value of the data warehouse can be truly realized, and the core coverage is three major categories of scenarios. The first category is government supervision applications, which mainly serve government departments such as agriculture and rural areas, natural resources, and auditing, to achieve "unified management and precise supervision". This includes building a digital village dashboard and supervision screen, connecting with visualization tools, and based on the indicator data of the data warehouse, constructing an industrial development map, a map of high-standard farmland construction, a traceability screen for agricultural product quality and safety, and a supervision dashboard for agricultural subsidies, so as to achieve a one-screen overview of the development of agriculture and rural areas in the county and assist leaders in decision-making. This includes compliance auditing of agricultural subsidies, which automatically screens for abnormalities in subsidy distribution through cross-comparison of multi-source data in the data warehouse, assists auditing departments in carrying out precise supervision, and improves the efficiency of subsidy fund use. This includes dynamic monitoring of arable land protection, which connects multi-source data to monitor the illegal use of arable land in real time, automatically warns of violations, and achieves normalized and precise supervision of arable land protection. This also includes full traceability of agricultural product quality and safety, which integrates full-chain data to achieve full traceability of agricultural products "from field to table", and can quickly locate the problem link when quality problems occur.

[0056] The second category is industry service applications, which primarily serve cooperatives, agricultural enterprises, and large-scale growers, helping them reduce costs, increase efficiency, stabilize production, and increase income. These include crop yield forecasting and pest and disease early warning, which uses full historical data in the data warehouse combined with predictive models to predict crop yields in advance, warn of pest and disease risks, and accurately push prevention and control plans; agricultural product price monitoring and production-sales matching, which integrates national price data, local production and demand data, monitors price fluctuations in real time, matches supply and demand, and helps solve the problem of connecting agricultural product production and sales; and agricultural financial and insurance services, which provide banks and insurance companies with verified production and operation data to assist in accurate credit granting and insurance underwriting and loss assessment, solving the pain point of information asymmetry in agricultural finance.

[0057] The third category is public service applications, which primarily serve farmers and rural collectives, enhancing the capacity of agricultural and rural public services. These include precision agricultural technology services, which accurately deliver personalized agricultural technology knowledge, weather warnings, and prevention and control plans based on farmers' planting conditions, crop growth cycles, and field monitoring data; rural collective asset supervision, which incorporates all data on collective land, funds, assets, and contracts into a data warehouse to monitor operations in real time and prevent the loss of collective assets; and rural living environment improvement monitoring, which integrates multi-source data to build monitoring dashboards, assisting in promoting living environment improvement work and enhancing rural governance capabilities.

[0058] Finally, continuous iteration and optimization of applications and the data warehouse are crucial. Agricultural and rural business is constantly evolving, and data warehouse construction is not a one-time project. A normalized iterative optimization mechanism needs to be established to form a complete business. First, a multi-channel, normalized application feedback collection mechanism should be established to regularly collect feedback from users such as government departments, business personnel, farmers, and enterprises. This feedback should include issues such as inconsistent indicator definitions, lack of analytical scenarios, untimely data updates, insufficient query performance, and new business requirements, forming a clear feedback list. Then, based on the collected feedback, the content of the first four steps should be iteratively optimized in reverse to achieve continuous iteration of the data warehouse. For new business requirements, business research should be conducted again, the indicator system supplemented, and corresponding subject areas and data models added. For indicator definition issues, dimensional models should be optimized, indicator definitions unified, and processing logic adjusted. For data quality issues, cleaning rules should be reviewed, verification rules optimized, and synchronization strategies improved. For performance issues, aggregation logic optimized, scheduling strategies adjusted, and query and interface response speeds improved. At the same time, establish a regular monthly and quarterly operation mechanism for the data warehouse, regularly conduct data asset inventory, data source updates, model optimization, invalid data cleanup, and permission management, regularly evaluate the business value of the data warehouse, generate operation reports, and ensure that the data warehouse continues to adapt to business development and continuously generate business value, avoiding idleness after construction.

[0059] This step involves using DataWorks' data service module to complete interface development and management, using visualization tools to complete report and dashboard development, using the data asset module to complete data asset operation, and finally delivering data service interface documentation, business application integration solutions, visualization report and dashboard design documents, data warehouse iterative optimization and normalized operation mechanisms.

[0060] The foregoing has only described certain exemplary embodiments of the present invention by way of illustration. Undoubtedly, those skilled in the art can modify the described embodiments in various ways without departing from the spirit and scope of the present invention. Therefore, the foregoing drawings and descriptions are illustrative in nature and should not be construed as limiting the scope of protection of the claims of the present invention.

[0061] It should be noted that, in this document, the use of relational terms such as "first" and "second" is merely for distinguishing one entity or operation from another, and does not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes the element.

[0062] It should be understood that in the various embodiments of this application, the order of the above-mentioned processes does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0063] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0064] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for constructing an agricultural and rural data warehouse based on Dataworks, characterized in that: include: The entire process of the three core business lines of agriculture and rural affairs, namely government supervision, production and operation and public services, is sorted out. The multi-source heterogeneous data sources of agriculture and rural areas are inventoried and sensitive data are classified and marked. The business users are connected to break down the feasible analysis scenarios, construct a standardized three-level indicator system for agriculture and rural areas and unify the indicator caliber of the whole warehouse, and define the red line for data compliance management. Based on the best practices of DataWorks data warehouse construction, a five-layer data warehouse architecture adapted to the characteristics of agricultural and rural data is built. The five-layer architecture includes ODS operation data layer, DWD data detail layer, DWS data aggregation layer, DIM common dimension layer, and ADS application data layer. It divides the exclusive business theme domains for agriculture and rural areas, constructs a bus matrix that associates agricultural business processes with common dimensions, and formulates full-process data warehouse design and development specifications. Based on the DataWorks data integration module, the system adapts and securely connects to all types of agricultural and rural data sources. It formulates differentiated offline batch synchronization and real-time synchronization strategies for different agricultural data update frequencies and real-time requirements, configures a synchronization task stability guarantee mechanism, and standardizes and aggregates scattered multi-source heterogeneous agricultural data to the ODS layer. Based on the DataWorks data development module, the system completes the full-link layered data processing from the ODS layer to the ADS layer according to the five-layer data warehouse architecture. It achieves multi-source data fusion and characteristic processing specific to agricultural scenarios, configures a full-link automated task scheduling and operation and maintenance system, and implements full-process data governance capabilities based on DataWorks.

2. The method for constructing an agricultural and rural data warehouse based on Dataworks according to claim 1, characterized in that, Also includes: Based on the DataWorks data service module, a standardized data service API system for agricultural scenarios is built and full lifecycle security management is completed. Through standardized APIs, the value of data warehouse business is realized by connecting with various business scenarios in agriculture and rural areas.

3. The method for constructing an agricultural and rural data warehouse based on Dataworks according to claim 1, characterized in that, Three levels of sensitive data classification and labeling have been completed for agricultural and rural data: The first-level sensitive data includes farmers' personal identity information, bank card information, and other private data, as well as surveying and mapping data such as the red line for arable land and the confidential coordinates of high-standard farmland. The second-level sensitive data includes unpublished agricultural statistics and agricultural enterprise business operation data. The third-level sensitive data consists of publicly available meteorological and industry statistics. The standardized three-level indicator system includes indivisible atomic indicators, derived indicators bound to dimensions and statistical periods, and business composite indicators combining multiple indicators. All indicators have clear and unified statistical logic, data sources, and calculation rules.

4. The method for constructing an agricultural and rural data warehouse based on Dataworks according to claim 1, characterized in that, In the five-layer data warehouse architecture, the ODS layer fully preserves the original agricultural data traces and adapts to the compliant separate storage of confidential spatial data; the DWD layer splits the fact table according to the minimum agricultural business process and completes data cleaning, desensitization, and standardization; the DWS layer completes light aggregation and pre-calculation of general indicators according to the high-frequency analysis dimensions of agriculture; the DIM layer constructs a unified public dimension specifically for agriculture and rural areas; and the ADS layer generates directly usable result data for specific agricultural business scenarios. The unified public dimension specifically for agriculture and rural areas includes the national standard five-level administrative division, unified land parcel coding, standard crop variety codes of the Ministry of Agriculture and Rural Affairs, unified credit codes for agricultural business entities, and agricultural solar term dimensions.

5. The method for constructing an agricultural and rural data warehouse based on Dataworks according to claim 1, characterized in that, The dedicated primary thematic domains for agriculture and rural areas include five major thematic domains: agricultural production, agricultural product circulation and quality safety, rural economy and business entities, arable land and resource environment, and agricultural government supervision. Each primary thematic domain is further divided into corresponding secondary sub-themes. The bus matrix uses the minimum agricultural business process as the row and a unified public dimension as the column, clarifying the relationship between each business process and the public dimension, and achieving a unified dimensional standard across the entire warehouse.

6. The method for constructing an agricultural and rural data warehouse based on Dataworks according to claim 1, characterized in that, The compatible agricultural and rural data sources include relational databases from government and production business systems, IoT devices for soil moisture, pests, and weather in the fields, manual ledger documents, satellite / drone remote sensing geospatial data, and third-party meteorological / price / public opinion API interface data. The differentiated synchronization strategy employs daily incremental and monthly full offline batch synchronization for static low-frequency data, second-level FlinkCDC real-time synchronization for high-frequency real-time IoT monitoring and agricultural product price data, and a compliance strategy of separating attribute data from spatial subject data for the synchronization of classified spatial data.

7. The method for constructing an agricultural and rural data warehouse based on Dataworks according to claim 1, characterized in that, In the end-to-end layered data processing, the ODS layer to the DWD layer completes data deduplication, outlier filtering, null value filling, sensitive data anonymization, dimension standardization mapping, and unique primary key generation; the DWD layer to the DWS layer completes the association of multiple detailed fact tables and the construction of highly reusable summary wide tables; and the DWS layer to the ADS layer completes the calculation of scenario-based indicators and the implementation of complex business logic. All processing tasks are included in version management, supporting rapid rollback in abnormal scenarios.

8. The method for constructing an agricultural and rural data warehouse based on Dataworks according to claim 1, characterized in that, Specialized processing tailored to agricultural scenarios includes: By integrating agricultural records across the entire chain according to plot, crop, and growth cycle, a full-cycle agricultural growth archive is constructed. Through unified plot coding and management entity coding, multi-source fusion of remote sensing growth data, IoT monitoring data, agricultural operation records, and agricultural subsidy data is achieved. For core agricultural indicators, multi-dimensional drilling and processing of five-level administrative divisions (province-city-county-township-village) and industry types is completed, as well as the linkage and correlation processing of GIS spatial analysis results and business data.

9. The method for constructing an agricultural and rural data warehouse based on Dataworks according to claim 1, characterized in that, The automated task scheduling strictly follows the hierarchical order of the data warehouse to configure upstream and downstream dependencies. It configures differentiated scheduling cycles of daily, hourly, monthly, and trigger-based scheduling to meet the real-time needs of different agricultural business scenarios. The full-process data governance includes metadata management and full-link data lineage construction based on DataWorks data maps, field-level fine-grained permission control, data lifecycle management, and standardized data asset catalog construction.

10. The method for constructing an agricultural and rural data warehouse based on Dataworks according to claim 2, characterized in that, The standardized data service API system for agricultural scenarios includes four core interfaces: indicator query API, detailed data query API, real-time monitoring API, and statistical report API. The full lifecycle security management includes API access permission and IP whitelist management, traffic and call frequency management, automatic anonymization of returned data, HTTPS encrypted transmission, full call log retention, and operation auditing.

Images