Agricultural machine-oriented data asset report creation method and device, and storage medium

Abstract

The invention provides an agricultural machinery-oriented data asset report creation method and device and a storage medium, and relates to the technical field of report data processing. The method comprises the following steps: synchronizing multi-source agricultural machine data from a plurality of specified agricultural machine data sources to a temporary database based on a data synchronization tool after setting synchronous configuration parameters; performing data hierarchical treatment on the multi-source agricultural machine data, and synchronizing the treated data to a ClickHouse database to generate a target multi-source agricultural machine data model; constructing a report management framework, and establishing an index relationship between the report management framework and a pre-established database; and synchronizing the target multi-source agricultural machine data model to a report management framework based on the index relationship, and generating a flow to generate an agricultural machine asset report. According to the method, efficient integration and standardized management of multi-source agricultural machine data are achieved, high-performance guarantee configuration and report generation efficiency are achieved by means of ClickHouse, data quality is ensured through layered management, and accurate, efficient and reusable data support is provided.

Description

TECHNICAL FIELD

[0001] The present application mainly relates to the technical field of report data processing, and in particular to a data asset report creation method and device for agricultural machinery and a storage medium. BACKGROUND

[0002] In the process of digital management of the agricultural machinery industry, agricultural machinery related business data is stored in multiple heterogeneous data sources such as a CRM agricultural life cycle business management system, a WMS spare parts inventory management system, and a SES after-sales maintenance spare parts query system, and there is a lack of unified data integration mechanism, which makes it difficult to efficiently converge multi-source agricultural machinery data. At the same time, the existing data processing method does not form a standardized hierarchical governance system, and the quality of raw data is uneven, such as inconsistent formats and abnormal data, which cannot directly support the construction of high-quality data models. In addition, the report management framework for agricultural machinery data assets is missing, and it is difficult to establish an effective association between data models and report generation, which cannot quickly and accurately generate asset reports that adapt to agricultural machinery business scenarios (such as work order management, spare parts inventory analysis, and after-sales maintenance statistics), ultimately leading to the difficulty of converting agricultural machinery data value and the inability to provide efficient and reliable data support for agricultural machinery business decision-making. SUMMARY

[0003] The technical problem to be solved by the present application is to provide a data asset report creation method and device for agricultural machinery and a storage medium to overcome the shortcomings of the prior art.

[0004] The technical solution of the present application to solve the above technical problem is as follows: a data asset report creation method for agricultural machinery, comprising the following steps: Synchronizing multi-source agricultural machinery data from multiple specified agricultural machinery data sources to a temporary database based on a data synchronization tool after setting synchronization configuration parameters; Performing data hierarchical governance on the multi-source agricultural machinery data to obtain target multi-source agricultural machinery data, and synchronizing the target multi-source agricultural machinery data to a pre-built MPP architecture ClickHouse database to generate a target multi-source agricultural machinery data model; Building a report management framework based on data asset configuration requirements, and establishing an index relationship between the report management framework and the MPP architecture ClickHouse database; Synchronizing the target multi-source agricultural machinery data model to the report management framework based on the index relationship, and generating an agricultural machinery asset report through a report generation process of the report management framework.

[0005] Another technical solution of the present application to solve the above technical problem is as follows: a data asset report creation device for agricultural machinery, applied to the data asset report creation method, characterized in that it comprises: Synchronize multi-source agricultural machinery data from multiple specified agricultural machinery data sources to a temporary database based on the data synchronization tool after setting the synchronization configuration parameters; Perform data hierarchical management on the multi-source agricultural machinery data to obtain target multi-source agricultural machinery data, and synchronize the target multi-source agricultural machinery data to a pre-built MPP architecture ClickHouse database to generate a target multi-source agricultural machinery data model; Construct a report management framework based on data asset configuration requirements, and establish an index relationship between the report management framework and the MPP architecture ClickHouse database; Synchronize the target multi-source agricultural machinery data model to the report management framework based on the index relationship, and generate agricultural machinery asset reports through the report generation process of the report management framework.

[0006] Another technical solution of the present application to solve the above technical problems is as follows: a data asset report creation device for agricultural machinery, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the data asset report creation method for agricultural machinery as described above.

[0007] Another technical solution of the present application to solve the above technical problems is as follows: a computer readable storage medium storing a computer program, wherein when the computer program is executed by a processor, the data asset report creation method for agricultural machinery as described above is implemented.

[0008] The present application has the following advantages: by synchronizing data from multiple agricultural machinery data sources and storing them in the ClickHouse database of the MPP architecture after hierarchical management, and by establishing an index relationship with the report management framework and generating agricultural machinery asset reports, the efficient integration and standardized management of multi-source agricultural machinery data are realized, the high performance query capability of ClickHouse ensures the efficiency of data asset configuration and report generation, and the hierarchical management ensures data quality, finally providing precise, efficient and reusable data support for agricultural machinery service resource scheduling, work order management and spare parts deployment decision-making, significantly improving the value conversion efficiency of agricultural machinery data assets and the scientificity of business decisions. BRIEF DESCRIPTION OF DRAWINGS

[0009] Figure 1 A flowchart of the data asset report creation method provided by the embodiment of the present application; Figure 2 A logic diagram of the report generation process provided by the embodiment of the present application; Figure 3 An architectural schematic diagram of the data asset report creation device provided by the embodiment of the present application; Figure 4A block diagram of a data asset report creation device provided in an embodiment of the present invention. Detailed Implementation

[0010] The principles and features of the present invention are described below with reference to the accompanying drawings. The examples given are only for explaining the present invention and are not intended to limit the scope of the present invention.

[0011] Example 1: As Figure 1 As shown in the figure, this embodiment of the invention provides a method for creating data asset reports for agricultural machinery, including the following steps: S1. Based on the set synchronization configuration parameters, the data synchronization tool synchronizes multi-source agricultural machinery data from multiple specified agricultural machinery data sources to a temporary database; S2. Perform data layering and management on the multi-source agricultural machinery data to obtain target multi-source agricultural machinery data, and synchronize the target multi-source agricultural machinery data to the pre-built MPP architecture ClickHouse database to generate the target multi-source agricultural machinery data model; S3. Construct a report management framework based on data asset configuration requirements, and establish an index relationship between the report management framework and the ClickHouse database of the MPP architecture; S4. Based on the index relationship, synchronize the target multi-source agricultural machinery data model to the report management framework, and generate agricultural machinery asset reports through the report generation process of the report management framework.

[0012] In the above embodiments, by synchronizing data from multiple agricultural machinery data sources and storing it in a ClickHouse database with an MPP architecture after hierarchical governance, and then combining it with a report management framework to build index relationships and generate agricultural machinery asset reports, efficient integration and standardized governance of multi-source agricultural machinery data are achieved. The high-performance query capabilities of ClickHouse ensure the efficiency of data asset configuration and report generation, while hierarchical governance ensures data quality. Ultimately, it provides accurate, efficient, and reusable data support for agricultural machinery service resource scheduling, work order management, and spare parts allocation decisions, significantly improving the value conversion efficiency of agricultural machinery data assets and the scientific nature of business decisions.

[0013] Preferably, S1, the data synchronization tool, based on the set synchronization configuration parameters, synchronizes multi-source agricultural machinery data from multiple specified agricultural machinery data sources to a temporary database, including: DataX was used as the data synchronization tool, and the values ​​of the single task concurrency, write rate, synchronization period and storage format of the data synchronization tool were set. Based on the data synchronization tool DataX with the synchronization configuration parameters set, multi-source agricultural machinery data from the CRM agricultural machinery lifecycle business management system, WMS parts inventory management system and SES after-sales maintenance parts query system were synchronized to the temporary hive database.

[0014] It should be understood that the data of the agricultural machine from production to factory, from sales to cross-region operation, to fault repair and other whole life cycle involves different architecture business systems, among which the systems with the largest proportion of business data are CRM, WMS and SES respectively.

[0015] The CRM system focuses on the whole life cycle management of agricultural machines, covering core businesses such as whole machine sales, shipment, and dealer basic information maintenance; at the same time, it undertakes parts order processing, retail management and sales report generation, and is responsible for the reporting and distribution of repair work orders and claim management, and is the information hub of agricultural machine resource scheduling and parts management. The SES system, as a key support for after-sales maintenance, focuses on the accurate query of parts required for fault vehicle repair, and through the network affiliation of maintenance product codes, assemblies and drawing numbers, it can achieve fast retrieval based on vehicle number, product code, parts drawing number and other multi-dimensions. The WMS system focuses on the fine management of parts inventory, and stores the parts inventory data of service vehicles, service personnel, dealers and (sub) center warehouses in real time, and is the terminal execution node of parts scheduling process.

[0016] Although each system is functionally independent but deeply related, it is coordinated and intercommunicated: from the functional level, each system has clear division of labor, seems parallel, but in fact complements each other and forms a closed loop. Different systems jointly build a complete information portrait of the same thing from multiple dimensions, realizing the interconnection of data.

[0017] For different architecture business data, the data synchronization tool selects DataX, sets the number of concurrent tasks to 5, and the database write rate to 2MB / s; based on the above configuration, the business data of each system is synchronized to the big data hive database. During synchronization, based on the business attributes of the original data, it is necessary to judge whether incremental synchronization is needed or new synchronization is needed. If incremental synchronization is needed, it is necessary to determine the conditions for filtering data in DataX based on business, generally based on time to filter, such as synchronizing data of the past seven days, it is necessary to configure the filtering condition update_time>DATE_ADD(NOW(),INTERVAL-7 DAY) after WHERE. The specific filtering condition needs to be added based on the specific business and the specific database type. When storing hive data, analyze ORC\TextFile\SequenceFile\RCFile\Parquet storage formats, based on compression ratio, query performance, usage scenarios and other aspects, decide to use ORC format storage.

[0018] It should be understood that the data governance stage, considering the data volume is relatively large, and with time, data will grow exponentially, based on the core capabilities of hive mass data processing, the business data synchronized will be processed into ODS\DIM\DWD\DWS\ADS five-layer structure data respectively, the data storage format uses ORC storage, and the corresponding data standards and models are created.

[0019] ODS layer: the original data is synchronized to the table in this layer without changing, as a source layer; there are two ways of synchronization: one is new synchronization, the first access or low-frequency update table (such as configuration table); the other is incremental synchronization: high-frequency update business table (such as order table, user table), based on timestamp (create_time), self-incrementing ID, Binlog log (CDC method) to realize incremental extraction, to ensure real-time data.

[0020] DIM layer: store business dimension data (such as user, commodity, region) ①Dimension unification: eliminate cross-business system dimension ambiguity, unified naming; ②Coding standardization: establish a unified coding dictionary for enumerated values (such as vehicle sales status: 1 - inventory, 2 - sold), to ensure consistent meaning throughout the link.

[0021] ③Periodic update: daily / hourly update to ensure synchronization with business system dimensions.

[0022] ④Filter out abnormal data, such as missing key fields, duplicate data, and deleted data.

[0023] DWD layer: clean, convert, and associate ODS layer data to generate business process detail fact table, which is the "core detail layer" of data warehouse.

[0024] ①Remove dirty data: filter out data that does not meet business rules.

[0025] ②Format standardization: unify date format (yyyy-MM-dd HH:mm:ss), numerical precision (retain 2 decimal places).

[0026] ③Field completion: supplement fields by associating DIM layer or other ODS tables.

[0027] DWS layer: based on DWD layer detail data, aggregate by business theme (such as user, commodity, order), provide middle-layer aggregated data, reduce the calculation pressure of ADS layer.

[0028] ①Aggregation logic is reusable: encapsulate calculation logic (address resolution) through public functions (UDF) to avoid repeated development.

[0029] ②Compare the consistency of the summary data and the detailed data ADS layer: oriented to specific business scenarios (such as report, dashboard, business system), providing directly available result data.

[0030] ①Performance optimization: index and partition are established for high-frequency query ADS table, such as using date and product type to set partition for working condition data, to increase query speed; in the vehicle detail table, index is established using vehicle number, etc.

[0031] ②Data timeliness: ensure real-time according to business needs.

[0032] When creating a table in practice, a corresponding model needs to be created first. When creating the model, the business domain and table name need to be defined, and then the table fields need to be added. The table field source data standard is uniformly maintained in the system. For the missing data standard, the data standard needs to be added first, and then the field is added in the model.

[0033] In the above embodiment, by adopting DataX as a data synchronization tool, the number of concurrent tasks, write rate, synchronization period and storage format can be flexibly configured, and the data can be efficiently synchronized from CRM, WMS, SES and other multi-source agricultural machinery systems to the temporary hive database. Through parameter configuration, the synchronization process is fine-tuned and controlled, avoiding overload of the source system or congestion of the target database write, and ensuring the stability and adaptability of multi-source heterogeneous agricultural machinery data synchronization, providing a standardized and complete raw data basis for subsequent data governance.

[0034] Preferably, S2, data layering management is performed on the multi-source agricultural machinery data to obtain target multi-source agricultural machinery data, and the target multi-source agricultural machinery data is synchronized to a pre-built MPP architecture ClickHouse database to generate a target multi-source agricultural machinery data model, including: Based on the MPP architecture, a plurality of data processing layers are constructed, including ODS layer, DIM layer, DWD layer, DWS layer and ADS layer; The ODS layer is used to store all the original multi-source agricultural machinery data based on the new synchronization command or the incremental synchronization command, wherein the new synchronization command is used to store the multi-source agricultural machinery data that is accessed for the first time or updated at a low frequency, and the incremental synchronization command is used to store the multi-source agricultural machinery data that is updated at a high frequency; The DIM layer is used to store the data with agricultural business dimension attributes in the multi-source agricultural machinery data, and sequentially perform unified dimension naming, coding and abnormal data elimination processing on the multi-source agricultural machinery data with agricultural business dimension attributes; The DWD layer is used for sequentially cleaning, format standardization and field completion processing of the original multi-source agricultural machinery data in the ODS layer, and is associated and integrated with the data having agricultural machinery business dimension attributes in the DIM layer, to generate an agricultural machinery business process detail fact table; The DWS layer is used for summarizing based on the agricultural machinery business process detail fact table and according to different agricultural machinery business themes (such as work order, spare part inventory, operation progress), and encapsulating the summary calculation process of the agricultural machinery business theme through a public function UDF to obtain target multi-source agricultural machinery data, and comparing the consistency of the target multi-source agricultural machinery data and the agricultural machinery data in the agricultural machinery business process detail fact table; The ADS layer is used for establishing a field index of the target multi-source agricultural machinery data associated with the agricultural machinery business scenario; The target multi-source agricultural machinery data is synchronized to a ClickHouse database of an MPP architecture, a corresponding target multi-source agricultural machinery data model is generated in the ClickHouse database, and a corresponding engine is configured according to the data size of the target multi-source agricultural machinery data model.

[0035] It should be understood that in the data governance stage, considering that the data volume is relatively large and will increase exponentially over time, based on the core capability of hive mass data processing, the synchronized business data is processed into ODS\DIM\DWD\DWS\ADS five-layer structure data respectively, the data storage format uses ORC storage, and the corresponding data standards and models are created.

[0036] The ODS layer: the original data is synchronized to the table in this layer as it is, as a source layer; there are two synchronization methods: one is new synchronization, the first access or low-frequency update table (such as configuration table); the other is incremental synchronization: high-frequency update business table (such as order table, user table), based on timestamp (create_time), self-incrementing ID, Binlog log (CDC method) to realize incremental extraction, to ensure real-time data.

[0037] The DIM layer: stores business dimension data (such as user, commodity, region) ①Dimension unification: eliminate dimension ambiguity across business systems, and unify naming; ②Coding standardization: establish a unified coding dictionary for enumerated values (such as vehicle sales status: 1 - inventory, 2 - sold), to ensure consistent meaning throughout the link.

[0038] ③Periodic update: daily / hourly update to ensure synchronization with business system dimensions.

[0039] ④Filter out abnormal data, such as missing key fields, duplicate data, and deleted data.

[0040] DWD layer: Cleansing, transforming, and correlating ODS layer data to generate business process detail fact tables, which are the "core detail layer" of the data warehouse.

[0041] ① Remove dirty data: Filter data that does not meet business rules.

[0042] ② Format standardization: Standardize date format (yyyy-MM-dd HH:mm:ss) and numerical precision (2 decimal places).

[0043] ③ Field completion: Supplement fields through association with DIM layer or other ODS tables.

[0044] DWS layer: Based on DWD layer detail data, aggregate by business theme (such as user, product, order), provide middle-level aggregated data, reduce the calculation pressure of ADS layer.

[0045] ① Reusable aggregation logic: Encapsulate calculation logic (address resolution) through common functions (UDF) to avoid repeated development.

[0046] ② Compare the consistency of summary data and detail data ADS layer: Face specific business scenarios (such as reports, dashboards, business systems), provide directly usable result data.

[0047] ① Performance optimization: Index high-frequency query ADS tables, set partitions, such as using date and product type to set partitions for working condition data to increase query speed; in the vehicle detail table, use vehicle number to establish index.

[0048] ② Data timeliness: Ensure real-time according to business needs.

[0049] When creating tables in practice, you need to create corresponding models first. When creating models, you need to define business domains and table names, then add table fields, and the source data standard of the table fields. Maintain all data standards uniformly in the system. For missing data standards, add data standards first, then add fields in the model.

[0050] Synchronize well-governed data to ClickHouse, an MPP large-scale parallel processing architecture, to support real-time queries for data asset systems and report systems. ClickHouse not only supports calculation but also supports storage, with response speed in milliseconds for data access of millions, simple to use, fast and efficient.

[0051] Based on the size of the report data, the execution engine of Clickhouse needs to be set according to the situation. When the data volume is small, engine = Log can meet the requirements. When the data volume is large, if Clickhouse does not set the engine, it will also cause the query speed to be too slow. Therefore, engine = ReplacingMergeTree() needs to be considered. This engine can automatically remove duplicate data based on the index field setting and automatically add an index for data that repeats. For example, the daily operation table of agricultural machinery, the data of each day will have repeated queries, and the last data of each day needs to be retained. Therefore, the use of this engine is very suitable. The specific configuration is as follows: engine = ReplacingMergeTree() PARTITION BY substring(work_day,1,7) PRIMARY KEY (work_day,serial_no) ORDER BY (work_day,serial_no) Store by month partition, date, and agricultural machinery number as index to ensure data uniqueness.

[0052] In the above embodiment, by constructing the five-layer data processing architecture of ODS, DIM, DWD, DWS, and ADS, layered management is carried out, which can not only accurately store raw data according to the data update frequency (first time / low frequency using new synchronization, high frequency using incremental synchronization) through ODS layer, but also unify business dimension specification through DIM layer, complete data cleaning and integration to generate detailed fact table through DWD layer, summarize and calculate according to business theme through DWS layer, and verify data consistency, and establish business scene correlation index through ADS layer, realizing the orderly transformation of multi-source agricultural machinery data from raw data to standardized target data; At the same time, the target data is synchronized to the ClickHouse database of MPP architecture and the engine is configured as needed, which not only guarantees data quality and business adaptability relying on layered management, but also lays a high-efficiency and stable data model foundation for subsequent data application relying on the high-performance storage and computing power of ClickHouse, effectively balancing the standardization of data governance and the efficiency of data call.

[0053] Preferably, a report management framework is constructed based on data asset configuration requirements, including: The data asset configuration requirements include data asset source configuration requirements, preview data configuration requirements, report creation configuration requirements, and asset release and revocation configuration requirements. A data asset management layer of the report management framework is constructed based on the data asset source configuration requirement, and is configured with a corresponding directory, asset name, unique code, data source, and data asset source of a model, and is configured with an update parameter of the target multi-source agricultural machinery data model; A preview data management layer of the report management framework is constructed based on the preview data configuration requirement, and is configured with a data standard field to be displayed from the target multi-source agricultural machinery data model, and is configured with a display order of each data standard field, and is configured with a filtering condition based on the set data standard, and is configured with a query mode option; A report creation layer of the report management framework is constructed based on the report creation configuration requirement, and is configured with report data including a tree structure and a sheet page name, and is configured with a template format whether supporting a merged cell; A report management layer of the report management framework is constructed based on the asset publishing and revoking configuration requirement, and is configured with a publishing or revoking operation on a created asset, and is configured with a historical agricultural machinery asset report.

[0054] It should be understood that the creation of a data asset requires the creation of a directory first. The addition of a directory is to find the corresponding data asset in a large number of data assets. The directory is equivalent to a dictionary index, which can guide the location of the data asset.

[0055] The creation of a data asset requires operation based on the completion of the creation of a data model.

[0056] The steps of creating a data asset are as follows: 1) Data asset source configuration a. Directory determination (set the business directory of the data asset, which can quickly find the corresponding data asset); b. Data asset name (the name requires a clear expression of the content of the data asset); c. Data asset code (the code is unique, which is a lock key for the connection between the data asset system and the report system); d. Data source (the data source of the data asset may be a multi-party data integration, such as CRM, MES, and Internet of Vehicles, etc. The data source of the asset needs to be input); e. Data asset source (the data source of the data asset, most of the assets are stored in the ClickHouse database, the data asset source needs to be selected first, and then the model is followed); f. Model list (Data models created when data governance is performed in hive before selection, data models need to be created before this step to be able to select here and preview data on the page later); g. Update frequency (Data update frequency corresponding to the data model, manual maintenance); h. Data filtering (There may be the same data type but different types of data in a model corresponding table, such as water rice machine and wheat machine in the vehicle information table, but data assets are used by specific business personnel, and users may be wheat machine or water rice machine business personnel, because the filtering condition needs to be used, data filtering is performed, and the filtering SQL condition is manually added here, which has nothing to do with the model); i. Order number (The order of data assets display); j. Asset description (A simple description of the content data of the data asset).

[0057] 2) Preview data configuration a. List field (Data model selected in the first step, data model has corresponding data standard, this place selects the data standard field to be displayed in the model, and the order of data standard display can be selected); b. Query condition (In order to quickly filter the data you want to see, you need to add a filtering condition, you need to select the data model first, then you can select this item, the filtering condition is also selected through the data standard in the data model, supports input box, drop-down box, date selector, date and time selector, etc. Format input box; Query method has fuzzy query, equal query, range query and other query methods.

[0058] 3) Report generation configuration a. Whether to generate a report (Yes: there will be subsequent configuration settings; No: no subsequent configuration and no report will be generated); b. Report update cycle (Report generation frequency); c. Report template upload and download (Report templates need to be generated in advance); d. Report data (Specific report data information generated: includes whether to use tree structure, sheet name (write the data to the sheet page, if empty, write according to the first data); e. Calculation statistics (Support for corresponding data when merging cells), regional statistics (merge cells to calculate regional data), and executed scripts.

[0059] The report generation configuration page supports writing multiple sheet page data, and needs to create several report data configurations for several sheet pages.

[0060] In order to better support the display of data report, the report template has two formats: one is the normal data format, which does not support merged cells and the like, and the template style is as follows{.xxx}, wherein xxx is the field code returned by the execution in the script, such as the field code of id, and the writing format in the template is{.id}; the other is the format supporting merged cells: merged cells, which is actually the data display of tree structure, and a row of data format has start field, middle field and end field, respectively, and the formats are as follows{{fe: t.xxx, t.childrenKey.aaa, t.childrenKey.bbb}}.

[0061] 3) Data asset publishing and revoking The created data asset supports publishing and revoking operations, supports historical version viewing, and generated report viewing.

[0062] In the above embodiment, around the four types of configuration requirements of data asset source, preview, creation, publishing and revoking, a report management framework including data asset management layer, preview data management layer, report creation layer and report management layer is constructed, which can not only clearly define the basic information and update rules of model association through the data asset management layer, flexibly filter and display fields and configure query conditions through the preview data management layer, but also adapt to different data formats and template requirements through the report creation layer, realize asset operation and historical tracking through the report management layer, form a standardized configuration system covering the whole life cycle of the report, which not only guarantees the flexibility and adaptability of report management, but also improves the standardization degree and operation traceability of data asset configuration, and provides a clear and controllable framework support for efficient generation and management of agricultural machinery asset report.

[0063] Preferably, the target multi-source agricultural machinery data model is synchronized into the report management framework based on the index relationship, and an agricultural machinery asset report is generated through a report generation process of the report management framework, including: After the data asset system completes the related configuration saving of the target multi-source agricultural machinery data model, the configuration information is synchronized into the report management framework based on the index relationship; The report management framework generates a report configuration with version identification based on the asset code corresponding to the target multi-source agricultural machinery data model, the report configuration includes template position information, execution script and callback function information, and judges whether the task corresponding to the to-be-generated agricultural machinery asset report exists, if yes, updates the configuration information and version information of the task, if not, creates a new report task; Based on the template position information in the report configuration, check whether there is a report template matching the current execution task timestamp locally, if not, download the template from the specified location and store it locally, if it already exists, directly call it; Optimize the execution script in the report configuration, and convert the dynamic time format contained in the execution script into specific time information; Based on the processed execution script, extract the target multi-source agricultural machinery data model from the MPP architecture ClickHouse database. During the extraction process, first obtain the total number of data through the script. When the data volume reaches the set threshold, use the paging query method to obtain the data in batches. According to the format requirements of the report template, write the obtained target multi-source agricultural machinery data into a table file. If the data volume reaches the set threshold, write it in batches until all data is written, and generate the agricultural machinery asset report.

[0064] As shown in Figure 2 After the data asset system is saved, it will request the interface of the report service to synchronize the configuration of the report task to be generated to the report database. The report service will generate a data report configuration with version based on the data asset code, including template location, execution script, callback function, etc. It will also determine whether the report task exists. If it exists, it will update the configuration information and version information of the task. If it does not exist, it will create a task for the report and wait for the task scheduler to schedule the task.

[0065] When the report task is started by manual triggering or timing triggering, the service will extract the configuration information of the corresponding report from the report service database based on the data asset code. These configuration information covers data sources, execution scripts, whether to generate tree structure, callback functions, etc. After obtaining the configuration information, the system will execute the sub-tasks in the following steps to complete the report generation: 1) Template acquisition and management: Extract the template details according to the configuration information. First, check whether there is a template matching the execution task timestamp locally. If the template already exists, there is no need to download it again; if it does not exist, download it according to the template position information and store it locally; 2) Execution script conversion: Optimize the execution script, focusing on converting the dynamic time format (such as the current day, month, year, etc.) that may be contained in the script into specific time information to ensure that the script can be accurately executed; 3) Data acquisition and paging processing: Extract data from the corresponding data source based on the converted script. During the process, first obtain the total number of data through the script. When the data volume exceeds 500,000, the system will use the paging query method to obtain the data in batches. Based on the total number of 500,000, the number of times needed to be written will be obtained to ensure the efficiency and stability of data acquisition; 4) Data Writing and Validation: Write the acquired data into Excel according to the template information. If the data volume exceeds 500,000 rows, it needs to be written in stages, writing 500,000 rows at a time until all data is written to the file. 5) Excel Upload: After the Excel file is created, the system will upload it to the OSS database and return the corresponding storage path information; 6) Template cleanup: Delete downloaded templates locally to free up storage space; 7) Information Synchronization and Callback: Write the relevant information of the report into the report service database and call the preset callback function to send the detailed information of the generated report back to the data asset service.

[0066] At this point, the report generation process is complete.

[0067] In the above embodiments, the target multi-source agricultural machinery data model is synchronized to the report management framework through index relationships. In the report generation process, a report configuration with version identifier is first generated based on the asset code, and task judgment and updates are performed. Then, local template compatibility is ensured through template management. Dynamic time format is processed through script optimization. Combined with pagination query and batch writing mechanism, data is efficiently extracted and written. The consistency and traceability of report configuration are guaranteed by version management and task update mechanism. The efficiency of data extraction and writing is improved by template adaptation, script optimization and batch processing. At the same time, with the linkage with the MPP architecture ClickHouse database, the standardization, efficiency and flexible adaptation of agricultural machinery asset report generation are realized, providing reliable support for the rapid application of data assets.

[0068] Example 2: As Figure 4 As shown, this embodiment of the invention also provides a data asset report creation device for agricultural machinery, applied to the aforementioned data asset report creation method, including: The data synchronization tool, based on the configured synchronization parameters, synchronizes multi-source agricultural machinery data from multiple specified agricultural machinery data sources to a temporary database; The multi-source agricultural machinery data is subjected to data layering and management to obtain target multi-source agricultural machinery data, and the target multi-source agricultural machinery data is synchronized to the pre-built MPP architecture ClickHouse database to generate the target multi-source agricultural machinery data model. A report management framework is constructed based on data asset configuration requirements, and an index relationship is established between the report management framework and the ClickHouse database with MPP architecture. Based on the index relationship, the target multi-source agricultural machinery data model is synchronized to the report management framework, and an agricultural machinery asset report is generated through the report generation process of the report management framework.

[0069] Specifically, as shown in Figure 3 The architecture of the device can include an OSS storage system, a data asset system, a report generation system, and a ClickHouse database.

[0070] OSS storage system: as a top-level storage module, it interacts with the data asset system and the report generation system below bidirectionally, and is used for storing report templates, generated reports, and other file resources.

[0071] Data asset system: with 7 core functions, including data asset coding, report update frequency setting, data source information management, script execution configuration, template information management, online preview, and download report; it downwardly interfaces with the ClickHouse database to obtain data support, and simultaneously outputs "configuration information" to the report generation system on the right side and receives the feedback "report information".

[0072] Report generation system: mainly responsible for "generating reports" and "uploading reports"; it relies on the ClickHouse database to obtain data, simultaneously completes report generation based on the "configuration information" obtained from the data asset system, and returns "report information" to the data asset system, and also interacts with the OSS storage system to realize report storage and management.

[0073] ClickHouse database: as a bottom-layer data support module, it provides storage and query services for agricultural machinery data for the data asset system and the report generation system, and is the data foundation of the entire architecture.

[0074] Preferably, the multi-source agricultural machinery data is synchronized from multiple specified agricultural machinery data sources to a temporary database based on a data synchronization tool after setting synchronization configuration parameters, including: DataX is adopted as the data synchronization tool, and the values of the single task concurrency number of the data synchronization tool, the write rate, the synchronization period, and the storage format are set, and the multi-source agricultural machinery data is synchronized from the CRM agricultural machinery life cycle business management system, the WMS spare parts inventory management system, and the SES after-sales maintenance spare parts query system to the temporary hive database based on the data synchronization tool DataX after setting the synchronization configuration parameters.

[0075] Embodiment 3: The embodiment of the present application also provides a data asset report creation device for agricultural machinery, characterized in that it comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, and when the processor executes the computer program, the data asset report creation method for agricultural machinery is realized.

[0076] Embodiment 4: The embodiments of the present application also provide a computer readable storage medium, which stores a computer program, and the computer program is characterized in that when the computer program is executed by a processor, the computer program realizes the agricultural machinery-oriented data asset report creation method.

[0077] It should be noted that the relative terms, such as first and second, and the like are used herein only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between such entities or operations. Moreover, the terms "comprising", "including", or any other variant thereof are intended to cover non-exclusive inclusion, so that processes, methods, articles, or equipment including a series of elements not only include those elements, but also include other elements not explicitly listed, or inherent to such processes, methods, articles, or equipment.

[0078] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the above-described device and unit can refer to the corresponding process in the foregoing method embodiments, which will not be described here.

[0079] In several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of units is only a logical function division, and actual implementation can have another division manner, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed.

[0080] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they can be located in one place, or distributed on a plurality of network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiments of the present application.

[0081] In addition, each functional unit in each embodiment of the present application can be integrated into a processing unit, or each unit can exist physically, or two or more units can be integrated into one unit. The above integrated unit can be realized in the form of hardware or in the form of a software functional unit.

[0082] The above description is only the preferred embodiment of the present application, and is not intended to limit the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A method for creating data asset reports for agricultural machinery, characterized in that, Includes the following steps: The data synchronization tool, based on the configured synchronization parameters, synchronizes multi-source agricultural machinery data from multiple specified agricultural machinery data sources to a temporary database; The multi-source agricultural machinery data is subjected to data layering and management to obtain target multi-source agricultural machinery data, and the target multi-source agricultural machinery data is synchronized to the pre-built MPP architecture ClickHouse database to generate the target multi-source agricultural machinery data model. A report management framework is constructed based on data asset configuration requirements, and an index relationship is established between the report management framework and the ClickHouse database with MPP architecture. Based on the index relationship, the target multi-source agricultural machinery data model is synchronized to the report management framework, and an agricultural machinery asset report is generated through the report generation process of the report management framework.

2. The data asset report creation method according to claim 1, characterized in that, Based on the configured synchronization parameters, the data synchronization tool synchronizes multi-source agricultural machinery data from multiple specified agricultural machinery data sources to a temporary database, including: DataX was used as the data synchronization tool, and the values ​​of the single task concurrency, write rate, synchronization period and storage format of the data synchronization tool were set. Based on the data synchronization tool DataX with the synchronization configuration parameters set, multi-source agricultural machinery data from the CRM agricultural machinery lifecycle business management system, WMS parts inventory management system and SES after-sales maintenance parts query system were synchronized to the temporary hive database.

3. The data asset report creation method according to claim 1, characterized in that, The multi-source agricultural machinery data is subjected to data layering and governance to obtain target multi-source agricultural machinery data. This target multi-source agricultural machinery data is then synchronized to a pre-built MPP architecture ClickHouse database to generate a target multi-source agricultural machinery data model, including: Multiple data processing layers are constructed based on the MPP architecture, including ODS layer, DIM layer, DWD layer, DWS layer and ADS layer; The ODS layer is used to store all the original multi-source agricultural machinery data that has been synchronized based on the new synchronization command or the incremental synchronization command. The new synchronization command is used to store the multi-source agricultural machinery data that is accessed for the first time or updated in a low frequency, and the incremental synchronization command is used to store the multi-source agricultural machinery data that is updated in a high frequency. The DIM layer is used to store data with agricultural machinery business dimension attributes in multi-source agricultural machinery data, and to perform unified dimension naming, encoding and abnormal data removal processing on multi-source agricultural machinery data with agricultural machinery business dimension attributes in sequence. The DWD layer is used to clean, standardize, and complete the original multi-source agricultural machinery data in the ODS layer, and to associate and integrate it with the data in the DIM layer that has agricultural machinery business dimension attributes to generate a detailed fact table of agricultural machinery business processes. The DWS layer is used to summarize the agricultural machinery business process detailed fact table according to different agricultural machinery business themes, and to encapsulate the summary calculation process of agricultural machinery business themes through public function UDF to obtain target multi-source agricultural machinery data, and to compare the consistency between the target multi-source agricultural machinery data and the agricultural machinery data in the agricultural machinery business process detailed fact table. The ADS hierarchy is used to establish field indexes that associate target multi-source agricultural machinery data with agricultural machinery business scenarios; The target multi-source agricultural machinery data is synchronized to the ClickHouse database with MPP architecture. The corresponding target multi-source agricultural machinery data model is generated in the ClickHouse database, and the corresponding engine is configured according to the data size of the target multi-source agricultural machinery data model.

4. The data asset report creation method according to claim 3, characterized in that, A report management framework is built based on data asset configuration requirements, including: The data asset configuration requirements include data asset source configuration requirements, preview data configuration requirements, report creation configuration requirements, and asset publication and revocation configuration requirements. The data asset management layer of the report management framework is constructed based on the data asset source configuration requirements. The data asset management layer is used to configure the directory, asset name, unique code, data source and data asset source corresponding to the model, and to configure the update parameters of the target multi-source agricultural machinery data model. The preview data management layer is built based on the preview data configuration requirements to construct the report management framework. The preview data management layer is used to select the data standard fields to be displayed from the target multi-source agricultural machinery data model, determine the display order of each data standard field, add filtering conditions based on the set data standards, and set query method options. The report creation layer of the report management framework is constructed based on the aforementioned report creation configuration requirements. The report creation layer is used to configure report data, which includes a tree structure and sheet page names, and configures whether to support template formats for merging cells. The report management layer of the report management framework is constructed based on the asset release and cancellation configuration requirements. The report management layer is used to perform release or cancellation operations on the created assets and display the historically generated agricultural machinery asset reports.

5. The data asset report creation method according to claim 3, characterized in that, Based on the index relationship, the target multi-source agricultural machinery data model is synchronized to the report management framework, and an agricultural machinery asset report is generated through the report generation process of the report management framework, including: After the data asset system completes the saving of the relevant configurations for the target multi-source agricultural machinery data model, it synchronizes the configuration information to the report management framework based on the index relationship. The report management framework generates report configurations with version identifiers based on the asset codes corresponding to the target multi-source agricultural machinery data model. The report configurations include template location information, execution scripts and callback function information, and determine whether the task corresponding to the agricultural machinery asset report to be generated already exists. If it exists, the configuration information and version information of the task are updated; if it does not exist, a new report task is created. Based on the template location information in the report configuration, check if there is a report template that matches the timestamp of the currently executed task locally. If it does not exist, download the template from the specified location and store it locally. If it already exists, call it directly. The execution script in the report configuration is optimized to convert the dynamic time format contained in the execution script into specific time information; Based on the processed execution script, the target multi-source agricultural machinery data model is extracted from the MPP architecture ClickHouse database. During the extraction process, the total number of data entries is first obtained through the script. When the data volume reaches the set threshold, the data is obtained in batches using a paginated query method. According to the format requirements of the report template, the acquired target multi-source agricultural machinery data is written into a table file. If the data volume reaches the set threshold, it is written in batches until all data is written and an agricultural machinery asset report is generated.

6. The data asset report creation method according to claim 5, characterized in that, After generating the agricultural machinery asset report, the following steps are also included: Upload the agricultural machinery asset report to the OSS database and record the storage path in the OSS database; Once the upload is complete, the agricultural machinery asset report will be deleted locally.

7. A data asset report creation device for agricultural machinery, applied to the data asset report creation method described in claims 1 to 6, characterized in that, include: The data synchronization module is used to synchronize multi-source agricultural machinery data from multiple specified agricultural machinery data sources to a temporary database based on the data synchronization tool after setting the synchronization configuration parameters; The data layering and governance module is used to perform data layering and governance on the multi-source agricultural machinery data to obtain target multi-source agricultural machinery data, and to synchronize the target multi-source agricultural machinery data to the pre-built MPP architecture ClickHouse database to generate the target multi-source agricultural machinery data model. The report framework creation module is used to build a report management framework based on data asset configuration requirements and to establish the index relationship between the report management framework and the ClickHouse database with MPP architecture. The report generation module is used to synchronize the target multi-source agricultural machinery data model to the report management framework based on the index relationship, and generate agricultural machinery asset reports through the report generation process of the report management framework.

8. The data asset report creation device according to claim 6, characterized in that, Based on the configured synchronization parameters, the data synchronization tool synchronizes multi-source agricultural machinery data from multiple specified agricultural machinery data sources to a temporary database, including: DataX was used as the data synchronization tool, and the values ​​of the single task concurrency, write rate, synchronization period and storage format of the data synchronization tool were set. Based on the data synchronization tool DataX with the synchronization configuration parameters set, multi-source agricultural machinery data from the CRM agricultural machinery lifecycle business management system, WMS parts inventory management system and SES after-sales maintenance parts query system were synchronized to the temporary hive database.

9. A data asset report creation device for agricultural machinery, characterized in that, The system includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the method for creating data asset reports for agricultural machinery as described in any one of claims 1 to 6.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the method for creating data asset reports for agricultural machinery as described in any one of claims 1 to 6.

Images