Agricultural big data management method, device, electronic device and storage medium

Abstract

The present invention provides a management method, device, electronic device and storage medium for agricultural big data, and relates to the technical field of agricultural big data. The method includes: collecting agricultural data, determining the agricultural theme of the agricultural data; determining the first target indicator set corresponding to the agricultural theme based on the theme-indicator set mapping relationship; classifying the agricultural data according to the indicators based on each indicator in the first target indicator set to obtain a plurality of classified data; and storing the plurality of classified data according to the indicators. The present invention can quickly and accurately retrieve the corresponding indicators according to the agricultural theme, and then can quickly and accurately retrieve the corresponding agricultural data according to the indicators, that is, improve the retrieval efficiency and retrieval accuracy of agricultural data, and finely divide the agricultural data, reduce the management confusion of agricultural big data, and improve the management accuracy of agricultural big data.

Description

Technical Field

[0001] The present invention relates to the technical field of agricultural big data, and in particular to a management method, device, electronic equipment and storage medium for agricultural big data. Background Art

[0002] With the rapid development of big data technology, the application scope of big data is becoming more and more extensive. In the field of agriculture, the application of agricultural big data is also becoming more and more extensive. Agricultural data is usually managed through the data middle platform, that is, agricultural data is stored in the data middle platform, and agricultural data is mined, analyzed and processed according to custom business logic.

[0003] At present, various agricultural data are directly stored. However, due to the need to store a variety of agricultural data, directly storing agricultural data will lead to confusion in the subsequent management of agricultural big data, and make it impossible to quickly find the corresponding agricultural data based on business needs. Summary of the invention

[0004] The present invention provides a management method, device, electronic device and storage medium for agricultural big data, which are used to solve the defects of chaotic agricultural big data management and low data retrieval efficiency in the prior art.

[0005] The present invention provides a method for managing agricultural big data, comprising:

[0006] collecting agricultural data and determining agricultural themes of the agricultural data;

[0007] Based on the theme-indicator set mapping relationship, determine the first target indicator set corresponding to the agricultural theme;

[0008] Based on each indicator in the first target indicator set, the agricultural data is classified according to the indicator to obtain a plurality of classified data;

[0009] The plurality of classified data are classified and stored according to the indicators.

[0010] According to a method for managing agricultural big data provided by the present invention, the plurality of classified data are classified and stored according to indicators, and then the method further includes:

[0011] receiving a topic selection instruction, and determining an agricultural topic to be searched based on the topic selection instruction;

[0012] Based on the topic-indicator set mapping relationship, determine the second target indicator set corresponding to the agricultural topic to be retrieved;

[0013] Display each indicator in the second target indicator set;

[0014] receiving an indicator selection instruction, and determining an indicator to be retrieved based on the indicator selection instruction;

[0015] Based on the index to be retrieved, the stored target agricultural data is retrieved.

[0016] A method for managing agricultural big data provided by the present invention further includes:

[0017] Collect target data of target data sources storing agricultural data;

[0018] Extracting keywords from the target data to obtain keywords;

[0019] Based on the keyword-field mapping relationship, determine the table field of the Hive table corresponding to the keyword;

[0020] Based on the table fields, the target data is stored.

[0021] A method for managing agricultural big data provided by the present invention further includes:

[0022] Get information configuration instructions acting on the configuration information input interface;

[0023] Based on the information configuration instruction, determining at least two configuration information;

[0024] Based on the at least two pieces of configuration information, a data synchronization code script is constructed, where the data synchronization code script is used to import data or export data;

[0025] Among them, the at least two configuration information include the resource name of the data resource to be synchronized and the data source type of the external data source, the external data source is the data source of the original storage location of the data to be imported or the data source of the target storage location of the data to be exported; the resource name is used as the script name of the data synchronization code script.

[0026] A method for managing agricultural big data provided by the present invention further includes:

[0027] Based on the association relationships between the multiple data tables and the primary key fields of each data table in the multiple data tables, a data table lineage map is generated.

[0028] A method for managing agricultural big data provided by the present invention further includes:

[0029] receiving a theme setting instruction, and determining a target data table to be set and a theme to be set based on the theme setting instruction;

[0030] Setting the subject of the target data table as the subject to be set;

[0031] Receiving a data table search instruction, and determining a subject to be searched based on the data table search instruction;

[0032] Based on the subject to be searched, a data table with the subject being the subject to be searched is obtained.

[0033] A method for managing agricultural big data provided by the present invention further includes:

[0034] Obtain user management instructions for agricultural big data, and determine the functional authority and data authority of the user corresponding to the user management instructions;

[0035] Based on the function permission and the data permission, determining whether the operation indicated by the user management instruction can be implemented;

[0036] The functional permission is used to determine whether the function can be used, and the data permission is used to determine the scope of use of the permission.

[0037] The present invention also provides a management device for agricultural big data, comprising:

[0038] A theme determination module, used for collecting agricultural data and determining the agricultural theme of the agricultural data;

[0039] An indicator determination module, used to determine a first target indicator set corresponding to the agricultural theme based on a theme-indicator set mapping relationship;

[0040] A data classification module, used for classifying the agricultural data according to the indicators in the first target indicator set to obtain a plurality of classified data;

[0041] The classification storage module is used to classify and store the multiple classification data according to indicators.

[0042] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, a method for managing agricultural big data as described in any one of the above is implemented.

[0043] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the agricultural big data management method as described in any one of the above.

[0044] The agricultural big data management method, device, electronic device and storage medium provided by the present invention collect agricultural data, determine the agricultural theme of the agricultural data, and determine the first target indicator set corresponding to the agricultural theme based on the theme-indicator set mapping relationship, so that the agricultural data can be finely divided according to the agricultural theme, which is convenient for quickly and accurately retrieving the agricultural data according to the agricultural theme, that is, improving the retrieval efficiency and retrieval accuracy of the agricultural data, and finely dividing the agricultural data, reducing the management chaos of agricultural big data, and improving the management accuracy of agricultural big data; based on each indicator in the first target indicator set, the agricultural data is classified according to the indicator to obtain a plurality of classified data, so that the agricultural data can be finely divided according to the indicator, which is convenient for quickly and accurately retrieving the agricultural data according to the indicator, that is, improving the retrieval efficiency and retrieval accuracy of the agricultural data, and finely dividing the agricultural data, reducing the management chaos of agricultural big data, and improving the management accuracy of agricultural big data; the plurality of classified data are classified and stored according to the indicators, so that the corresponding indicators can be quickly and accurately retrieved according to the agricultural theme, and then the corresponding agricultural data can be quickly and accurately retrieved according to the indicators, that is, improving the retrieval efficiency and retrieval accuracy of the agricultural data, and finely dividing the agricultural data, reducing the management chaos of agricultural big data, and improving the management accuracy of agricultural big data. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0046] Figure 1 A schematic diagram of the process of the agricultural big data management method provided by the present invention;

[0047] Figure 2 A schematic diagram of the structure of the agricultural big data management device provided by the present invention;

[0048] Figure 3 This is a schematic structural diagram of an electronic device provided by the present invention. DETAILED DESCRIPTION

[0049] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0050] With the rapid development of big data technology, the application scope of big data is becoming more and more extensive. In the field of agriculture, the application of agricultural big data is also becoming more and more extensive. Agricultural data is usually managed through the data middle platform, that is, agricultural data is stored in the data middle platform, and agricultural data is mined, analyzed and processed according to custom business logic.

[0051] At present, various agricultural data are directly stored. However, due to the need to store a variety of agricultural data, directly storing agricultural data will lead to confusion in the subsequent management of agricultural big data, and make it impossible to quickly find the corresponding agricultural data based on business needs.

[0052] In view of the above problems, the present invention proposes the following embodiments. Figure 1 A schematic diagram of the process flow of the agricultural big data management method provided by the present invention is as follows: Figure 1 As shown, the management method of agricultural big data includes:

[0053] Step 110, collecting agricultural data and determining the agricultural theme of the agricultural data.

[0054] Here, agricultural data refers to data related to agriculture. The agricultural subject can be at least one of the following: agricultural resources and environment data, farmland data, agricultural scientific research data, business management data, agricultural product data, agricultural business entity data, agricultural production data, and industrial ecology data.

[0055] Step 120, based on the theme-indicator set mapping relationship, determine the first target indicator set corresponding to the agricultural theme.

[0056] Here, the topic-indicator set mapping relationship is used to characterize the one-to-one correspondence between multiple agricultural topics and multiple indicator sets. In other words, different agricultural topics correspond to different indicator sets, that is, different agricultural topics correspond to different indicators. Any indicator set includes at least one indicator, such as agricultural indicators such as accumulated temperature and accumulated rainfall.

[0057] Based on this, agricultural data can be finely divided according to agricultural themes, which facilitates the rapid retrieval of agricultural data according to agricultural themes, that is, improves the retrieval efficiency of agricultural data; and finely divides agricultural data to reduce the management chaos of agricultural big data and improve the management accuracy of agricultural big data.

[0058] Correspondingly, a subject selection instruction is received, and the agricultural subject to be retrieved is determined based on the subject selection instruction. Based on the agricultural subject to be retrieved, agricultural data corresponding to the agricultural subject to be retrieved is retrieved, thereby facilitating rapid retrieval of agricultural data according to the agricultural subject, i.e., improving the retrieval efficiency of agricultural data.

[0059] Step 130, based on each indicator in the first target indicator set, classify the agricultural data according to the indicator to obtain a plurality of classified data.

[0060] It should be understood that one indicator corresponds to one classified data, and the multiple classified data are combined to obtain the agricultural data. And the number of indicators in the first target indicator set is greater than the number of multiple classified data. For example, the first target indicator set includes 10 indicators, but the agricultural data only involves 8 of them. Based on this, 8 classified data corresponding to the 8 indicators are obtained.

[0061] Based on this, agricultural data can be finely divided according to indicators, which is convenient for rapid retrieval of agricultural data according to indicators, that is, improving the retrieval efficiency of agricultural data; and finely dividing agricultural data can reduce the management chaos of agricultural big data and improve the management accuracy of agricultural big data.

[0062] Correspondingly, an indicator selection instruction is received, the indicator to be retrieved is determined based on the indicator selection instruction, and based on the indicator to be retrieved, the agricultural data corresponding to the indicator to be retrieved is retrieved, thereby facilitating the rapid retrieval of agricultural data according to the indicator, that is, improving the retrieval efficiency of agricultural data.

[0063] Step 140: Classify and store the multiple classified data according to the indicators.

[0064] It should be understood that multiple classified data are classified and stored according to indicators to facilitate rapid retrieval of agricultural data according to the indicators, that is, to improve the retrieval efficiency of agricultural data.

[0065] The management method of agricultural big data provided by the embodiment of the present invention collects agricultural data, determines the agricultural theme of the agricultural data, and determines the first target indicator set corresponding to the agricultural theme based on the theme-indicator set mapping relationship, so that the agricultural data can be finely divided according to the agricultural theme, which is convenient for quickly and accurately retrieving agricultural data according to the agricultural theme, that is, improving the retrieval efficiency and retrieval accuracy of agricultural data, and finely dividing the agricultural data, reducing the management chaos of agricultural big data, and improving the management accuracy of agricultural big data; based on each indicator in the first target indicator set, the agricultural data is classified according to the indicator to obtain a plurality of classified data, so that the agricultural data can be finely divided according to the indicator, which is convenient for quickly and accurately retrieving agricultural data according to the indicator, that is, improving the retrieval efficiency and retrieval accuracy of agricultural data, and finely dividing the agricultural data, reducing the management chaos of agricultural big data, and improving the management accuracy of agricultural big data; the plurality of classified data are classified and stored according to the indicator, so that the corresponding indicator can be quickly and accurately retrieved according to the agricultural theme, and then the corresponding agricultural data can be quickly and accurately retrieved according to the indicator, that is, improving the retrieval efficiency and retrieval accuracy of agricultural data, and finely dividing the agricultural data, reducing the management chaos of agricultural big data, and improving the management accuracy of agricultural big data.

[0066] Based on any of the above embodiments, after the above step 140, the method further includes:

[0067] receiving a topic selection instruction, and determining an agricultural topic to be searched based on the topic selection instruction;

[0068] Based on the topic-indicator set mapping relationship, determine the second target indicator set corresponding to the agricultural topic to be retrieved;

[0069] Display each indicator in the second target indicator set;

[0070] receiving an indicator selection instruction, and determining an indicator to be retrieved based on the indicator selection instruction;

[0071] Based on the index to be retrieved, the stored target agricultural data is retrieved.

[0072] Here, the topic selection instruction is an instruction triggered by a user, for example, an agricultural worker triggers the topic selection instruction through a client. The topic selection instruction indicates that an agricultural topic is to be retrieved.

[0073] It should be understood that the indicators in the second target indicator set are displayed for users to select indicators to be retrieved, so that non-professionals can also select indicators simply and quickly. Compared with setting indicators by users, since it requires users to have agricultural experience, the present invention can lower the management threshold of agricultural big data.

[0074] Here, the indicator selection instruction is an instruction triggered by a user, for example, an agricultural worker triggers the indicator selection instruction through a client. The indicator selection instruction indicates an indicator to be retrieved. Further, the indicator selection instruction may indicate one or more indicators.

[0075] Since the agricultural data is classified according to indicators to obtain multiple classified data, and the multiple classified data are classified and stored according to the indicators, the stored target agricultural data can be retrieved based on the indicator to be retrieved, and the indicator corresponding to the target agricultural data is the indicator to be retrieved.

[0076] The agricultural big data management method provided by the embodiment of the present invention receives a subject selection instruction, determines the agricultural subject to be retrieved based on the subject selection instruction, determines the second target indicator set corresponding to the agricultural subject to be retrieved based on the subject-indicator set mapping relationship, displays each indicator in the second target indicator set for users to select the indicator to be retrieved, so that non-professionals can also select indicators simply and quickly, thereby lowering the management threshold of agricultural big data and improving the retrieval efficiency of agricultural data; receives an indicator selection instruction, determines the indicator to be retrieved based on the indicator selection instruction, and quickly and accurately retrieves the stored target agricultural data based on the indicator to be retrieved, so that the corresponding indicator can be quickly and accurately retrieved according to the agricultural subject, and then the corresponding agricultural data can be quickly and accurately retrieved according to the indicator, that is, the retrieval efficiency and retrieval accuracy of agricultural data are improved.

[0077] Based on any of the above embodiments, considering that in agricultural big data, agricultural data from multiple types of data sources need to be collected, and currently different data sources have different storage formats, resulting in data separation, and data is difficult to share interactively, that is, it is impossible to cope with the storage requirements of large amounts of data, resulting in separate storage of data, resulting in data dispersion; based on this, the method also includes:

[0078] Collect target data of target data sources storing agricultural data;

[0079] Extracting keywords from the target data to obtain keywords;

[0080] Based on the keyword-field mapping relationship, determine the table field of the Hive table corresponding to the keyword;

[0081] Based on the table fields, the target data is stored.

[0082] Here, the target data source can be at least one of the following: MySQL, Oracle, Db2, MongoDB, Kafka, etc. That is, the present invention can support data collection from multiple data sources and increase the amount of sources collected for agricultural big data. The target data is also agricultural data.

[0083] In a specific embodiment, keyword feature information is extracted from data information of target data to obtain keyword feature information.

[0084] Here, the keyword-field mapping relationship is used to represent the one-to-one correspondence between multiple keywords and multiple fields. Further, the keyword-field mapping relationship is used to represent the one-to-one correspondence between multiple keyword feature information and multiple fields.

[0085] In Hive, based on table fields, the target data can be stored in the corresponding fields in the corresponding data table. If there are multiple keywords extracted, there can be multiple corresponding table fields, and the target data can be stored in the corresponding multiple fields in the corresponding data table.

[0086] Furthermore, the JSON format data of MongoDB is directly compiled to obtain keywords, and based on the keyword-field mapping relationship, the table field of the Hive table corresponding to the keyword is determined, and based on the table field, the target data of MongoDB is stored.

[0087] The management method of agricultural big data provided by the embodiment of the present invention collects target data of a target data source storing agricultural data, extracts keywords from the target data, obtains keywords, determines the table fields of the Hive table corresponding to the keywords based on the keyword-field mapping relationship, and stores the target data based on the table fields, so that keyword extraction can be performed on data from different data sources, and the keyword-field mapping relationship is unified for all data sources, that is, the table fields of the Hive table corresponding to the keywords can be determined based on the keyword-field mapping relationship for all data sources, and the target data is stored based on the table fields, that is, the data of all data sources can be stored in the same storage method, so that data separation will not be caused, and data can be interactively shared, so that the storage requirements of large amounts of data can be met, data storage can be avoided separately, and data dispersion can be reduced. That is, data from different data sources and different data formats of each agricultural system are formatted and stored uniformly, data separation is avoided, and data sharing and intercommunication are improved.

[0088] Based on any of the above embodiments, it is considered that data synchronization code is manually written to form a data synchronization code script, so as to realize data synchronization of different data sources through the data synchronization code script. However, according to different scenarios, such as according to the different two data sources between data transmission, different data synchronization code scripts need to be written separately, and manual writing of data synchronization code is very cumbersome, resulting in reduced efficiency in generating data synchronization code scripts, and manual writing of data synchronization code requires users to have certain code writing capabilities, and over-reliance on manual experience leads to low fault tolerance, which in turn leads to reduced accuracy in generating data synchronization code scripts. In addition, manual writing of data synchronization code requires users to have certain code writing capabilities, and the writing cost is high, resulting in increased generation costs of data synchronization code scripts. In addition, manual writing of data synchronization code requires users to have certain code writing capabilities, resulting in certain technical barriers to the generation of data synchronization code scripts; based on this, the method also includes:

[0089] Step 210, obtaining information configuration instructions acting on the configuration information input interface;

[0090] Step 220, determining at least two configuration information based on the information configuration instruction;

[0091] Step 230: construct a data synchronization code script based on the at least two pieces of configuration information, wherein the data synchronization code script is used to import data or export data.

[0092] Here, the configuration information input interface is used for the user to input the configuration information for building the data synchronization code script. In one embodiment, the configuration information input interface can be a WEB interface, which is more convenient for the user to input the configuration information, thereby improving the convenience and efficiency of generating the data synchronization code script.

[0093] Here, the information configuration instruction can be triggered by the user, such as on the client. The information configuration instruction indicates that there is configuration information. Based on this, the user can visually input the configuration information, that is, realize low-code and visual operation, thereby improving the generation efficiency of the data synchronization code script; and improving the fault tolerance rate, thereby improving the generation accuracy of the data synchronization code script; and reducing the technical threshold for the generation of the data synchronization code script, thereby reducing the writing cost, thereby reducing the generation cost of the data synchronization code script.

[0094] Here, the information configuration instruction indicates at least two pieces of configuration information, so that at least two pieces of configuration information can be directly obtained. The at least two pieces of configuration information include the resource name of the data resource to be synchronized and the data source type of the external data source, the external data source is the data source of the original storage location of the data to be imported or the data source of the target storage location of the data to be exported; the resource name is used as the script name of the data synchronization code script.

[0095] In some embodiments, the user is required to input all the configuration information for building the data synchronization code script. Based on this, determine the data synchronization type and multiple preset configuration items corresponding to the data synchronization type, and the multiple preset configuration items are the configuration items required to build the script; match at least two configuration information with multiple preset configuration items; if at least one preset configuration item that does not match at least two configuration information is determined from multiple preset configuration items, then return to obtain the information configuration instruction acting on the configuration information input interface, such as outputting prompt information to prompt the user to re-enter the configuration information, until no preset configuration item that does not match at least two configuration information is determined from multiple preset configuration items. For example, in the configuration information input interface, multiple preset configuration items are all required items.

[0096] In other embodiments, the user does not need to input all the configuration information for constructing the data synchronization code script. Based on this, it is determined whether at least two pieces of configuration information include the resource name of the data resource to be synchronized and the data source type of the external data source. If they are included, no processing is required. If not, it is returned to obtain the information configuration instruction acting on the configuration information input interface, such as outputting prompt information to prompt the user to re-enter the configuration information, until at least two pieces of configuration information include the resource name of the data resource to be synchronized and the data source type of the external data source.

[0097] It should be noted that the resource name of the data resource to be synchronized and the data source type of the external data source are configuration information that the user must enter; for example, in the configuration information input interface, the configuration items corresponding to the resource name and the configuration items corresponding to the data source type are required items.

[0098] Here, the data resource to be synchronized may be a data resource to be imported or a data resource to be exported. The resource name is used as the script name of the data synchronization code script.

[0099] Furthermore, the resource name is globally unique, that is, the resource name is used to uniquely identify the data resource to be synchronized. In one embodiment, it is determined whether the resource name is unique. If it is unique, no processing is required. If it is not unique, it returns to obtain information configuration instructions acting on the configuration information input interface, such as outputting prompt information to prompt the user to re-enter the configuration information until the resource name is unique.

[0100] Here, if you need to build a code script for data import, the external data source is the data source where the data to be imported is originally stored; if you need to build a code script for data export, the external data source is the data source where the data to be exported is to be imported, that is, the external data source is the data source where the data to be exported is to be stored. For example, if you need to build a code script for data import, the data source type can include but is not limited to: MySQL, Oracle, SQLServer, DB2, GreenPlum, PosgreSQL or MongoDB, etc., to support data import from multiple external data sources.

[0101] The at least two pieces of configuration information may also include, but are not limited to, at least one of the following: storage mode of data to be synchronized, data synchronization mode of data to be synchronized, connection information of external data sources, storage location information of data to be synchronized, target table name of data to be synchronized, whether to overwrite data, extended parameters, yarn queue information, etc. The data to be synchronized is data to be imported or data to be exported.

[0102] The storage method is an import method or an export method of the data to be synchronized. If the data to be synchronized is data to be imported, the storage method may include but is not limited to: Hive or HDFS, etc., to support multiple data storage forms. That is, the storage method is an import method of the data to be imported or an export method of the data to be exported.

[0103] The data synchronization method may include but is not limited to: single-table, multi-table or conditional data synchronization methods, to support multiple data synchronization methods. The single-table data synchronization method is to synchronize the data of a single table through the single-table method; the multi-table data synchronization method is to synchronize the data of multiple tables with the same structure at the same time through the multi-table method; the conditional data synchronization method is to pre-process the data and realize incremental synchronization through the conditional method.

[0104] The connection information of the external data source may include but is not limited to at least one of the following: database IP address, port, user name, password, etc.

[0105] The storage location information is the synchronization location information of the data to be synchronized. If the data to be synchronized is data to be imported, the storage location information may be a storage path (such as an HDFS path). If the data to be synchronized is data to be exported, the storage location information may be a stored database name.

[0106] The target table name is the storage table name of the data to be synchronized.

[0107] Whether to overwrite data specifies whether to overwrite existing data when executing the synchronization task.

[0108] The extended parameter can be set according to actual needs; for example, the extended parameter is num-mappers, which sets the number of map executions; the extended parameter is hive-drop-import-delims, which removes characters such as \r, \n, \013 or \010 in the data when importing data into HIVE; the extended parameter is null-non-string, which replaces null (empty) columns of non-String type with the specified string.

[0109] The YARN queue information is used to specify the execution queue of the YARN task.

[0110] In some embodiments, at least two input information indicated by the information configuration instruction are obtained; the at least two input information are corrected to obtain at least two configuration information for building the script. Based on this, considering that the configuration information input by the user may not be accurate, it is corrected, thereby improving the accuracy of generating the data synchronization code script. In one embodiment, at least two configuration information matching the at least two input information are determined from a plurality of preset configuration information; the plurality of preset configuration information is accurate configuration information set in advance.

[0111] In some embodiments, the user is required to input all configuration information for constructing the data synchronization code script. Based on this, based on the at least two configuration information, the script is directly constructed to obtain the data synchronization code script.

[0112] In other embodiments, the user is not required to input all the configuration information for building the data synchronization code script. Based on this, determine the data synchronization type and multiple preset configuration items corresponding to the data synchronization type, the data synchronization type includes data import or data export, and the multiple preset configuration items are the configuration items required for building the script; match at least two configuration information with multiple preset configuration items; if at least one preset configuration item that does not match the at least two configuration information is determined from the multiple preset configuration items, determine the configuration information corresponding to the at least one preset configuration item; based on the at least two configuration information and the configuration information corresponding to the at least one preset configuration item, build the data synchronization code script; if the preset configuration item that does not match the at least two configuration information is not determined from the multiple preset configuration items, based on the at least two configuration information, perform script construction to obtain the data synchronization code script.

[0113] In one embodiment, based on multiple preset information required for building a script, configuration information corresponding to at least one preset configuration item is determined. Further, preset information corresponding to at least one second preset configuration item matching at least one preset configuration item is determined from multiple preset information required for building a script; if at least one first preset configuration item that does not match at least one second preset configuration item is determined from at least one preset configuration item, configuration information corresponding to at least one first preset configuration item is obtained based on the data source type; the configuration information corresponding to at least one first preset configuration item and the preset information corresponding to at least one second preset configuration item are used as the configuration information corresponding to at least one preset configuration item; if no first preset configuration item that does not match at least one second preset configuration item is determined from at least one preset configuration item, the configuration information corresponding to at least one second preset configuration item is used as the configuration information corresponding to at least one preset configuration item.

[0114] Furthermore, an information update instruction acting on the configuration information update interface is obtained; based on the information update instruction, at least one configuration item to be updated and new configuration information corresponding to the at least one configuration item to be updated are determined; based on the at least one configuration item to be updated, at least one information to be updated is determined from multiple preset information; and the at least one information to be updated is updated to the new configuration information corresponding to the at least one configuration item to be updated. Based on this, multiple preset information can be flexibly set according to actual needs, and the update method is a visual update method, that is, a low-code update method.

[0115] In one embodiment, a data synchronization instruction is received, and based on the resource name indicated by the data synchronization instruction, the at least two configuration information are obtained, so as to construct a data synchronization code script based on the at least two configuration information. That is, when a user executes a synchronization task, the data synchronization instruction can be triggered to obtain the at least two configuration information based on the resource name (globally unique) of the synchronization task, so as to generate a data synchronization code script when the synchronization task is executed, rather than generating the data synchronization code script in advance, so as to ensure that the data synchronization code script is generated only when data synchronization is needed, thereby reducing the waste of computing resources and storage resources.

[0116] Furthermore, if an information change instruction is received, at least one configuration item to be changed and new configuration information corresponding to at least one configuration item to be changed are determined based on the information change instruction; at least two configuration items are changed based on at least one configuration item to be changed and new configuration information corresponding to at least one configuration item to be changed, and at least two configuration items after the change are obtained, so as to construct a data synchronization code script based on the at least two configuration items after the change. Based on this, a data synchronization code script is generated when the synchronization task is executed, rather than generating the data synchronization code script in advance, to ensure that it is generated only when data synchronization is required, and supports visual changes to configuration information when data synchronization is required, so that there is no need to re-enter the configuration information, thereby improving the generation efficiency of the data synchronization code script.

[0117] Exemplarily, the parser parses the configuration information into a standard Sqoop script, and the compiler translates the Sqoop script into a mapreduce program to achieve data synchronization through the MR computing engine.

[0118] It can be understood that the present invention supports data import and export functions. Under different servers, data exchange can be achieved through import and export, and import and export scripts can be quickly generated.

[0119] The management method of agricultural big data provided by the embodiment of the present invention obtains an information configuration instruction acting on a configuration information input interface, determines at least two configuration information based on the information configuration instruction, and automatically constructs a data synchronization code script based on the at least two configuration information, so that the user only needs to visually input the configuration information required for constructing the data synchronization code script to automatically generate the data synchronization code script, thereby improving the convenience of generating the data synchronization code script, and further improving the efficiency of generating the data synchronization code; and visually inputting the configuration information, the user only needs to input according to the actual data synchronization task requirements, and the user does not need to have the ability to write code, thereby improving the fault tolerance rate, and further improving the accuracy of generating the data synchronization code script; and the user does not need to have the ability to write code, thereby reducing the cost of writing code, and further reducing the cost of generating the data synchronization code script; and the user does not need to have the ability to write code, thereby reducing the technical threshold for generating the data synchronization code script; and the at least two configuration information include the resource name of the data resource to be synchronized and the data source type of the external data source, thereby ensuring that the user inputs the resource name and the data source type, and further ensuring the successful generation of the data synchronization code script.

[0120] Based on any of the above embodiments, in the method, the above step 230 includes the following steps 231-235:

[0121] Step 231, determining a data synchronization type and a plurality of preset configuration items corresponding to the data synchronization type, wherein the data synchronization type includes data import or data export, and the plurality of preset configuration items are configuration items required for constructing a script.

[0122] Here, the data synchronization type indicates whether the data synchronization code script to be constructed is a data import script or a data export script. That is, the data synchronization type includes a data import type or a data export type.

[0123] In one embodiment, a data synchronization instruction is received, and the data synchronization type is determined based on the data synchronization instruction, that is, the data synchronization instruction indicates whether it is a data import task or a data export task. In addition, based on the resource name indicated by the data synchronization instruction, the at least two configuration information are obtained to build a data synchronization code script based on the at least two configuration information. That is, when a user executes a synchronization task, the data synchronization instruction can be triggered to obtain the at least two configuration information based on the resource name (globally unique) of the synchronization task, so as to generate a data synchronization code script when the synchronization task is executed, rather than generating the data synchronization code script in advance, to ensure that it is generated only when data synchronization is needed, thereby reducing the waste of computing resources and storage resources.

[0124] Furthermore, if an information change instruction is received, at least one configuration item to be changed and new configuration information corresponding to at least one configuration item to be changed are determined based on the information change instruction; at least two configuration items are changed based on at least one configuration item to be changed and new configuration information corresponding to at least one configuration item to be changed, and at least two configuration items after the change are obtained, so as to construct a data synchronization code script based on the at least two configuration items after the change. Based on this, a data synchronization code script is generated when the synchronization task is executed, rather than generating the data synchronization code script in advance, to ensure that it is generated only when data synchronization is required, and supports visual changes to configuration information when data synchronization is required, so that there is no need to re-enter the configuration information, thereby improving the generation efficiency of the data synchronization code script.

[0125] Here, the multiple preset configuration items are all the configuration items required to build the data synchronization code script. The multiple preset configuration items are set in advance for different data synchronization types. It should be understood that the configuration items required for the data import code script and the data export code script may be different.

[0126] The multiple preset configuration items may include but are not limited to at least one of the following: a configuration item corresponding to the resource name of the data resource to be synchronized, a configuration item corresponding to the data source type of the external data source, a configuration item corresponding to the storage method of the data to be synchronized, a configuration item corresponding to the data synchronization method of the data to be synchronized, a configuration item corresponding to the connection information of the external data source, a configuration item corresponding to the storage location information of the data to be synchronized, a configuration item corresponding to the target table name of the data to be synchronized, a configuration item corresponding to whether to overwrite the data, a configuration item corresponding to the extended parameter, a configuration item corresponding to the yarn queue information, and the like.

[0127] Step 232: Match the at least two pieces of configuration information with the multiple preset configuration items.

[0128] Specifically, at least two configuration information are matched with multiple preset configuration items to determine whether the at least two configuration information cover all of the multiple preset configuration items. If not, at least one preset configuration item that does not match the at least two configuration information can be determined from the multiple preset configuration items. If covered, no preset configuration item that does not match the at least two configuration information is determined from the multiple preset configuration items.

[0129] Step 233: If at least one preset configuration item that does not match the at least two pieces of configuration information is determined from the plurality of preset configuration items, configuration information corresponding to the at least one preset configuration item is determined based on the data source type.

[0130] Here, at least one preset configuration item is a configuration item required to build the data synchronization code, but the user has not entered the configuration information corresponding to at least one preset configuration item. Based on this, it is necessary to determine the configuration information corresponding to at least one preset configuration item before the data synchronization code script can be successfully built.

[0131] Here, the data source type can be used to determine the configuration information corresponding to individual preset configuration items, that is, the configuration information corresponding to individual preset configuration items is stored in advance based on the data source type, so that the stored configuration information can be obtained based on the data source type. For example, if the preset configuration item is a configuration item corresponding to the connection information of an external data source, the connection information corresponding to the configuration item can be obtained based on the data source type, such as directly obtaining the database IP address, port, user name, password, etc.

[0132] In one embodiment, at least one preset configuration item can be configured based on the data source type, that is, configuration information corresponding to all preset configuration items can be obtained based on the data source type. In other words, the preset configuration items among the multiple preset configuration items, except for all the first preset configuration items that can be configured based on the data source type, all require user input.

[0133] In another embodiment, at least one first preset configuration item that can be configured based on the data source type is determined from at least one preset configuration item; based on the data source type, configuration information corresponding to the at least one first preset configuration item is obtained; if at least one second preset configuration item that does not match the at least one first preset configuration item is determined from the at least one preset configuration item, preset information corresponding to the at least one second preset configuration item is determined from multiple preset information required to build the script; the configuration information corresponding to the at least one first preset configuration item and the preset information corresponding to the at least one second preset configuration item are used as the configuration information corresponding to the at least one preset configuration item; if the second preset configuration item that does not match the at least one first preset configuration item is not determined from the at least one preset configuration item, the configuration information corresponding to the at least one first preset configuration item is used as the configuration information corresponding to the at least one preset configuration item.

[0134] Step 234: construct a data synchronization code script based on the at least two pieces of configuration information and the configuration information corresponding to the at least one preset configuration item.

[0135] Here, the configuration information corresponding to at least two configuration information and at least one preset configuration item is already all the configuration information required to build a data synchronization code script. Based on this, the data synchronization code script can be directly constructed based on the configuration information corresponding to at least two configuration information and at least one preset configuration item.

[0136] Step 235: If no preset configuration item that does not match the at least two configuration information is determined from the multiple preset configuration items, a script is constructed based on the at least two configuration information to obtain a data synchronization code script.

[0137] If no preset configuration item that does not match at least two configuration information is determined from multiple preset configuration items, that is, at least two configuration information already include all configuration information required to construct the data synchronization code script, based on this, based on the at least two configuration information, the script is directly constructed to obtain the data synchronization code script.

[0138] The agricultural big data management method provided by the embodiment of the present invention determines the data synchronization type and multiple preset configuration items corresponding to the data synchronization type, and the data synchronization type includes data import or data export, so as to ensure the accuracy of determining the multiple preset configuration items, and then ensure the successful creation of the data synchronization code script, that is, improve the generation accuracy of the data synchronization code script; if at least one preset configuration item that does not match at least two configuration information is determined from the multiple preset configuration items, based on the data source type, the configuration information corresponding to the at least one preset configuration item is determined, so as to construct the data synchronization code script based on the at least two configuration information and the configuration information corresponding to the at least one preset configuration item, so as to ensure that the script is generated only after all the configuration information for creating the data synchronization code script is obtained, and then ensure that the data synchronization code script is successfully created, that is, improve the generation accuracy of the data synchronization code script.

[0139] Based on any of the above embodiments, in the method, determining the configuration information corresponding to the at least one preset configuration item based on the data source type includes:

[0140] Determine at least one first preset configuration item that can be configured based on the data source type from the at least one preset configuration item;

[0141] Based on the data source type, obtaining configuration information corresponding to the at least one first preset configuration item;

[0142] If at least one second preset configuration item that does not match the at least one first preset configuration item is determined from the at least one preset configuration item, determining preset information corresponding to the at least one second preset configuration item from a plurality of preset information required to construct the script;

[0143] Using the configuration information corresponding to the at least one first preset configuration item and the preset information corresponding to the at least one second preset configuration item as the configuration information corresponding to the at least one preset configuration item;

[0144] If no second preset configuration item that does not match the at least one first preset configuration item is determined from the at least one preset configuration item, the configuration information corresponding to the at least one first preset configuration item is used as the configuration information corresponding to the at least one preset configuration item.

[0145] Considering that determining configuration information based on the data source type is more accurate than determining configuration information from preset information, based on this, at least one first preset configuration item that can be configured based on the data source type is first determined from at least one preset configuration item.

[0146] Here, the data source type can be used to determine the configuration information corresponding to at least one first preset configuration item, that is, the configuration information corresponding to at least one first preset configuration item is stored in advance based on the data source type, so that the stored configuration information can be obtained based on the data source type. For example, if the preset configuration item is a configuration item corresponding to the connection information of an external data source, the connection information corresponding to the configuration item can be obtained based on the data source type, such as directly obtaining the database IP address, port, user name, password, etc.

[0147] If there are preset configuration items in at least one preset configuration item that cannot be configured based on the data source type, that is, at least one second preset configuration item that does not match the at least one first preset configuration item is determined from the at least one preset configuration item, and preset information corresponding to the at least one second preset configuration item is determined from the multiple preset information required to build the script.

[0148] Here, the plurality of preset information includes configuration information that cannot be configured based on the data source type. Further, the plurality of preset information may not include the resource name of the data resource to be synchronized and the data source type of the external data source, because these two items are required. The preset information can also be set in advance for direct use later.

[0149] The multiple preset information may include but is not limited to at least one of the following: storage method of the data to be synchronized, data synchronization method of the data to be synchronized, connection information of the external data source, storage location information of the data to be synchronized, target table name of the data to be synchronized, whether to overwrite data, extended parameters and yarn queue information, etc.

[0150] Here, if at least one second preset configuration item that does not match at least one first preset configuration item is determined from at least one preset configuration item, the configuration information corresponding to the at least one first preset configuration item and the preset information corresponding to the at least one second preset configuration item, and at least two configuration information are already all the configuration information required to construct the data synchronization code script; if a second preset configuration item that does not match at least one first preset configuration item is not determined from at least one preset configuration item, the configuration information corresponding to the at least one first preset configuration item, and at least two configuration information are already all the configuration information required to construct the data synchronization code script.

[0151] The management method of agricultural big data provided by an embodiment of the present invention, through the above-mentioned method, if at least one second preset configuration item that does not match at least one first preset configuration item is determined from at least one preset configuration item, the preset information corresponding to the at least one second preset configuration item is determined from multiple preset information required to build the script, thereby ensuring that the script is generated only after all configuration information for creating the data synchronization code script is obtained, thereby ensuring that the data synchronization code script is successfully created, that is, improving the generation accuracy of the data synchronization code script.

[0152] Based on any of the above embodiments, the method further includes the following steps 240 to 270.

[0153] Step 240: Obtain information update instructions acting on the configuration information update interface.

[0154] In one embodiment, the method provided by the embodiment of the present invention may be applied to a server, and the configuration information update interface may be an interface on a client, that is, the information update instruction is received by the client and forwarded to the server.

[0155] Here, the configuration information update interface is used for users to update preset information. In one embodiment, the configuration information update interface can be a WEB interface, so that it is more convenient for users to update preset information, thereby improving the convenience and efficiency of updating the preset information.

[0156] Here, the information update instruction can be triggered by the user, such as on the client. The information update instruction indicates that there is at least one configuration item to be updated and at least one new configuration information corresponding to the configuration item to be updated. Based on this, the user can visually update the preset configuration information, that is, realize low-code and visual operations, thereby improving the updating efficiency of the preset information; and improving the fault tolerance rate, thereby improving the updating accuracy of the preset information; and reducing the technical threshold for updating the preset information, thereby reducing the writing cost, and thereby reducing the generation cost of the data synchronization code script.

[0157] Step 250: Determine at least one configuration item to be updated and new configuration information corresponding to the at least one configuration item to be updated based on the information update instruction.

[0158] The at least one configuration item to be updated may include but is not limited to at least one of the following: a configuration item corresponding to the storage method of the data to be synchronized, a configuration item corresponding to the data synchronization method of the data to be synchronized, a configuration item corresponding to the connection information of the external data source, a configuration item corresponding to the storage location information of the data to be synchronized, a configuration item corresponding to the target table name of the data to be synchronized, a configuration item corresponding to whether to overwrite the data, a configuration item corresponding to the extended parameter, a configuration item corresponding to the yarn queue information, and the like.

[0159] Step 260: Based on the at least one configuration item to be updated, determine at least one piece of information to be updated from the plurality of preset information.

[0160] Here, the configuration item to be updated is a configuration item to be updated, based on which, information to be updated corresponding to at least one configuration item to be updated can be determined from a plurality of preset information.

[0161] Step 270: Update the at least one information to be updated to new configuration information corresponding to the at least one configuration item to be updated.

[0162] Specifically, at least one to-be-updated information is replaced with new configuration information corresponding to at least one to-be-updated configuration item, that is, the multiple preset information used subsequently includes the new configuration information corresponding to at least one to-be-updated configuration item.

[0163] The agricultural big data management method provided by the embodiment of the present invention, through the above-mentioned method, the user only needs to visually input the configuration information that needs to be updated, and multiple preset information can be automatically updated, thereby improving the convenience of updating the preset information, and then improving the updating efficiency of the preset information; and for visual input of configuration information, the user only needs to input it according to the actual data synchronization task needs, and the user does not need to have code writing ability, thereby improving the fault tolerance rate, and then improving the generation accuracy of data synchronization code scripts; and the user does not need to have code writing ability, thereby reducing the code writing cost, and then reducing the generation cost of data synchronization code scripts; and the user does not need to have code writing ability, thereby reducing the technical threshold for generating data synchronization code scripts; and multiple preset information can be updated according to actual needs, rather than being immutable, thereby ensuring the successful generation of data synchronization code scripts, that is, improving the generation accuracy of data synchronization code scripts.

[0164] Based on any of the above embodiments, before the above step 230, the method further includes:

[0165] A mapping relationship between the resource name and the at least two pieces of configuration information is constructed.

[0166] Based on the mapping relationship, storing the at least two pieces of configuration information;

[0167] A data synchronization instruction is received, and based on the resource name indicated by the data synchronization instruction, the at least two pieces of configuration information stored are acquired.

[0168] Here, at least two pieces of configuration information are stored based on the mapping relationship, so that the at least two pieces of configuration information stored can be subsequently acquired based on the resource name.

[0169] In one embodiment, at least two configuration information are stored in a MySQL database. Exemplarily, at least two configuration information are stored in a MySQL database through a springboot program.

[0170] Here, the data synchronization instruction can be triggered by the user, so that when the user executes a synchronization task, the data synchronization instruction can be triggered to obtain the at least two configuration information based on the resource name (globally unique) of the synchronization task, so as to generate a data synchronization code script when the synchronization task is executed, rather than generating the data synchronization code script in advance, to ensure that it is generated only when data synchronization is needed, thereby reducing the waste of computing resources and storage resources.

[0171] The agricultural big data management method provided by the embodiment of the present invention can generate a data synchronization code script when the synchronization task is executed, instead of generating the data synchronization code script in advance, to ensure that data synchronization is generated only when it is needed, thereby reducing the waste of computing resources and storage resources.

[0172] Based on any of the above embodiments, after acquiring the at least two pieces of configuration information stored based on the resource name indicated by the data synchronization instruction, the method further includes:

[0173] If an information change instruction is received, determining at least one configuration item to be changed and new configuration information corresponding to the at least one configuration item to be changed based on the information change instruction;

[0174] Based on the at least one configuration item to be changed and the new configuration information corresponding to the at least one configuration item to be changed, the at least two configuration information are changed to obtain the at least two configuration information after the change, so as to construct a data synchronization code script based on the at least two configuration information after the change.

[0175] Here, the information change instruction can be triggered by the user, such as on the client. The information change instruction indicates that there is at least one configuration item to be changed and at least one new configuration information corresponding to the configuration item to be changed. Based on this, the user can visualize the configuration information entered before the change, that is, to achieve low-code and visual operations, thereby improving the efficiency of updating the configuration information; and improving the fault tolerance rate, thereby improving the accuracy of updating the configuration information; and reducing the technical threshold for updating the configuration information, thereby reducing the writing cost, and thereby reducing the cost of generating the data synchronization code script.

[0176] The at least one configuration item to be changed may include but is not limited to at least one of the following: a configuration item corresponding to the resource name of the data resource to be synchronized, a configuration item corresponding to the data source type of the external data source, a configuration item corresponding to the storage method of the data to be synchronized, a configuration item corresponding to the data synchronization method of the data to be synchronized, a configuration item corresponding to the connection information of the external data source, a configuration item corresponding to the storage location information of the data to be synchronized, a configuration item corresponding to the target table name of the data to be synchronized, a configuration item corresponding to whether to overwrite the data, a configuration item corresponding to the extended parameter, a configuration item corresponding to the yarn queue information, and the like.

[0177] Specifically, based on at least one configuration item to be changed, at least one piece of information to be changed is determined from at least two pieces of configuration information, and the at least one piece of information to be changed is replaced with new configuration information corresponding to the at least one configuration item to be changed.

[0178] Furthermore, after at least two pieces of configuration information are changed, a data synchronization code script may be immediately constructed based on the at least two pieces of configuration information after the changes.

[0179] The management method of agricultural big data provided by the embodiment of the present invention, through the above-mentioned method, the user only needs to visually input the configuration information that needs to be changed, and the previously input configuration information can be automatically changed, thereby improving the convenience of changing the configuration information, and then improving the efficiency of changing the configuration information; and to visually input the configuration information, the user only needs to input it according to the actual data synchronization task needs, and the user does not need to have code writing ability, thereby improving the fault tolerance rate, and then improving the generation accuracy of the data synchronization code script; and the user does not need to have code writing ability, thereby reducing the code writing cost, and then reducing the generation cost of the data synchronization code script; and the user does not need to have code writing ability, thereby reducing the technical threshold for generating data synchronization code scripts; and the previously input configuration information can be updated according to actual needs, rather than being unchanged, thereby ensuring the successful generation of the data synchronization code script, that is, improving the generation accuracy of the data synchronization code script; and it supports visual changes in configuration information when data synchronization is required, so that there is no need to re-enter more configuration information, thereby improving the generation efficiency of the data synchronization code script.

[0180] Based on any of the above embodiments, the method further includes:

[0181] Get the parameter configuration instructions for the new interface of the data source;

[0182] If the user who triggers the parameter configuration instruction has the authority to create a new data source, determining at least one configuration parameter based on the parameter configuration instruction;

[0183] Based on the at least one configuration parameter, a data source is constructed.

[0184] In one embodiment, the method provided by the embodiment of the present invention may be applied to a server, and the data source newly added interface may be an interface on a client, that is, the parameter configuration instruction is received by the client and forwarded to the server.

[0185] Here, the data source adding interface is used for the user to visually add a new data source. In one embodiment, the data source adding interface can be a WEB interface, so that it is more convenient for the user to input the new data source, thereby improving the convenience and efficiency of adding a new data source.

[0186] Here, the parameter configuration instruction can be triggered by the user, such as on the client. The parameter configuration instruction indicates that there is at least one configuration parameter. Based on this, the user can visually input the configuration parameters, that is, realize low-code and visual operation, thereby improving the efficiency of adding new data sources; and improving the fault tolerance rate, thereby improving the accuracy of adding new data sources; and reducing the technical threshold for adding new data sources, thereby reducing the writing cost, and thereby reducing the cost of adding new data sources.

[0187] It should be noted that for the management of data sources, the components in data source management are business components. Each tenant member has the right to use the data source configured by the administrator, and the data source configured by the tenant can only be used by users under the tenant. Specifically, the administrator has the right to create, edit, delete and view details of all data sources, and the data source configured by the administrator is granted use permission to all users by default and only has use permission. Ordinary users have the right to edit, delete and view the data source created by the user. Based on this, if the user who triggers the parameter configuration instruction has the permission to create a new data source, at least one configuration parameter is determined based on the parameter configuration instruction.

[0188] Exemplarily, if a new MySQL data source is added, at least one configuration parameter includes the resource name of the data source (the resource name for configuring MySQL, which is a unique identifier), the database IP address (host name) (the full domain name or IP address of the machine where MySQL is located), the port (MySQL communication port), the user name (the user name used to log in to the MySQL database), the password (the password corresponding to the user name when logging in to the MySQL database), the database name (the database name of the MySQL connection) and the JDBC (Java Database Connectivity, Java database connection) connection parameters (parameters required when JDBC connects to MySQL), etc.

[0189] In addition, other newly added data sources can set the required configuration parameters according to actual needs, which will not be described here one by one.

[0190] The management method of agricultural big data provided by the embodiment of the present invention, through the above-mentioned method, the user only needs to visually input the configuration information for constructing the data source to automatically construct the data source, thereby improving the convenience of adding the data source, and further improving the efficiency of adding the data source; and the configuration information is visually input, and the user only needs to input it according to the actual task needs, and the user does not need to have the ability to write code, thereby improving the fault tolerance rate, and further improving the accuracy of adding the data source; and the user does not need to have the ability to write code, thereby reducing the cost of code writing, and further reducing the cost of adding the data source; and the user does not need to have the ability to write code, thereby reducing the technical threshold for adding the data source.

[0191] Based on any of the above embodiments, agricultural data can be automatically synchronized and stored by the data analysis platform, or the data table can be manually created by the developer, the data table can be automatically generated, and the metadata information of the data table can be automatically stored in the database. In addition, the present invention can create tables by visualizing pages, define the meaning of table fields by logical models and physical models, and thus automatically generate data tables. In addition, a convenient and fast visual interface can also be provided for the project, the number of scripts, the number of workflows, and the dependencies between tables of the project.

[0192] Based on any of the above embodiments, considering that the user is unclear about the dependency relationship between multiple data tables; based on this, the method further includes:

[0193] Based on the association relationships between the multiple data tables and the primary key fields of each data table in the multiple data tables, a data table lineage map is generated.

[0194] Here, the data table lineage map is used to indicate the relationship between multiple data tables, that is, it is a table relationship map. The data table lineage map clearly indicates the association relationship and primary key fields between various data tables, so that developers can improve their development efficiency with the help of the data table lineage map.

[0195] The agricultural big data management method provided by the embodiment of the present invention can generate a data table lineage map through the above-mentioned method, and the data table lineage map clearly indicates the association relationship and primary key field between each data table, so that developers can use the data table lineage map to improve the development efficiency of developers, thereby improving the background development efficiency of the agricultural big data platform, providing great convenience for supporting the management of back-end personnel, that is, improving the convenience of managing agricultural big data.

[0196] Based on any of the above embodiments, the method further includes:

[0197] Generate a data table view based on the table information of the data table; display the data table view so that users can view the table information quickly and conveniently.

[0198] Here, the table information may include but is not limited to at least one of the following: the source of the data table, the fields included in the data table, the meaning of the fields, the source of the upstream table, the source of the downstream table, etc. The data table view includes the table information.

[0199] Based on any of the above embodiments, the method further includes:

[0200] Obtain a data table naming standard setting instruction, and determine the data table naming standard based on the data table naming standard setting instruction, so as to facilitate users to control data quality. For problems with non-standard data table naming, automatic verification can be performed and an error report can be generated.

[0201] Based on any of the above embodiments, the method further includes:

[0202] Obtain metadata standard setting instructions, and determine metadata standards based on the metadata standard setting instructions, so as to facilitate users to control data quality. For problems with non-standard metadata, automatic verification and error report generation can be performed.

[0203] Based on any of the above embodiments, the method further includes:

[0204] Obtain the coding standard setting instruction, and determine the coding standard based on the coding standard setting instruction, so as to facilitate the user to control the data quality. For the problem of non-standard coding, it can automatically check and generate an error report.

[0205] Based on any of the above embodiments, the method further includes:

[0206] Obtain field standard setting instructions, and determine field standards based on the field standard setting instructions, so as to facilitate users to control data quality. For problems with non-standard fields, automatic verification can be performed and error reports can be generated.

[0207] Based on any of the above embodiments, the method further includes:

[0208] Obtain the running status of the data table; generate a visual report based on the running status of the data table.

[0209] Specifically, statistics are collected on the operation of the data table every day to form a visual report, which may include but is not limited to at least one of the following: number of script executions per day, script operation market ranking, daily incremental data, data resource access volume, data update time, etc.

[0210] Based on any of the above embodiments, the present invention also supports a data query function of a data packet, and can easily query part of the sample data in a data table.

[0211] Based on any of the above embodiments, the method further includes:

[0212] Obtain the association relationship between multiple data tables, generate text data based on the association relationship between the multiple data tables, and export the text data. It should be understood that the exported text data is in text format and is only available under the execution subject of the present invention, and is not available on other devices.

[0213] Based on any of the above embodiments, the method further includes:

[0214] The data quality report can be sent to a designated mailbox at regular intervals.

[0215] Based on any of the above embodiments, the method further includes:

[0216] receiving a theme setting instruction, and determining a target data table to be set and a theme to be set based on the theme setting instruction;

[0217] Setting the subject of the target data table as the subject to be set;

[0218] Receiving a data table search instruction, and determining a subject to be searched based on the data table search instruction;

[0219] Based on the subject to be searched, a data table with the subject being the subject to be searched is obtained.

[0220] Here, the theme setting instruction is an instruction triggered by a user, for example, an agricultural worker triggers the theme setting instruction through a client. The theme setting instruction indicates that a theme is to be set.

[0221] Furthermore, after the theme of each data table is set, the hierarchical relationship of each data table can be customized. Specifically, a hierarchical setting instruction is received, and the hierarchical relationship of multiple data tables is set based on the hierarchical setting instruction.

[0222] The agricultural big data management method provided by the embodiment of the present invention edits the data table in the above-mentioned manner, that is, customizes the subject domain or subject level to which the data table belongs, so that the corresponding agricultural data can be quickly and accurately retrieved according to the agricultural theme, thereby improving the retrieval efficiency and retrieval accuracy of agricultural data.

[0223] Based on any of the above embodiments, the method further includes:

[0224] Obtain user management instructions for agricultural big data, and determine the functional authority and data authority of the user corresponding to the user management instructions;

[0225] Based on the function permission and the data permission, determining whether the operation indicated by the user management instruction can be implemented;

[0226] The functional permission is used to determine whether the function can be used, and the data permission is used to determine the scope of use of the permission.

[0227] Here, the user management instruction is an instruction triggered by a user, for example, an agricultural worker triggers the user management instruction through a client. The user management instruction indicates that there is a corresponding operation.

[0228] Here, functional permissions are used to solve opposing problems such as whether permissions exist or not, can or cannot, can or cannot, available or unavailable, yes or no, etc., so as to determine whether the user can use the corresponding function, that is, whether the user can execute the corresponding function. For example, functional permissions refer to the permission for functions such as read, write, check, modify, and delete for a certain business. Data permissions are used to solve the problem of the scope of permissions available. Therefore, based on functional permissions and data permissions, it can be determined whether the operation indicated by the user management instruction can be implemented.

[0229] If the operation indicated by the user management instruction is implementable, the operation indicated by the user management instruction is implemented; if the operation indicated by the user management instruction is not implementable, the operation indicated by the user management instruction is not implemented, and further, a prompt message of no authority can be output.

[0230] It should be noted that the present invention supports the management and control of data resources, data interfaces and functional permissions through account roles, that is, different roles can be assigned different functional permissions and data permissions.

[0231] Furthermore, the present invention supports organizational structure permission control, and the organization is the basic unit of a multi-user system. Within an organization, users can create and manage resources (data sources, data views, visualization views, scheduled tasks, etc.), invite other users to join the organization, create and manage roles, and set user / role permissions for resources. Each user can create an organization and can also be invited to other users' organizations. User permissions support direct joining after a user is invited by email; support role management and support organization-based user management.

[0232] It should be noted that data sources have strict permission opening management, which is divided into administrators and ordinary users. Administrators have the permissions to create, edit, delete, and view details of all data sources, while ordinary users only have usage permissions, provided that they have been authorized by the administrator to use the component. Business service components are isolated by user, and ordinary users can create business resources individually.

[0233] The agricultural big data management method provided by the embodiment of the present invention can ensure data security and improve the management security and management accuracy of agricultural big data through the above-mentioned method.

[0234] In summary, the present invention solves the problems of unclear data links and chaotic data table management through unified management methods, standard system construction, and quality management construction, improves the management efficiency of data resources, solves the computing bottleneck of large-scale agricultural data analysis and processing, and more efficiently mines data value, providing efficient and convenient data support for agricultural business decisions and management. Furthermore, by building a data middle platform system that integrates data collection, storage, and calculation, it improves the sharing, intercommunication, and security of data, improves the storage and computing capabilities for large-scale agricultural data, and more efficiently mines data value, providing integrated services for data management and data application in the agricultural field.

[0235] The agricultural big data management device provided by the present invention is described below. The agricultural big data management device described below and the agricultural big data management method described above can be referenced to each other.

[0236] Figure 2 A schematic diagram of the structure of the agricultural big data management device provided by the present invention is shown in FIG. Figure 2 As shown, the agricultural big data management device includes:

[0237] The theme determination module 210 is used to collect agricultural data and determine the agricultural theme of the agricultural data;

[0238] An indicator determination module 220, for determining a first target indicator set corresponding to the agricultural theme based on a theme-indicator set mapping relationship;

[0239] A data classification module 230, configured to classify the agricultural data according to the indicators in the first target indicator set to obtain a plurality of classified data;

[0240] The classification storage module 240 is used to classify and store the multiple classification data according to indicators.

[0241] Figure 3 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 3 As shown, the electronic device may include: a processor 310, a communication interface 320, a memory 330 and a communication bus 340, wherein the processor 310, the communication interface 320 and the memory 330 communicate with each other through the communication bus 340. The processor 310 may call the logic instructions in the memory 330 to execute the management method of agricultural big data, which includes: collecting agricultural data and determining the agricultural theme of the agricultural data; determining the first target indicator set corresponding to the agricultural theme based on the theme-indicator set mapping relationship; classifying the agricultural data according to the indicators based on the indicators in the first target indicator set to obtain multiple classified data; and storing the multiple classified data according to the indicators.

[0242] In addition, the logic instructions in the above-mentioned memory 330 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0243] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, it is implemented to execute the agricultural big data management method provided by the above-mentioned methods. The method includes: collecting agricultural data and determining the agricultural theme of the agricultural data; based on the theme-indicator set mapping relationship, determining the first target indicator set corresponding to the agricultural theme; based on each indicator in the first target indicator set, classifying the agricultural data according to the indicator to obtain multiple classified data; and storing the multiple classified data according to the indicator.

[0244] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0245] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0246] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for managing agricultural big data, characterized in that: include: collecting agricultural data and determining agricultural themes of the agricultural data; Based on the theme-indicator set mapping relationship, determine the first target indicator set corresponding to the agricultural theme; Based on each indicator in the first target indicator set, the agricultural data is classified according to the indicator to obtain a plurality of classified data; The plurality of classified data are combined to obtain the agricultural data; Classify and store the multiple classified data according to the indicators; The storing of the plurality of classified data according to the indicators further includes: receiving a topic selection instruction, and determining an agricultural topic to be searched based on the topic selection instruction; Based on the topic-indicator set mapping relationship, determine the second target indicator set corresponding to the agricultural topic to be retrieved; Display each indicator in the second target indicator set; receiving an indicator selection instruction, and determining an indicator to be retrieved based on the indicator selection instruction; Based on the index to be retrieved, retrieve the stored target agricultural data; Also includes: Collect target data of target data sources storing agricultural data; Extracting keywords from the target data to obtain keywords; Based on the keyword-field mapping relationship, determine the table field of the Hive table corresponding to the keyword; Based on the table fields, storing the target data; Also includes: Get information configuration instructions acting on the configuration information input interface; Based on the information configuration instruction, determining at least two configuration information; Based on the at least two pieces of configuration information, a data synchronization code script is constructed, where the data synchronization code script is used to import data or export data; The at least two pieces of configuration information include a resource name of the data resource to be synchronized and a data source type of an external data source, wherein the external data source is a data source of an original storage location of the data to be imported or a data source of a target storage location of the data to be exported; the resource name is used as a script name of the data synchronization code script; The step of constructing a data synchronization code script based on the at least two pieces of configuration information, wherein the data synchronization code script is used to import data or export data, includes: Determine a data synchronization type and a plurality of preset configuration items corresponding to the data synchronization type, wherein the data synchronization type includes data import or data export, and the plurality of preset configuration items are configuration items required for building a script; Matching the at least two pieces of configuration information with the plurality of preset configuration items; If at least one preset configuration item that does not match the at least two pieces of configuration information is determined from the plurality of preset configuration items, determining configuration information corresponding to the at least one preset configuration item based on the data source type; Building a data synchronization code script based on the at least two pieces of configuration information and the configuration information corresponding to the at least one preset configuration item; If no preset configuration item that does not match the at least two pieces of configuration information is determined from the multiple preset configuration items, a script is constructed based on the at least two pieces of configuration information to obtain a data synchronization code script.

2. The method for managing agricultural big data according to claim 1, characterized in that: Also includes: Based on the association relationships between the multiple data tables and the primary key fields of each data table in the multiple data tables, a data table lineage map is generated.

3. The method for managing agricultural big data according to claim 1, characterized in that: Also includes: receiving a theme setting instruction, and determining a target data table to be set and a theme to be set based on the theme setting instruction; Setting the subject of the target data table as the subject to be set; Receiving a data table search instruction, and determining a subject to be searched based on the data table search instruction; Based on the subject to be searched, a data table with the subject being the subject to be searched is obtained.

4. The method for managing agricultural big data according to claim 1, characterized in that: Also includes: Obtain user management instructions for agricultural big data, and determine the functional authority and data authority of the user corresponding to the user management instructions; Based on the function permission and the data permission, determining whether the operation indicated by the user management instruction can be implemented; The functional permission is used to determine whether the function can be used, and the data permission is used to determine the scope of use of the permission.

5. A management device for agricultural big data, characterized in that: include: A theme determination module, used for collecting agricultural data and determining the agricultural theme of the agricultural data; An indicator determination module, used to determine a first target indicator set corresponding to the agricultural theme based on a theme-indicator set mapping relationship; A data classification module, used for classifying the agricultural data according to the indicators in the first target indicator set to obtain a plurality of classified data; The plurality of classified data are combined to obtain the agricultural data; A classification storage module, used for classifying and storing the plurality of classification data according to indicators; The device also includes a data retrieval module, which is used to: receiving a topic selection instruction, and determining an agricultural topic to be searched based on the topic selection instruction; Based on the topic-indicator set mapping relationship, determine the second target indicator set corresponding to the agricultural topic to be retrieved; Display each indicator in the second target indicator set; receiving an indicator selection instruction, and determining an indicator to be retrieved based on the indicator selection instruction; Based on the index to be retrieved, retrieve the stored target agricultural data; The device further comprises a data storage module, wherein the data storage module is used for: Collect target data of target data sources storing agricultural data; Extracting keywords from the target data to obtain keywords; Based on the keyword-field mapping relationship, determine the table field of the Hive table corresponding to the keyword; Based on the table fields, storing the target data; Also includes: An acquisition module, used to acquire information configuration instructions acting on the configuration information input interface; A determination module, configured to determine at least two pieces of configuration information based on the information configuration instruction; A construction module, used to construct a data synchronization code script based on the at least two pieces of configuration information, wherein the data synchronization code script is used to import data or export data; The at least two pieces of configuration information include a resource name of the data resource to be synchronized and a data source type of an external data source, wherein the external data source is a data source of an original storage location of the data to be imported or a data source of a target storage location of the data to be exported; the resource name is used as a script name of the data synchronization code script; The step of constructing a data synchronization code script based on the at least two pieces of configuration information, wherein the data synchronization code script is used to import data or export data, includes: Determine a data synchronization type and a plurality of preset configuration items corresponding to the data synchronization type, wherein the data synchronization type includes data import or data export, and the plurality of preset configuration items are configuration items required for building a script; Matching the at least two pieces of configuration information with the plurality of preset configuration items; If at least one preset configuration item that does not match the at least two pieces of configuration information is determined from the plurality of preset configuration items, determining configuration information corresponding to the at least one preset configuration item based on the data source type; Building a data synchronization code script based on the at least two pieces of configuration information and the configuration information corresponding to the at least one preset configuration item; If no preset configuration item that does not match the at least two pieces of configuration information is determined from the multiple preset configuration items, a script is constructed based on the at least two pieces of configuration information to obtain a data synchronization code script.

6. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the method for managing agricultural big data as described in any one of claims 1 to 4 is implemented.

7. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for managing agricultural big data as described in any one of claims 1 to 4 is implemented.

Images