Data acquisition method and device, equipment and storage medium

Abstract

The invention discloses a data collection method and device, equipment and a storage medium, and relates to the technical field of data processing, and the method comprises the steps: creating an HDFS data source and a Hive data source, creating a target class, compiling an evaluate function in the target class, and compiling the compiled target class to obtain a compiled file; determining a resource uploading path according to the HDFS data source, uploading the compiled file to the resource uploading path, and displaying and managing resource information; creating a user-defined function based on the function name, the function class path, the compiling file and the Hive data source, and determining a target database authorized by the user-defined function; a Hive structured query language is written by using a user-defined function, the function of the Hive structured query language is tested, and after the test is passed, data is collected from a Hive data source by executing the Hive structured query language. Therefore, the flexibility and efficiency of data acquisition are improved.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a data acquisition method, device, equipment and storage medium. Background Art

[0002] In the field of big data processing and analysis, data collection is a crucial link. Traditional data collection methods often rely on fixed data formats and interfaces, which are inadequate when dealing with complex and changing data sources. With the rise of big data processing technologies such as Hadoop and Hive, how to efficiently and flexibly collect and process data has become the focus of the industry.

[0003] As a data warehouse tool based on Hadoop (an open source software framework), Hive provides users with a query language HQL (Hive Query Language, hibernate query language) similar to SQL (Structured Query Language), allowing users to easily query and analyze data stored in the Hadoop cluster. However, Hive itself provides limited built-in functions and features, which cannot meet the diverse data processing needs of users. Therefore, Hive provides a user-defined function (UDF, User Defined Function) mechanism, allowing users to write custom functions according to their own needs and use them in Hive queries.

[0004] On the other hand, NIFI (NiagaraFiles), as a powerful open source data integration tool, provides a variety of data processors and transmitters, which can easily realize data collection, conversion, loading and distribution. NIFI's graphical user interface and high configurability allow users to easily build data flows and implement complex data processing processes.

[0005] However, existing data collection methods either rely on fixed data formats and interfaces, or require users to write a lot of code to implement data processing and transmission. These methods are not only inefficient, but also difficult to adapt to complex and changing data sources. Summary of the invention

[0006] In view of this, the purpose of the present invention is to provide a data collection method, device, equipment and storage medium, which can quickly respond to changing business needs, efficiently collect and process data from Hive, and can be applied to complex data analysis scenarios. The specific scheme is as follows:

[0007] In a first aspect, the present application discloses a data collection method, comprising:

[0008] Create an HDFS data source and a Hive data source, create a target class according to user requirements, write an evaluate function in the target class, and compile the corresponding compiled target class to obtain a compiled file;

[0009] Determine a resource upload path according to the HDFS data source, upload the compiled file to the resource upload path, and display and manage resource information corresponding to the uploaded compiled file;

[0010] Create a user-defined function based on a predefined function name, a predefined function class path, the compiled file, and the Hive data source, and determine a target database authorized by the user-defined function in the Hive data source;

[0011] The Hive structured query language is written by using the user-defined function, and the function of the Hive structured query language is tested based on the target database. After the test passes, data is collected from the Hive data source by executing the Hive structured query language based on NIFI.

[0012] Optionally, the target class inherits the custom function org.apache.hadoop.hive.ql.UDF class in Hive.

[0013] Optionally, after determining the resource upload path according to the HDFS data source, the method further includes:

[0014] Defining a resource name of the compiled file according to a function corresponding to the compiled file, so as to manage and query the resource information based on the resource name;

[0015] A resource description of the compiled file is defined according to a function function corresponding to the compiled file, so as to determine the function of the compiled file based on the resource description.

[0016] Optionally, the displaying and managing resource information corresponding to the uploaded compiled file includes:

[0017] The resource information corresponding to the uploaded compiled file is displayed on a preset page, and the resource information is managed; the management includes any one or a combination of viewing, editing, deleting and downloading operations.

[0018] Optionally, after testing the function of the Hive structured query language based on the target database, the method further includes:

[0019] If the function of the Hive structured query language is abnormal, abnormal information is displayed so that the user can modify the Hive structured query language based on the abnormal information.

[0020] Optionally, collecting data from the Hive data source by executing the Hive structured query language based on NIFI includes:

[0021] A target component is developed based on NIFI, and the Hive data source is connected and the Hive structured query language is executed through the operation of the target component to complete data collection.

[0022] Optionally, after collecting data from the Hive data source by executing the Hive structured query language based on NIFI, the method further includes:

[0023] Use NIFI to process, convert or distribute the collected data.

[0024] In a second aspect, the present application discloses a data collection device, comprising:

[0025] The compilation file acquisition module is used to create an HDFS data source and a Hive data source, create a target class according to user requirements, write an evaluate function in the target class, and compile the corresponding written target class to obtain a compilation file;

[0026] A display and management module, used to determine a resource upload path according to the HDFS data source, upload the compiled file to the resource upload path, and display and manage resource information corresponding to the uploaded compiled file;

[0027] A target database determination module is used to create a user-defined function based on a predefined function name, a predefined function class path, the compiled file, and the Hive data source, and determine a target database authorized by the user-defined function in the Hive data source;

[0028] The data collection module is used to write Hive structured query language using the user-defined function, test the function of the Hive structured query language based on the target database, and after the test passes, collect data from the Hive data source by executing the Hive structured query language based on NIFI.

[0029] In a third aspect, the present application discloses an electronic device, including:

[0030] Memory, used to store computer programs;

[0031] The processor is used to execute the computer program to implement the data collection method as described above.

[0032] In a fourth aspect, the present application discloses a computer-readable storage medium for storing a computer program, wherein the computer program implements the aforementioned data acquisition method when executed by a processor.

[0033] When collecting data, this application first creates an HDFS data source and a Hive data source, creates a target class according to user needs, writes the evaluate function in the target class, and compiles the corresponding written target class to obtain a compiled file; then determines the resource upload path according to the HDFS data source, uploads the compiled file to the resource upload path, and displays and manages the resource information corresponding to the uploaded compiled file; creates a user-defined function based on the pre-defined function name, the pre-defined function class path, the compiled file and the Hive data source, and determines the target database authorized by the user-defined function in the Hive data source; finally, uses the user-defined function to write the Hive structured query language, tests the function of the Hive structured query language based on the target database, and after the test passes, collects data from the Hive data source by executing the Hive structured query language based on NIFI. It can be seen that this application allows users to write custom functions to meet specific data processing requirements by creating HDFS and Hive data sources, and uploads the compiled custom function resources to HDFS for unified management. Then, through the creation and authorization of custom functions, users can create and authorize target databases in Hive to achieve cross-library sharing. Users can use user-defined functions to write Hive structured query language to query and test functions based on business needs. Based on NIFI, the present invention allows users to customize development components to reference the tested Hive structured query language and collect data from Hive by executing these statements. In this way, the application improves the flexibility and efficiency of data collection, reduces development costs, and simplifies the data collection process, making the process of extracting data from a large data warehouse more automated and efficient, and providing users with powerful data processing capabilities. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0035] Figure 1A flow chart of a data collection method disclosed in this application;

[0036] Figure 2 A flow chart of a specific data collection method disclosed in this application;

[0037] Figure 3 This is a structural schematic diagram of a data acquisition device disclosed in this application;

[0038] Figure 4 This is a structural diagram of an electronic device disclosed in this application. DETAILED DESCRIPTION

[0039] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0040] Existing data collection methods either rely on fixed data formats and interfaces, or require users to write a lot of code to implement data processing and transmission. These methods are not only inefficient, but also difficult to adapt to complex and changeable data sources. In order to solve the above technical problems, the present application discloses a data collection method that can quickly respond to changing business needs, efficiently collect and process data from Hive, and can be applied to complex data analysis scenarios.

[0041] See also Figure 1 As shown, an embodiment of the present invention discloses a data collection method, including:

[0042] Step S11, create an HDFS data source and a Hive data source, create a target class according to user requirements, and write an evaluate function in the target class, and compile the corresponding written target class to obtain a compiled file.

[0043] In this embodiment, first, an HDFS data source is created for subsequent JAR (Java Archive, a software package file format) file resource upload, and a Hive data source is created for subsequent function creation and use. In a specific embodiment, Hive contains three databases, db1, db2, and db3. Then, a target class is created according to user needs, and the evaluate function is written in the target class. It should be noted that the target class inherits the custom function org.apache.hadoop.hive.ql.UDF class in Hive. Finally, the corresponding written target class is compiled to obtain a compiled file. Specifically, assuming that the user's specific requirement is to calculate the distance between two places based on two longitudes and latitudes, it is necessary to write a new class that inherits the user-defined function org.apache.hadoop.hive.ql.UDF class in Hive; according to user-defined requirements, the evaluate function is rewritten in the new class. The specific contents are as follows:

[0044] package udf;

[0045] import org.apache.hadoop.hive.ql.exec.UDF;

[0046] import utils.ComputationUtil;

[0047] / ***

[0048] * Write registration udf

[0049] * udf to calculate distance between two longitude and latitude

[0050] * * /

[0051] public class calDistance extends UDF {

[0052] public double evaluate(double lon1, double lat1, double lon2,double lat 2) {

[0053] return ComputationUtil.getDistance(lon1, lat1, lon2, lat 2);

[0054] }

[0055] }.

[0056] After the final writing is completed, compile the project into calDistance.jar.

[0057] Step S12: determine a resource upload path according to the HDFS data source, upload the compiled file to the resource upload path, and display and manage resource information corresponding to the uploaded compiled file.

[0058] In this embodiment, the resource upload path is determined according to the HDFS data source, and the compiled file is uploaded to the resource upload path. After the resource upload path is determined according to the HDFS data source, it also includes: defining the resource name of the compiled file according to the function function corresponding to the compiled file, so as to manage and query the resource information based on the resource name; defining the resource description of the compiled file according to the function function corresponding to the compiled file, so as to determine the function of the compiled file based on the resource description. In addition, the present application can also display and manage the resource information corresponding to the uploaded compiled file. That is, the resource information corresponding to the uploaded compiled file is displayed on the preset page, and the resource information is managed; management includes any one or a combination of viewing, editing, deleting and downloading operations. Specifically, first select the HDFS data source created in step S11, and further select the resource upload path. In this embodiment, the / usr / lib directory is selected. Then define the function resource name and description: according to the actual function of the function corresponding to calDistance.jar, enter the "resource name" as calDistance on the page for subsequent resource management and query; enter the "resource description" as "calculate the distance according to the two input longitudes and latitudes" on the page for the functional description of calDistance.jar. Then select and upload the function resource. Select calDistance.jar generated in step S11 and upload it to the selected upload path / usr / lib. Finally, the page can display the function resource information that has been successfully uploaded, and can be viewed, edited, deleted and downloaded.

[0059] Step S13: Create a user-defined function based on the predefined function name, the predefined function class path, the compiled file, and the Hive data source, and determine a target database authorized by the user-defined function in the Hive data source.

[0060] In this embodiment, when creating a Hive user-defined function, a user-defined function can be created based on a predefined function name, a predefined function class path, a compiled file, and a Hive data source. It should be noted that the user-defined function writing step involves the implementation of specific business logic, such as calculation, text processing, data format conversion, etc., to enhance the flexibility and adaptability of data processing. In a specific embodiment, the function name is first defined as cal_distance. This name is the function name actually used in subsequent SQL execution. Secondly, the function class path is defined as package name.class name, that is, udf.calDistance. Select the uploaded function resource calDistance and the created Hive data source again. Finally, after selecting Create, the CREATE FUNCTION command will be automatically assembled to create and register the Hive user-defined function. Then determine the target database authorized by the user-defined function in the Hive data source. For example, the function cal_distance is authorized to the Hive database db1. In this way, access rights are assigned to Hive user-defined functions in a specific database to ensure data security and compliance.

[0061] Step S14: Use the user-defined function to write Hive structured query language, test the function of the Hive structured query language based on the target database, and after the test passes, collect data from the Hive data source by executing the Hive structured query language based on NIFI.

[0062] In this embodiment, after determining the target database authorized by the user-defined function in the Hive data source, the user-defined function can be used to write the Hive structured query language, and then the function of the Hive structured query language is tested based on the target database. If the function of the Hive structured query language is abnormal, the exception information is displayed so that the user can modify the Hive structured query language based on the exception information. If the test passes, data is collected from the Hive data source based on NIFI by executing the Hive structured query language. Specifically, the function cal_distance is authorized to the Hive database db1. The database db1 contains the data table table1. The table field place_name represents the location name, the table fields longitude1 and latitude1 represent the longitude and latitude of the first location, and the table fields longitude2 and latitude2 represent the longitude and latitude of the second location. Use the function cal_distance to write a query SQL to obtain the location and the corresponding distance:

[0063] SELECT place_name, cal_distance(longitude1,latitude1,longitude2,latitude2) FROM table1.

[0064] Select the Hive data source corresponding to database db1 to perform SQL function test. If the function is abnormal, the user can view the abnormal information to facilitate the user to modify the SQL accordingly. Then, based on NIFI, custom develop the SelectHiveSql component to collect Hive data. The component logic is as follows:

[0065] 1. Select the created Hive data source;

[0066] 2. Reference the Hive SQL task that passed the test;

[0067] 3. After the component is running, you can connect to the Hive data source and execute the Hive SQL task to collect Hive data. After the data is collected, you can use other NIFI components for subsequent data processing and data writing. Specifically, after the data is collected, it is combined with the NIFI data flow design to realize automatic processing, conversion and distribution of data after collection, thereby improving the overall efficiency of the data processing link.

[0068] In this way, combined with NIFI's powerful data stream processing capabilities, it supports immediate processing and distribution of data after collection, accelerating the data analysis cycle.

[0069] It can be seen that this application allows users to write custom functions to meet specific data processing needs by creating HDFS and Hive data sources, and upload the compiled custom function resources to HDFS for unified management. Then, through the creation and authorization of custom functions, users can create and authorize target databases in Hive to achieve cross-database sharing. Users can use user-defined functions to write Hive structured query language to query and test functions in accordance with business needs. Based on NIFI, the present invention allows users to customize development components to reference the tested Hive structured query language, and collect data from Hive by executing these statements. In this way, the present application improves the flexibility and efficiency of data collection, reduces development costs, and simplifies the data collection process, making the process of extracting data from a large data warehouse more automated and efficient, and providing users with powerful data processing capabilities.

[0070] Based on the above embodiment, it can be seen that the present application discloses a data collection method, which can improve the flexibility and efficiency of data collection, reduce development costs, and simplify the data collection process, making the process of extracting data from a big data warehouse more automated and efficient. Next, a detailed description of the specific data collection process will be given.

[0071] like Figure 2 As shown, this application first creates an HDFS data source for connecting to HDFS and a Hive data source for connecting to Hive. Then create a new Java class that inherits from org.apache.hadoop.hive.ql.exec.UDF. In this class, rewrite the evaluate function in the new class according to user-defined requirements. This method will process the function's input and return the result. After writing, compile the project into a JAR file.

[0072] After the project is compiled, use the HDFS data source to connect to HDFS and upload the compiled JAR file to the specified HDFS directory. After the upload is successful, you can view, edit, download, and delete JAR file resources through the UI (User Interface).

[0073] After that, define the function name, define the function class path, and then select the uploaded JAR file and the created Hive data source. After selecting Create, the CREATE FUNCTION command will be automatically assembled to create and register the user-defined function.

[0074] Next, specify which databases the user-defined function is allowed to share across. For example, Hive contains three databases, db1, db2, and db3. If the function is authorized to all databases, the function can be used in any database; if the function is only authorized to db1 and db2, the function can be used in db1 and db2 databases, but not in db3 database.

[0075] If the function is created and authorized successfully, you can use the function and write Hive SQL to implement the query according to specific business needs. This module supports selecting the Hive data source and performing SQL functional testing after the SQL is written. At the same time, based on the powerful open source data integration tool NIFI, the custom development component implementation can reference the SQL statement in the SQL writing test module. Finally, by executing SQL query statements to collect Hive source data, data processing and data writing can be performed later. This completes the data collection.

[0076] It can be seen that this application can directly meet specific business logic requirements through custom functions and improve the customization of data processing. Through the seamless integration of NIFI components and Hive UDF, the process of extracting data from the big data warehouse is more automated and efficient. And combined with NIFI's powerful data stream processing capabilities, it supports instant processing and distribution of data after collection, accelerating the data analysis cycle.

[0077] See also Figure 3 As shown, an embodiment of the present invention discloses a data acquisition device, including:

[0078] The compilation file acquisition module 11 is used to create an HDFS data source and a Hive data source, create a target class according to user requirements, and write an evaluate function in the target class, and compile the corresponding written target class to obtain a compilation file;

[0079] The display and management module 12 is used to determine the resource upload path according to the HDFS data source, upload the compiled file to the resource upload path, and display and manage the resource information corresponding to the uploaded compiled file;

[0080] A target database determination module 13 is used to create a user-defined function based on a predefined function name, a predefined function class path, the compiled file, and the Hive data source, and determine a target database authorized by the user-defined function in the Hive data source;

[0081] The data collection module 14 is used to write Hive structured query language using the user-defined function, test the function of the Hive structured query language based on the target database, and after the test passes, collect data from the Hive data source by executing the Hive structured query language based on NIFI.

[0082] It can be seen that this application allows users to write custom functions to meet specific data processing needs by creating HDFS and Hive data sources, and upload the compiled custom function resources to HDFS for unified management. Then, through the creation and authorization of custom functions, users can create and authorize target databases in Hive to achieve cross-database sharing. Users can use user-defined functions to write Hive structured query language to query and test functions in accordance with business needs. Based on NIFI, the present invention allows users to customize development components to reference the tested Hive structured query language, and collect data from Hive by executing these statements. In this way, the present application improves the flexibility and efficiency of data collection, reduces development costs, and simplifies the data collection process, making the process of extracting data from a large data warehouse more automated and efficient, and providing users with powerful data processing capabilities.

[0083] In some specific embodiments, the device may further include:

[0084] A resource name definition module, used to define the resource name of the compiled file according to the function corresponding to the compiled file, so as to manage and query the resource information based on the resource name;

[0085] The resource description definition module is used to define the resource description of the compiled file according to the function function corresponding to the compiled file, so as to determine the function of the compiled file based on the resource description.

[0086] In some specific embodiments, the display and management module 12 may specifically include:

[0087] The display and management unit is used to display the resource information corresponding to the uploaded compiled file on a preset page and manage the resource information; the management includes any one or a combination of viewing, editing, deleting and downloading operations.

[0088] In some specific embodiments, the device may further include:

[0089] The abnormal information display module is used to display abnormal information if the function of the Hive structured query language is abnormal, so that the user can modify the Hive structured query language based on the abnormal information.

[0090] In some specific embodiments, the data acquisition module 14 may specifically include:

[0091] The data collection unit is used to develop a target component based on NIFI, connect to the Hive data source and execute the Hive structured query language through the operation of the target component to complete data collection.

[0092] In some specific embodiments, the device may further include:

[0093] The data operation module is used to use NIFI to perform data processing, data conversion or data distribution operations on the collected data.

[0094] Furthermore, the embodiment of the present application also discloses an electronic device, Figure 4 This is a structural diagram of an electronic device 20 according to an exemplary embodiment. The content in the diagram cannot be regarded as any limitation on the scope of use of the present application.

[0095] Figure 4 A schematic diagram of the structure of an electronic device 20 provided in an embodiment of the present application. The electronic device 20 may specifically include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 is used to store a computer program, which is loaded and executed by the processor 21 to implement the relevant steps in the data acquisition method disclosed in any of the aforementioned embodiments. In addition, the electronic device 20 in this embodiment may specifically be an electronic computer.

[0096] In this embodiment, the power supply 23 is used to provide working voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and the external device, and the communication protocol it follows is any communication protocol that can be applied to the technical solution of the present application, and is not specifically limited here; the input and output interface 25 is used to obtain external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs and is not specifically limited here.

[0097] In addition, the memory 22 as a carrier for storing resources may be a read-only memory, a random access memory, a disk or an optical disk, etc. The resources stored thereon may include an operating system 221, a computer program 222, etc., and the storage method may be temporary storage or permanent storage.

[0098] The operating system 221 is used to manage and control the hardware devices and computer programs 222 on the electronic device 20, and can be Windows Server, Netware, Unix, Linux, etc. In addition to including a computer program that can be used to complete the data collection method performed by the electronic device 20 disclosed in any of the aforementioned embodiments, the computer program 222 can further include a computer program that can be used to complete other specific tasks.

[0099] Furthermore, the present application also discloses a computer-readable storage medium for storing a computer program; wherein the computer program, when executed by a processor, implements the aforementioned disclosed data acquisition method. The specific steps of the method can refer to the corresponding contents disclosed in the aforementioned embodiments, and will not be repeated here.

[0100] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0101] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0102] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0103] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0104] The technical solution provided by the present application is introduced in detail above. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea. At the same time, for general technicians in this field, according to the idea of ​​the present application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A data collection method, characterized in that: include: Create an HDFS data source and a Hive data source, create a target class according to user requirements, write an evaluate function in the target class, and compile the corresponding compiled target class to obtain a compiled file; Determine a resource upload path according to the HDFS data source, upload the compiled file to the resource upload path, and display and manage resource information corresponding to the uploaded compiled file; Create a user-defined function based on a predefined function name, a predefined function class path, the compiled file, and the Hive data source, and determine a target database authorized by the user-defined function in the Hive data source; The Hive structured query language is written by using the user-defined function, and the function of the Hive structured query language is tested based on the target database. After the test passes, data is collected from the Hive data source by executing the Hive structured query language based on NIFI.

2. The data collection method according to claim 1, characterized in that: The target class inherits the custom function org.apache.hadoop.hive.ql.UDF class in Hive.

3. The data collection method according to claim 1, characterized in that: After determining the resource upload path according to the HDFS data source, the method further includes: Defining a resource name of the compiled file according to a function corresponding to the compiled file, so as to manage and query the resource information based on the resource name; A resource description of the compiled file is defined according to a function function corresponding to the compiled file, so as to determine the function of the compiled file based on the resource description.

4. The data collection method according to claim 1, characterized in that: The display and management of resource information corresponding to the uploaded compiled file includes: The resource information corresponding to the uploaded compiled file is displayed on a preset page, and the resource information is managed; the management includes any one or a combination of viewing, editing, deleting and downloading operations.

5. The data collection method according to claim 1, characterized in that: After the function of the Hive structured query language is tested based on the target database, the method further includes: If the function of the Hive structured query language is abnormal, abnormal information is displayed so that the user can modify the Hive structured query language based on the abnormal information.

6. The data collection method according to claim 1, characterized in that: The collecting data from the Hive data source by executing the Hive structured query language based on NIFI includes: A target component is developed based on NIFI, and the Hive data source is connected and the Hive structured query language is executed through the operation of the target component to complete data collection.

7. The data collection method according to any one of claims 1 to 6, characterized in that: After collecting data from the Hive data source by executing the Hive structured query language based on NIFI, the method further includes: Use NIFI to process, convert or distribute the collected data.

8. A data acquisition device, characterized in that: include: The compilation file acquisition module is used to create an HDFS data source and a Hive data source, create a target class according to user requirements, write an evaluate function in the target class, and compile the corresponding written target class to obtain a compilation file; A display and management module, used to determine a resource upload path according to the HDFS data source, upload the compiled file to the resource upload path, and display and manage resource information corresponding to the uploaded compiled file; A target database determination module is used to create a user-defined function based on a predefined function name, a predefined function class path, the compiled file, and the Hive data source, and determine a target database authorized by the user-defined function in the Hive data source; The data collection module is used to write Hive structured query language using the user-defined function, test the function of the Hive structured query language based on the target database, and after the test passes, collect data from the Hive data source by executing the Hive structured query language based on NIFI.

9. An electronic device, characterized in that: include: Memory, used to store computer programs; A processor, configured to execute the computer program to implement the data collection method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: Used to store a computer program, wherein when the computer program is executed by a processor, the data collection method according to any one of claims 1 to 7 is implemented.

Images