A method and device for off-line data collection
By generating configuration files of the engine and scheduling framework, the collection of structured, semi-structured and unstructured data is realized, solving the problems of data acquisition complexity and insufficient monitoring in the existing technology, and providing user-friendly data acquisition Solution.
Patent Information
- Application Number
- CN202011024841.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2020-09-25
- Publication Date
- 2024-11-22
- Estimated Expiration
- 2040-09-25
AI Technical Summary
Existing data acquisition systems cannot effectively collect structured, semi-structured and unstructured data, and lack user-friendly visual configuration and monitoring management functions, which cannot meet the complex needs of big data acquisition.
An offline data acquisition method and device is designed. The configuration files can be identified by the generation engine and the scheduling framework can be identified, so as to realize the visual configuration and monitoring of data acquisition tasks, and support structured, semi-structured and unstructured data acquisition. , and provide detailed collection status information display.
It realizes visual configuration and perfect management and monitoring of the data acquisition process, adapts to the collection of multiple data types, and improves the user experience and data acquisition efficiency.
Smart Images

Figure CN112231376B_ABST
Abstract
Description
Technical Field
[0001] The subject matter disclosed herein generally relates to the field of computers, and more particularly to a method for off-line data acquisition and an apparatus using the method. Background Art
[0002] At present, with the strong support of national policies, big data technology has been widely used in government and enterprise projects. The key technologies of big data processing generally include: big data collection, big data preprocessing, big data storage and management, big data analysis and mining, big data presentation and application (big data retrieval, big data visualization, big data application, big data security, etc.). However, as the fields involved in big data become more and more extensive and the business handled becomes more and more in-depth, the types of data to be collected and processed are becoming more and more complex and diverse. Therefore, in the context of the big data era, how to collect useful information from big data has become one of the key factors in the development of big data.
[0003] The characteristics of traditional data collection are single source, small data volume compared to big data, single structure, relational database and parallel data warehouse. The characteristics of big data data collection are wide source, huge data volume, rich data types, including structured, semi-structured, unstructured, and distributed databases.
[0004] New methods for big data collection include system log collection methods and network data collection methods. Many Internet companies have their own massive data collection tools, which are mostly used for system log collection. These tools all adopt a distributed architecture and can meet the needs of log data collection and transmission of hundreds of MB per second. Network data collection refers to obtaining data information from a website through web crawlers or website public APIs. This method can extract unstructured data from web pages, store it as a unified local data file, and store it in a structured manner. For data with high confidentiality requirements such as enterprise production and operation data or subject research data, big data collection can also be carried out by cooperating with enterprises or research institutions and using specific system interfaces and other related methods.
[0005] In recent years, the data types of government and corporate customers have changed from traditional relational data to non-relational data, which has the characteristics of high speed, large capacity, diversity and high value. This has brought new challenges to big data collection. However, data collection at this stage cannot meet the new collection needs. For example, data collection still faces problems such as the data type of collected data is relatively single, the collection process is relatively complicated, it cannot provide a friendly user experience, it cannot encapsulate the data collection process well, there is a lack of monitoring of the collection process, and it cannot provide users with more detailed collection status information.
[0006] In view of the defects of the existing data collection system, it is necessary to design a new data collection system to solve the current problems related to data collection in the industry. The new design needs to realize the collection of structured data, semi-structured data and unstructured data, and users can also visualize the configuration function of the collection process, and provide convenient monitoring and management, providing a new solution for big data collection.
[0007] Therefore, the main purpose of the present invention is to provide users with a method for visually configuring the data collection process and improving the management and monitoring method to overcome the defects of the prior art.
[0008] The following terms are defined herein, at least some of which will be referred to in the following description:
[0009] Data acquisition (DAQ): Also known as data acquisition, it refers to the process of automatically collecting information from analog and digital units under test, such as sensors and other devices under test.
[0010] DataX: An offline synchronization tool for heterogeneous data sources, dedicated to achieving stable and efficient data synchronization between various heterogeneous data sources including relational databases.
[0011] Flume: A distributed, highly reliable, and highly available system that collects, aggregates, and moves large quantities of log data from different data sources to data centers for storage. It is a tool for log collection and aggregation.
[0012] ETL (Extract-Transform-Load): is used to describe the process of extracting, transforming, and loading data from the source to the destination.
[0013] Etlworks: In this article, it refers to an ETL tool developed by Etlworks.
[0014] HDFS (Hadoop Distributed File System): A distributed file system in the Hadoop technology framework that manages files deployed on multiple independent physical machines.
[0015] MySQ: A relational database management system developed by MySQL AB of Sweden.
[0016] Oracle: This article refers to a database product developed by Oracle Corporation.
[0017] Sqoop: Apache's tool for transferring data between Hadoop and relational database servers. Summary of the invention
[0018] The present invention provides a method and a device for collecting discrete data.
[0019] In one embodiment, a method for discrete data collection includes: setting collection task information; generating an engine-recognizable configuration file and a scheduling framework-recognizable configuration file based on the collection task information; the scheduling framework calling the engine to execute the task based on the engine-recognizable configuration file and the scheduling framework-recognizable configuration file; the engine storing task-related logs and other indicator information; and displaying the stored task-related logs and other indicator information.
[0020] Preferably, the method further comprises: configuring address information and user information of the source-end data source and the target-end data source; and selecting the source-end data source and the target-end data source based on the address information and the user information.
[0021] Preferably, the method further comprises: configuring data extraction rules.
[0022] Preferably, the method further comprises: configuring flow control information.
[0023] Optionally, the method further comprises: configuring a scheduling frequency and an engine to be used to execute the task.
[0024] Preferably, the method further includes: scanning the newly generated acquisition task configuration; parsing the acquisition task configuration, and generating the engine-recognizable configuration file according to a predefined engine configuration template; and parsing the acquisition task configuration, and generating the scheduling framework-recognizable configuration file according to the scheduling frequency and the engine-recognizable configuration file.
[0025] Preferably, the method further comprises: the scheduling framework scans and generates a scheduling framework-recognizable configuration file; the scheduling framework triggers a scheduling task according to the scheduling framework-recognizable configuration file; and the engine executes a data collection task according to the scheduling task.
[0026] According to another embodiment of the present invention, a device for discrete data acquisition includes: a communication unit, configured to perform data sending and receiving; a display unit, configured to display data; a memory, configured to store data and instructions; a processor, which is operably coupled to the communication unit, the communication unit and the memory, and is configured to: configure acquisition task information; generate an engine-recognizable configuration file and a scheduling framework-recognizable configuration file based on the acquisition task information; control the scheduling framework to call the engine to execute the task according to the engine-recognizable configuration file and the scheduling framework-recognizable configuration file; control the engine to store task-related logs and other indicator information in the memory; and control the display unit to display based on the stored task-related logs and other indicator information.
[0027] According to yet another embodiment of the present invention, a computer readable medium has a computer program stored thereon, wherein when the computer program is executed by a processor, the processor is caused to perform the method for discrete data acquisition according to the present invention.
[0028] By using the method and device proposed in the present invention, users can visually configure the data collection process and improve management and monitoring.
[0029] The technical effects that can be obtained by the present invention are not limited to the above. Other beneficial effects can be understood by those skilled in the art through practicing the present invention. BRIEF DESCRIPTION OF THE DRAWINGS
[0030] A more particular description of the embodiments briefly described above will be presented by reference to specific embodiments illustrated in the accompanying drawings. Understanding that these drawings depict only some embodiments and are therefore not to be considered limiting of scope, the embodiments will be described and explained with additional specificity and detail through use of the accompanying drawings.
[0031] Figure 1 is a block diagram depicting a system of DataX as a prior art.
[0032] Figure 2 : is a block diagram depicting a Flume system as a prior art.
[0033] Figure 3 is a block diagram depicting a discrete data acquisition system according to one embodiment of the present invention.
[0034] Figure 4 is a flow chart depicting a discrete data acquisition method according to one embodiment of the present invention.
[0035] Figure 5 is a flow chart depicting a discrete data collection method according to another embodiment of the present invention.
[0036] Figure 6 is a flow chart depicting a discrete data collection method according to yet another embodiment of the present invention.
[0037] Figure 7 is a block diagram illustrating an apparatus for performing a discrete data acquisition method according to an embodiment of the present invention. DETAILED DESCRIPTION
[0038] As will be appreciated by those skilled in the art, aspects of the embodiments may be embodied as systems, devices, methods, or program products. Thus, the embodiments may take the form of fully hardware embodiments, fully software embodiments, or embodiments combining software and hardware aspects. Furthermore, the embodiments may take the form of program products embodied in one or more computer-readable storage devices.
[0039] Certain functional units described in this specification may be labeled as modules in order to more specifically emphasize their implementation independence. For example, a module may be implemented as a hardware circuit including a customized VLSI or gate array, off-the-shelf semiconductor such as a logic chip, transistor, or other discrete component. A module may also be implemented in a programmable hardware device such as a field programmable gate array, programmable array logic, a programmable logic device, etc. A module may also be implemented in code and / or software to be executed by various types of processors.
[0040] Any combination of one or more computer readable media may be utilized. The computer readable medium may be a computer readable storage medium. The computer readable storage medium may be a storage device storing code. The storage device may be, for example, but not necessarily limited to, an electronic, magnetic, optical, electromagnetic, infrared, holographic, micromechanical or semiconductor system, apparatus or device, or any suitable combination of the foregoing.
[0041] A non-exhaustive list of more specific examples of storage devices would include the following: an electrical connection having one or more wires, a portable computer disk, a hard disk, a random access memory ("RAM"), a read-only memory ("ROM"), an erasable programmable read-only memory ("EPROM" or "flash memory"), a portable compact disc read-only memory ("CD-ROM"), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of this document, a computer-readable storage medium may be any tangible medium that can contain or store a program for use by or in connection with an instruction execution system, apparatus, or device.
[0042] The code for performing the operations of the embodiments may be any number of lines and may be written in any combination of one or more programming languages including object-oriented programming languages such as Python, Ruby, Java, Smalltalk, C++, and the like, and traditional procedural programming languages such as the "C" programming language, and / or machine languages such as assembly languages. The code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer, and partially on a remote computer or entirely on a remote computer or server. In the latter case, the remote computer may be connected to the user's computer via any type of network, including a local area network ("LAN") or a wide area network ("WAN"), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).
[0043] References in this specification to "one embodiment," "an embodiment," or similar language mean that a particular feature, structure, or characteristic described in conjunction with that embodiment is included in at least one embodiment. Therefore, unless expressly stated otherwise, the phrases "in one embodiment," "in an embodiment," and similar language appearing throughout the specification may, but do not necessarily, all refer to the same embodiment, but rather mean "one or more but not all embodiments." Unless expressly stated otherwise, the terms "include," "comprising," "having," and variations thereof mean "including but not limited to." Unless expressly stated otherwise, an enumerated list of items does not imply that any or all items are mutually exclusive.
[0044] In addition, the features, structures or characteristics of the described embodiments may be combined in any suitable manner. In the following description, many specific details are provided, such as examples of programming, software modules, user selections, network transactions, database queries, database structures, hardware modules, hardware circuits, hardware chips, etc., to provide a thorough understanding of the embodiments. However, those skilled in the relevant art will recognize that the embodiments may be practiced without one or more of the specific details, or using other methods, components, materials, etc. In other cases, well-known structures, materials or operations are not shown or described in detail to avoid obscuring some aspects of the embodiments.
[0045] Aspects of the embodiments are described below with reference to schematic flowchart diagrams and / or schematic block diagrams of methods, apparatuses, systems, and program products according to the embodiments.
[0046] The schematic flow charts or schematic block diagrams in the accompanying drawings illustrate the architecture, functions and operations of possible implementations of the devices, systems, methods and program products according to various embodiments. In this regard, each block in the schematic flow chart or schematic block diagram may represent a module, segment or portion of code, which includes one or more executable instructions of the code for implementing the specified logical function.
[0047] It should also be noted that in some alternative implementations, the functions annotated in the blocks may not occur in the order annotated in the drawings. For example, two blocks shown in succession may actually be executed substantially simultaneously, or the blocks may sometimes be executed in reverse order, depending on the functions involved. Other steps and methods are contemplated that are equivalent in function, logic, or effect to one or more blocks or portions thereof of the illustrated drawings.
[0048] Although various arrow types and line types can be used in flow charts or block diagrams, it should be understood that they do not limit the scope of the corresponding embodiments. In fact, some arrows or other connectors can only be used to indicate the logical flow of the depicted embodiment. For example, an arrow can indicate a waiting or monitoring period of unspecified duration between the enumeration steps of the depicted embodiment. It will also be noted that each block of the block diagram or flow chart and the combination of the blocks in the block diagram and / or flow chart can be implemented by a system based on dedicated hardware that performs a specific function or operation, or a combination of dedicated hardware and code.
[0049] The description of an element in each figure may refer to an element of the previous figure. The same numerals refer to the same elements in all figures, including alternative embodiments of the same elements.
[0050] Embodiments of the present invention will be described below with reference to the accompanying drawings.
[0051] Figure 1 A system block diagram of DataX as a prior art is depicted.
[0052] DataX is an open-source offline synchronization tool for heterogeneous data sources developed by Alibaba. It is dedicated to achieving stable and efficient data synchronization between various heterogeneous data sources, including relational databases (MySQL, Oracle, etc.), HDFS, Hive, ODPS, HBase, FTP, etc.
[0053] As a data synchronization framework, DataX abstracts the synchronization of different data sources into Reader plug-ins that read data from the source data source and Writer plug-ins that write data to the target end. In theory, the DataX framework can support data synchronization work of any data source type. At the same time, the DataX plug-in system is an ecosystem. Every time a new data source is connected, the newly added data source can communicate with the existing data source.
[0054] like Figure 1 As shown in the figure, DataX itself is an offline data synchronization framework, built with a framework + plug-in architecture. The data source reading and writing are abstracted into Reader / Writer plug-ins and incorporated into the entire synchronization framework.
[0055] Among them, Reader is a data collection module, responsible for collecting data from the data source and sending the data to the Framework. Writer is a data writing module, responsible for continuously obtaining data from the Framework and writing the data to the destination. Framework is used to connect Reader and Writer as a data transmission channel between the two, and handles core technical issues such as buffering, flow control, concurrency, and data conversion.
[0056] This technical solution requires users to write configuration files themselves and have a deep understanding of the DataX mechanism to use it well, and it does not have a good visual interface.
[0057] Figure 2 A system block diagram of Flume as a prior art is depicted.
[0058] Flume is a distributed, reliable, and highly available system for collecting, aggregating, and transmitting massive amounts of logs. Flume supports the customization of various data senders in the log system for collecting data. At the same time, Flume provides the ability to simply process data and write it to various data receivers (customizable).
[0059] like Figure 2 As shown in the figure, the core of Flume is Agent. Flume uses Agent as the smallest independent running unit. An Agent is a JVM. It is a complete data collection tool with three core components, namely Source, Channel, and Sink. Through these components, Event can flow from one place to another.
[0060] Source is the data collection end, responsible for special formatting of captured data, encapsulating data into events, and then pushing events into the Channel. Flume provides a variety of Sources and also provides custom Sources.
[0061] Channel is a short-lived storage container that caches the data in Event format received from Source until Sink processes the event. It acts as a bridge between Source and Sink.
[0062] Sink takes events from Channel and sends the data to other places. It can store data in file system, database, Hadoop, or be the Source of other Agents. When there is less log data, you can store the data in the file system and set a certain time interval to save the data.
[0063] Flume uses a transactional approach to ensure the reliability of the entire process of transmitting Events. The Sink must remove the Event from the Channel after the Event has been stored in the Channel, or has been transmitted to the next station Agent, or has been stored in an external data destination. In this way, the Event in the data stream can be guaranteed to be reliable whether it is flowing in one Agent or between multiple Agents, because the above transactions ensure that the Event will be successfully stored.
[0064] However, Flume targets fewer scenarios and is mainly used to handle scenarios related to log collection. It has no visual interface, and users are required to study its reliability, distribution and other features on their own.
[0065] In addition, there is also Etlworks in the existing technology. This technical solution relatively completely solves the related problems at the data collection level, including metadata management, scheduling configuration, source and target field mapping, simple ETL process, etc. However, this technology also has various shortcomings similar to Datax and Flume.
[0066] The above describes some existing technologies in the field of data acquisition, and introduces the characteristics and defects of these technologies. The following will describe the offline data acquisition system according to the present invention.
[0067] Figure 3 A block diagram of an offline data acquisition system according to one embodiment of the present invention is depicted.
[0068] like Figure 3 As shown, an offline data collection system 300 according to an embodiment of the present invention includes a data source 301 , an engine 302 , a control module 303 , a data target 304 , a storage module 305 , and a display module 306 .
[0069] The user configures the data collection task using the control module 303. The configuration information will be parsed and sent to the engine 302. The engine 302 executes the data collection task under the scheduling of the scheduling framework (not shown), including extracting data from the data source 301 and writing it to the data target 304.
[0070] In addition, the engine 302 will store the log files, indicator information, etc. generated during the task execution into the storage module 305. Using the information stored in the storage module 305, the display module 306 can summarize and display the progress of the data collection task to the user.
[0071] The offline data collection method according to an embodiment of the present invention will be described in detail below with reference to the flow chart.
[0072] Figure 4 A flow chart of a discrete data collection method according to an embodiment of the present invention is depicted.
[0073] like Figure 4 As shown, the discrete data collection method 400 includes steps S401-S405.
[0074] In step S401, the user configures the acquisition task information. Specifically, the user first configures the corresponding source and target data source information in the console, and then enters the acquisition task module to fill in the configuration information related to the acquisition task, such as the source and target data sources, flow control information, engine, scheduling information, etc., to complete the configuration work.
[0075] In step S402, a configuration file recognizable by the engine and a configuration file recognizable by the scheduling framework are generated according to the collection task information configured in step S401. After the task is written into the database, it will be parsed by the corresponding background scanning program into a configuration file recognizable by the corresponding engine and written into the machine corresponding to the engine and generate a configuration file recognizable by the scheduling framework, that is, a scheduling file.
[0076] In step S403, the scheduling framework calls the engine to execute the task according to the configuration information generated in step S402. The scheduling framework generates a series of scheduling tasks according to the scheduling information to start the corresponding engine to execute the corresponding task.
[0077] In step S404, after completing the corresponding task, the engine writes the log files, indicator information, etc. generated during the task into the corresponding storage module, such as the backend storage.
[0078] In step S405, the system uses the information stored in the storage module to display the execution status of the data collection task to the user.
[0079] The above generally describes the process of the method for discrete data collection according to an embodiment of the present invention. Each operation in the process will be described in detail below.
[0080] Figure 5 is a flow chart depicting a discrete data collection method according to another embodiment of the present invention. Specifically, Figure 5The flowchart of the operation of configuring acquisition task information in the discrete data acquisition method according to one embodiment of the present invention is depicted.
[0081] like Figure 5 As shown, the operation 500 of configuring the collection task information includes steps S501-S502.
[0082] In step S501, corresponding source and target data source information is configured.
[0083] The user first fills in the configuration information corresponding to the source end (the data source to extract data) and the target end (the data source to write data) in the data source management module, such as address information and user information. For example, for the MYSQL database, you need to fill in the IP address, port number, user name, password and other configurations.
[0084] In step S502, fill in task-related configuration information.
[0085] Filling in the configuration information related to the task includes selecting the source-end target-end data source to extract and input data according to the information filled in step S501, and configuring data extraction rules, such as whole library extraction. The configuration information related to the task further includes flow control information for controlling the maximum data flow in the data pipeline. The configuration information related to the task further includes engine configuration information to specify the engine to be used, such as Datax, Sqoop, etc., but the present invention is not limited thereto. The configuration information related to the task further includes scheduling information for specifying the frequency to be executed.
[0086] After step S502, the system will generate configuration information recognizable by the engine and the scheduling framework based on the collected task information, such as Figure 4 As shown in step S402 in FIG. Figure 6 Specific description.
[0087] Figure 6 is a flowchart depicting a discrete data collection method according to another embodiment of the present invention. Specifically, Figure 6 A flowchart is depicted of the operation of generating a configuration file recognizable by an engine and a configuration file recognizable by a scheduling framework according to collection task information in a discrete data collection method according to an embodiment of the present invention.
[0088] like Figure 6 As shown, the operation 600 of generating a configuration file recognizable by the engine and a configuration file recognizable by the scheduling framework includes steps S601-S603.
[0089] In step S601, the program scans the newly generated acquisition task configuration.
[0090] In step S602, the program parses the newly generated acquisition task configuration scanned, and generates a corresponding configuration file that can be recognized by the engine according to a predefined engine configuration template.
[0091] In step S603, the program continues to parse the newly generated acquisition task configuration scanned, and generates a scheduling configuration file that can be recognized by the scheduling framework according to the pre-defined scheduling frequency and the generated engine configuration file.
[0092] After step S603, the system will call the engine to execute the task according to the generated engine configuration file and scheduling configuration file. Figure 4 As shown in step S403 in .
[0093] The system will schedule and start the corresponding engine to perform specific tasks. The scheduling framework will trigger the scheduling task according to the scheduling configuration file generated in step S603 and execute the command of the corresponding engine. After the engine is started, it will perform specific tasks according to the configured parameters or configuration files.
[0094] Next, the process proceeds to step S404. The engine summarizes the log information, indicator information, etc. during the task execution and writes them into the backend storage.
[0095] Finally, the process proceeds to step S405, where the result information is displayed. Based on the log information, indicator information, etc. stored in the backend, the front-end interface summarizes the corresponding tasks and the corresponding task execution result information and displays them to the user.
[0096] Combination of the above Figure 3-6 A discrete data acquisition system and a method for discrete data acquisition according to embodiments of the present invention are described.
[0097] According to another embodiment, the present invention also includes an apparatus for performing a discrete data acquisition method.
[0098] Figure 7 is a block diagram illustrating an apparatus for performing a discrete data acquisition method according to an embodiment of the present invention.
[0099] The device 700 may include a central processing unit (CPU) 701, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 702 or a program loaded from a storage portion 708 into a random access memory (RAM) 703. In the RAM 703, various programs and data required for the operation of the system 700 are also stored. The CPU 701, the ROM 702, and the RAM 703 are connected to each other via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.
[0100] The following components are connected to the I / O interface 705: an input section 706 including a keyboard, a mouse, etc.; an output section 707 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 708 including a hard disk, etc.; and a communication section 709 including a network interface card such as a LAN card, a modem, etc. The communication section 709 performs communication processing via a network such as the Internet. A drive 710 is also connected to the I / O interface 705 as needed. A removable medium 711, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 710 as needed, so that a computer program read therefrom is installed into the storage section 708 as needed.
[0101] In particular, according to the embodiments disclosed in the present invention, the above reference Figure 3-6 The described method can be implemented as a computer software program. For example, the embodiment disclosed in the present invention includes a computer program product, which includes a computer program carried on a computer readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 709, and / or installed from the removable medium 711. When the computer program is executed by the central processing unit (CPU) 701, the above-mentioned functions defined in the discrete data acquisition system of the present invention are performed.
[0102] The present invention provides a discrete data acquisition system, which can provide users with visual configuration of data acquisition process in the entire link of data acquisition, improve management and monitoring of data acquisition process, and can adapt to multiple engines.
[0103] Embodiments may be practiced in other specific forms. The described embodiments should be considered in all respects as merely illustrative and not restrictive. Therefore, the scope of the present invention is indicated by the appended claims rather than the preceding description. All changes within the meaning and equivalent range of the claims are included within their scope.
Claims
1. A method for discrete data collection, the method comprising the steps of: Configure collection task information; Based on the collected task information, the generation engine can identify the configuration file and the scheduling framework can identify the configuration file; The scheduling framework calls the engine to execute the task according to the configuration file recognizable by the engine and the configuration file recognizable by the scheduling framework; The engine stores task-related logs and other indicator information; as well as Displays stored task-related logs and other indicator information; The step of configuring the collection task information further includes: Configure flow control information; configure scheduling frequency and the engine to be used to execute tasks; The step of generating an engine-recognizable configuration file and a scheduling framework-recognizable configuration file according to the collected task information further comprises: Scan the newly generated acquisition task configuration; Parsing the acquisition task configuration, and generating the engine-recognizable configuration file according to a predefined engine configuration template; and Parsing the acquisition task configuration, and generating the configuration file recognizable by the scheduling framework according to the scheduling frequency and the configuration file recognizable by the engine; The step of the scheduling framework calling the engine to execute the task according to the engine identifiable configuration file and the scheduling framework identifiable configuration file further includes: The scheduling framework can identify the configuration file generated by the scheduling framework scanning; The scheduling framework can identify a configuration file, and the scheduling framework triggers a scheduling task; and The engine executes the data collection task according to the scheduling task; The step of storing task-related logs and other indicator information by the engine further includes: After the task is completed, the engine writes the task-related logs and other indicator information to the backend storage for aggregation. Among them, the other indicator information at least includes data flow rate and memory usage.
2. The method according to claim 1, wherein: The step of configuring the collection task information further includes: Configure the address information and user information of the source data source and the target data source; and The source-end data source and the target-end data source are selected based on the address information and the user information.
3. The method according to claim 1, wherein: The step of configuring the collection task information further includes: Configure data extraction rules.
4. The method according to claim 1, wherein: The step of displaying based on the stored task-related logs and other indicator information further includes: The front end calls the back end interface according to the user's parameters, aggregates the task-related logs and other indicator information, filters them, and then displays them to the user.
5. A device for performing a discrete data acquisition method, the device comprising: A communication unit configured to perform data transmission and reception; a display unit configured to display data; a memory configured to store data and instructions; a processor operatively coupled to the communication unit, the display unit, and the memory, and configured to: Configure collection task information; Based on the collected task information, the generation engine can identify the configuration file and the scheduling framework can identify the configuration file; Controlling the scheduling framework to call the engine to execute tasks according to the configuration file recognizable by the engine and the configuration file recognizable by the scheduling framework; Controlling the engine to store task-related logs and other indicator information in the memory; as well as Controlling the display unit to display based on stored task-related logs and other indicator information; The processor is further configured to: configuring flow control information; and Configure the scheduling frequency and the engine to be used to execute the task; The processor is further configured to: Scan the newly generated acquisition task configuration; Parsing the acquisition task configuration, and generating the engine-recognizable configuration file according to a predefined engine configuration template; as well as Parsing the acquisition task configuration, and generating the configuration file recognizable by the scheduling framework according to the scheduling frequency and the configuration file recognizable by the engine; The processor is further configured to: Controlling the scheduling framework to scan and generate a configuration file that can be recognized by the scheduling framework; The scheduling framework can identify a configuration file and control the scheduling framework to trigger a scheduling task; as well as Controlling the engine to execute a data collection task according to the scheduling task; The processor is further configured to: After the task is completed, the engine writes the task-related log and the other indicator information into the backend storage for aggregation, wherein the other indicator information at least includes data flow rate and memory usage.
6. The device according to claim 5, wherein: The processor is further configured to: Configure the address information and user information of the source data source and the target data source; Selecting the source data source and the target data source based on the address information and the user information; Configure data extraction rules.
7. The device according to claim 5, wherein: The processor is further configured to: The front end calls the back end interface according to the user's parameters, aggregates the task-related logs and other indicator information, filters them, and then displays them to the user.
8. A computer readable medium having a computer program stored thereon, wherein: The computer program, when executed by a processor, causes the processor to perform a method according to any one of claims 1-4.
