Data acquisition method, device and system and storage medium
Obtaining data acquisition task information through work nodes and obtaining connectors from connector warehouses, solving the problem that traditional data acquisition methods are difficult to cope with diversified needs, achieving fast and accurate diversified data acquisition, reducing operation and maintenance costs and complexity.
Patent Information
- Application Number
- CN202411548412.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-31
- Publication Date
- 2025-02-25
AI Technical Summary
Traditional data acquisition methods are difficult to cope with diversified collection needs, need to develop specific application scenarios, and are difficult to achieve unified management and supervision, which increases operation and maintenance complexity and cost.
The work node obtains information of the data acquisition task from the configuration node, and acquires the connector from the connector warehouse based on the information of the first connector, and quickly connects to the target data source, thereby achieving diversified data acquisition.
It realizes rapid and accurate collection of diversified data, avoids repeated development, saves time and resources, and improves data collection efficiency and system maintainability.
Smart Images
Figure CN119513173A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of data processing technology, and in particular to a data acquisition method, device, system and storage medium. Background Art
[0002] Currently, the demand for data collection is continuously growing. However, since the collected data comes from many different types of events, logs, and sensor data, traditional data collection methods need to be developed for specific application scenarios and are difficult to cope with diverse collection needs.
[0003] Therefore, how to collect diverse data has become a problem that needs to be solved urgently. Summary of the invention
[0004] The present application provides a data collection method, device, system and storage medium for solving the problem of difficulty in collecting diversified data.
[0005] In order to achieve the above objectives, this application adopts the following technical solutions:
[0006] In a first aspect, the present application provides a data collection method, which is applied to a working node. In the method, information of a data collection task is obtained, and the data collection task is used to indicate that data is collected from a target data source, and the target data source supports a first interface protocol. The information of the data collection task includes: information of a first connector. The first connector is configured based on the first interface protocol. Based on the information of the first connector, the first connector is obtained. Based on the first connector, target data is obtained from the target data source.
[0007] Based on the above technical solution, the working node obtains the information of the data collection task from the configuration node. In this way, the working node can perform the data collection task based on the information of the data collection task to ensure that the collected data meets the requirements. Among them, the information of the data collection task includes the information of the first connector. The working node obtains the first connector from the connector warehouse based on the information of the first connector. In this way, the connector required for data collection can be quickly and accurately obtained through the information of the first connector. The working node establishes a connection with the target data source through the first connector, can quickly connect to the required data source, and obtain the target data from the target data source, thereby realizing diversified data collection. Furthermore, repeated development can be avoided, thereby saving time and resources.
[0008] In one possible design, a first request message is sent to a connector warehouse, where the first request message is used to request a first connector. The connector warehouse stores at least one connector, and one connector corresponds to one interface protocol. The first connector sent by the connector warehouse is received.
[0009] In a possible design, the information of the data collection task also includes: a first data processing method, the first data processing method is used to convert the format of the data obtained from the target data source. Initial data is collected from the target data source through a first connector. The initial data is processed based on the first data processing method to obtain target data, and the target data is data in a format required by the data collection task.
[0010] In a possible design, the information of the data collection task also includes: information of a destination node for receiving the target data, information of a second connector, and the destination node supports the second connector. Based on the information of the second connector, the second connector is obtained. Based on the second connector, the target data is sent to the destination node.
[0011] In a possible design, a task identifier sent by a management node is received, an information request message is sent to a configuration node, and the information request message is used to request information about a data collection task corresponding to the task identifier, and the information about the data collection task is received from the configuration node.
[0012] In a second aspect, the present application provides a data collection method, which is applied to a data collection system. The data collection system includes: a working node and a management node. In the method, the management node sends information about a data collection task to the working node, and the information about the data collection task is used to indicate that data is collected from a target data source, and the target data source supports a first interface protocol. The information about the data collection task includes: information about a first connector, and the first connector is configured based on the first interface protocol. The working node receives the information about the data collection task sent by the management node. The working node obtains the first connector based on the information about the first connector. The working node obtains target data from the target data source based on the first connector.
[0013] In a possible design, the data collection system further includes: a connector warehouse, wherein at least one connector is stored in the connector warehouse, and one connector corresponds to one interface protocol. The working node sends a first request message to the connector warehouse, wherein the first request message is used to request to obtain the first connector. In response to the first request message, the connector warehouse sends the first connector to the working node. The working node receives the first connector sent by the connector warehouse.
[0014] In a third aspect, the present application provides a data collection system, which includes: a working node and a management node. The management node sends information about a data collection task to the working node, and the information about the data collection task is used to indicate that data is collected from a target data source, and the target data source supports a first interface protocol. The information about the data collection task includes: information about a first connector, and the first connector is configured based on the first interface protocol. The working node receives the information about the data collection task sent by the management node. The working node obtains the first connector based on the information about the first connector. The working node obtains target data from the target data source based on the first connector.
[0015] In a fourth aspect, the present application provides a data acquisition device, which is applied to a working node. The device includes an acquisition module and a processing module.
[0016] The acquisition module is used to acquire information of a data acquisition task, where the data acquisition task is used to indicate that data is to be collected from a target data source, and the target data source supports a first interface protocol. The information of the data acquisition task includes information of a first connector, where the first connector is configured based on the first interface protocol. The processing module is used to acquire the first connector based on the information of the first connector. The processing module is also used to acquire target data from the target data source based on the first connector.
[0017] In one possible design, the processing module is further used to send a first request message to the connector warehouse, where the first request message is used to request to obtain a first connector. The connector warehouse stores at least one connector, and one connector corresponds to one interface protocol. The processing module is further used to receive the first connector sent by the connector warehouse.
[0018] In a possible design, the information of the data collection task also includes: a first data processing method, the first data processing method is used to convert the format of the data obtained from the target data source. The processing module is also used to collect initial data from the target data source through the first connector. The processing module is also used to process the initial data based on the first data processing method to obtain target data, and the target data is data in the format required by the data collection task.
[0019] In a possible design, the information of the data acquisition task also includes: information of a destination node for receiving the target data, information of a second connector, and the destination node supports the second connector. The acquisition module is further used to acquire the second connector based on the information of the second connector. The processing module is further used to send the target data to the destination node based on the second connector.
[0020] In a possible design, the processing module is further used to receive the task identifier sent by the management node. The processing module is further used to send an information request message to the configuration node, the information request message is used to request information about the data collection task corresponding to the task identifier. The processing module is further used to receive information about the data collection task sent by the configuration node.
[0021] In a fifth aspect, the present application provides a data acquisition device, comprising: a processor and a memory; the processor and the memory are coupled; the memory is used to store one or more programs, the one or more programs including computer execution instructions, and when the data acquisition device is running, the processor executes the computer execution instructions stored in the memory to implement the method described in any possible implementation of the first aspect and the second aspect.
[0022] In a sixth aspect, the present application provides a computer-readable storage medium, in which instructions are stored. When the instructions are executed on a computer, the computer executes the method described in any possible implementation of the first and second aspects above.
[0023] In a seventh aspect, the present application provides a chip, comprising a processor and a communication interface, wherein the communication interface and the processor are coupled, and the processor is used to run a computer program or instructions to implement a method as described in any possible implementation of the first aspect and the second aspect.
[0024] In an eighth aspect, the present application provides a computer program product comprising instructions, which, when executed by a computer, enables the computer to execute the method described in any possible implementation of the first and second aspects above.
[0025] In the above scheme, the technical problems that can be solved and the technical effects achieved by the data acquisition device, computer equipment, computer storage medium, chip or computer program product can be referred to the technical problems and technical effects solved by the above first aspect, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS
[0026] Figure 1 A system architecture diagram of a data acquisition system provided in an embodiment of the present application;
[0027] Figure 2 A flow chart of a data collection method provided in an embodiment of the present application;
[0028] Figure 3 A flowchart of another data collection method provided in an embodiment of the present application;
[0029] Figure 4 A flowchart of another data collection method provided in an embodiment of the present application;
[0030] Figure 5 A schematic diagram of an example of a data collection method provided in an embodiment of the present application;
[0031] Figure 6 A schematic diagram of another example of a data collection method provided in an embodiment of the present application;
[0032] Figure 7 A schematic diagram of the structure of a data acquisition device provided in an embodiment of the present application;
[0033] Figure 8 A schematic diagram of the structure of another data acquisition device provided in an embodiment of the present application;
[0034] Fig. 9 A conceptual partial view of a computer program product provided for an embodiment of the present application. DETAILED DESCRIPTION
[0035] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.
[0036] The terms “first” and “second” in the description and claims of the present application are used to distinguish different objects rather than to describe a specific order of the objects.
[0037] In addition, the terms "including" and "having" and any variations thereof mentioned in the description of this application are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or modules is not limited to the listed steps or modules, but may optionally include other steps or modules that are not listed, or may optionally include other steps or modules that are inherent to these processes, methods, products or devices.
[0038] In addition, in the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present concepts in a specific way.
[0039] At present, the demand for data collection is continuously growing, and the collected data comes from many different types of events, logs and sensor data. However, due to the differences between different collection interface protocols, data format protocols and data sources, traditional data collection methods are difficult to cope with diverse collection needs.
[0040] Traditional data collection methods usually develop specific interface protocols and data formats. As a result, similar code snippets or tool classes often appear in multiple collection projects. Although code duplication can be solved by code reuse, there are still problems with version control and maintenance, which can easily lead to code redundancy. In addition, interface protocols and data formats may change over time. Traditional methods often seem incapable of handling multiple versions of protocols, which can easily lead to system confusion and compatibility issues. When faced with new collection requirements or protocol changes, a large amount of customized development is often required, which lacks flexibility and increases development costs and maintenance difficulties.
[0041] In addition, traditional collection methods are usually difficult to cope with large-scale collection needs. In the scenario of a large number of collection nodes, traditional methods are difficult to achieve unified management and supervision. Operations such as deployment, updating, and fault recovery of collection nodes often require manual intervention, which increases the complexity and workload of operation and maintenance. Moreover, when the number of collection nodes increases sharply, it may lead to performance bottlenecks and low resource utilization. In addition, disaster recovery migration of collection nodes usually requires a lot of manual intervention and configuration adjustments. This not only increases operation and maintenance costs, but may also cause interruptions and inconsistencies in data collection.
[0042] Therefore, how to collect diverse data has become a problem that needs to be solved urgently.
[0043] In order to solve the above technical problems, an embodiment of the present application provides a data collection method. In this method, the working node obtains the information of the data collection task from the configuration node. In this way, the working node can perform the data collection task based on the information of the data collection task to ensure that the collected data meets the requirements. Among them, the information of the data collection task includes the information of the first connector. The working node obtains the first connector from the connector warehouse based on the information of the first connector. In this way, the connector required for data collection can be quickly and accurately obtained through the information of the first connector. The working node establishes a connection with the target data source through the first connector, can quickly connect to the required data source, and obtain the target data from the target data source, thereby realizing diversified data collection. Furthermore, repeated development can be avoided, thereby saving time and resources.
[0044] The implementation environment of the embodiments of the present application is introduced below.
[0045] like Figure 1As shown, it is a system architecture diagram of a data acquisition system provided by an embodiment of the present application, and the system includes: a working node 101, a management node 102 and a connector warehouse 103. Among them, the working node 101 is connected to the management node 102, the working node 101 is connected to the connector warehouse 103, and the connector warehouse 103 is connected to the management node 102.
[0046] In a possible design, the working node 101 establishes a connection with the management node 102 by registering with the management node 102. The working node 101 communicates directly with the connector warehouse 103 through a network protocol. The connector warehouse 103 establishes a connection with the management node 102 by registering with the management node 102.
[0047] In a possible design, the working node 101 may be a cluster node in a Kubernetes platform.
[0048] The work node 101 is used to obtain the information of the data collection task to execute the data collection task. The data collection task is used to instruct to collect data from the data source. The data source can support at least one interface protocol. The information of the data collection task can include the information of the connector configured based on the interface protocol. In addition, the work node 101 is also used to obtain the connector from the connector warehouse 103 to connect the data source and obtain the initial data. Afterwards, the work node 101 can process the initial data to obtain the target data.
[0049] The management node 102 is used to obtain the information of the data collection task and send the information of the data collection task to the working node 101. The management node 102 is also used to receive connectors, dependency packages and plug-ins, and send the connectors, dependency packages and plug-ins to the connector warehouse 103. The management node 102 is also used to obtain the collection information of the working node 101, so as to display the collection information in real time in the form of a chart through the WEB management page. The collection information includes the health status and collection progress of the collection task. The health status of the collection task is used to indicate whether the data collection process is running normally.
[0050] In a possible design, the management node 102 stores format information, and the format information is used to indicate the format of the information of the data collection task. The management node 102 can determine the format of the information of the acquired data collection task. If the format of the information of the acquired data collection task is the same as the format information, the management node 102 sends the information of the data collection task to the working node 101. If the format of the information of the acquired data collection task is different from the format information, the management node 102 rejects the information of the data collection task.
[0051] In a possible design, the management node 102 stores connector parameter configuration information, and the connector parameter configuration information is used to indicate the configuration format of the connector. The management node 102 can judge the obtained connector. If the obtained connector meets the configuration format requirements, the management node 102 sends the connector to the connector warehouse 103. If the obtained connector does not meet the configuration format requirements, the management node 102 rejects the connector.
[0052] In one possible design, the management node 102 may record the version information of each connector and perform version control on the connector so that the working node 101 can obtain the correct version of the connector each time it executes a data collection task.
[0053] In a possible design, the management node 102 stores the tasks currently running on all the working nodes 101. When the working node 101 crashes due to an abnormality, it can be restarted and the registration can be completed. The management node 102 can resend the tasks currently being executed by the working node 101.
[0054] The connector warehouse 103 is used to receive connectors, dependent packages and plug-ins from the management node 102, and store connectors, dependent packages of connectors and plug-ins. One connector corresponds to one interface protocol. The connector warehouse 103 can also distribute connectors, dependent packages of connectors and plug-ins.
[0055] In a possible design, the connector repository 103 pre-stores at least one connector, a dependent package of the connector, and a plug-in.
[0056] The working node 101 and the connector warehouse 103 in the embodiment of the present application can be a server. The server can be a physical server or a cloud server. The server can be a single server, or it can be a server cluster composed of multiple servers. In some implementations, the server cluster can also be a distributed cluster. The present disclosure does not limit the specific implementation of the server.
[0057] In some embodiments, the system further includes: a configuration node 104. The configuration node 104 is used to store the information of the data collection task. Furthermore, the configuration node 104 can also send the information of the data collection task to the working node 101. Furthermore, when the information of the data collection task is modified, the configuration node 104 is also used to push the updated information of the data collection task to the working node 101.
[0058] In a possible design, the configuration node 104 may receive the information of the data collection task from the management node 102 , store the information of the data collection task, and generate a task identifier for the information of the data collection task. The configuration node 104 sends the task identifier to the management node 102 .
[0059] It should be noted that the present application does not limit the configuration node 104. For example, the configuration node 104 in the embodiment of the present application may be Zookeeper, Consul, Nacos, etc.
[0060] It should be noted that the connector warehouse 103 and the configuration node 104 may be a form of expression of the working node 101. Alternatively, the working node 101, the connector warehouse 103 and the configuration node 104 may be different devices. This embodiment of the application does not limit this.
[0061] In some embodiments, the system further includes a destination node 105. The destination node 105 is used to receive target data from the working node 101.
[0062] In one possible design, the working node 101 may send target data to the destination node 105 .
[0063] Exemplarily, the management node 102 and the destination node 105 in the embodiment of the present application can be tablet computers, mobile phones, desktop computers, laptops, handheld computers, notebook computers, ultra-mobile personal computers (UMPCs), netbooks and other devices. The embodiment of the present application does not impose any special restrictions on the specific forms of the management node 102 and the destination node 105.
[0064] The embodiments of the present application are described in detail below in conjunction with the accompanying drawings.
[0065] like Figure 2 As shown, a data collection method provided by an embodiment of the present application includes:
[0066] S201. The working node obtains information of the data collection task.
[0067] The data collection task is used to instruct to collect data from a target data source, and the target data source supports a first interface protocol. The information of the data collection task includes: information of a first connector, and the first connector is configured based on the first interface protocol.
[0068] In a possible implementation, the working node may receive the task identifier sent by the management node. Afterwards, the working node may send an information request message to the configuration node based on the task identifier, the information request message being used to request information about the data collection task corresponding to the task identifier, the information request message including the task identifier. The working node receives the information about the data collection task sent by the configuration node.
[0069] In a possible design, the working node registers with the management node after startup and reports its own survival status through heartbeat. After completing the registration, the working node enters the standby state and listens to the task instructions and task identifiers sent by the management node through the listener. The task instructions are used to instruct the working node to send an information request message to the configuration node based on the task identifier. In response to the task instruction, the working node registers with the configuration node and sends an information request message to the configuration node based on the task identifier. The configuration node sends the information of the data collection task corresponding to the task identifier to the working node, and the working node receives the information of the data collection task sent by the configuration node.
[0070] In another possible design, when the information of the data collection task changes, the working node monitors that the management node sends a task update instruction to the working node, and the configuration node pushes the updated data collection task information to the working node. In response to the task update instruction, the working node obtains the updated data collection task information.
[0071] S202: The working node obtains the first connector based on the information of the first connector.
[0072] In a possible implementation, the working node sends a first request message to the connector repository, the first request message is used to request to obtain the first connector, and the first request message includes information about the first connector. Afterwards, the working node receives the first connector sent by the connector repository.
[0073] It should be noted that the present application does not limit the connector. For example, the present application may define a connector interface, and the connector may be a jar package that implements the connector interface.
[0074] In one possible design, the connector interface includes connector parameters. The connector parameters include at least one of the following: a connector interface type parameter, a connector version parameter, and a connector description parameter. The connector interface type parameter is used to declare the interface type implemented by the connector, the connector version parameter is used to declare the version of the connector, and the connector description parameter is used to describe the interface implemented by the connector in detail.
[0075] Exemplarily, the connector interface type parameter may be a type parameter, the connector version parameter may be a version parameter, and the connector description parameter may be a description parameter.
[0076] Exemplarily, the connector interface type may be a Kafka interface type, may be a file transfer protocol (FTP) interface type, or may be a Socket interface type.
[0077] It should be noted that the connector description parameters will be displayed through the WEB management page and also serve as retrieval information for future inquiries.
[0078] S203: The working node obtains target data from the target data source based on the first connector.
[0079] In a possible implementation, the information of the first connector includes first connection data. The first connection data is used to indicate that the first connector establishes a connection with a target data source. Based on the first connector and the first connection data, the working node establishes a connection with the target data source, executes a data collection task, and obtains target data from the target data source.
[0080] In one possible design, the first connection data includes at least one of the following: a database host name, a database port number, a database name, a database user name, a database password, an application program interface (API) uniform resource locator (URL), an API request header, API parameters, an access address stored by a target data source, and authentication information stored by the target data source.
[0081] In some embodiments, the information of the data collection task also includes: information of the destination node receiving the target data, information of the second connector, and the destination node supports the second connector. The working node obtains the second connector based on the information of the second connector. The working node sends the target data to the destination node based on the second connector.
[0082] In a possible implementation, the information of the second connector includes second connection data, and the second connection data is used to indicate that the second connector establishes a connection with the destination node. The working node sends a second request message to the connector warehouse, and the second request message is used to request to obtain the second connector. The working node receives the second connector sent by the connector warehouse. Based on the information of the destination node of the target data, the second connector, and the second connection data, the working node establishes a connection with the destination node and sends the target data to the destination node.
[0083] In one possible design, the second connection data includes at least one of the following: database host name, database port number, database name, database user name, database password, API URL, API request header, API parameters, destination node access address and destination node authentication information.
[0084] Based on the above technical solution, the working node obtains the information of the data collection task from the configuration node. In this way, the working node can perform the data collection task based on the information of the data collection task to ensure that the collected data meets the requirements. Among them, the information of the data collection task includes the information of the first connector. The working node obtains the first connector from the connector warehouse based on the information of the first connector. In this way, the connector required for data collection can be quickly and accurately obtained through the information of the first connector. The working node establishes a connection with the target data source through the first connector, can quickly connect to the required data source, and obtain the target data from the target data source, thereby realizing diversified data collection. Furthermore, repeated development can be avoided, thereby saving time and resources.
[0085] like Figure 3 As shown, another data collection method provided by an embodiment of the present application, in which S203 may include:
[0086] S301. A working node collects initial data from a target data source through a first connector.
[0087] Among them, the initial data refers to data that has not undergone any form of processing, conversion or aggregation.
[0088] In a possible design, the initial data is character string data.
[0089] It should be noted that this application defines multiple interfaces that connectors need to implement. Connectors can implement interface protocols, and working nodes can obtain data on the interfaces in sequence through connectors.
[0090] In a possible implementation, the working node establishes a connection with the target data source through the first connector. Based on the connector interface definition, the working node collects a string from the target data source each time to obtain initial data.
[0091] S302: The working node processes the initial data based on the first data processing method to obtain target data.
[0092] The target data is the data in the format required by the data collection task.
[0093] In a possible design, the information of the data collection task also includes: a first data processing method, and the first data processing method is used to convert the format of the data obtained from the target data source.
[0094] Exemplarily, the first data processing method may be to add a data field, to delete a data field, to assign a value to a data field, to extract a data field, or to calculate a data field.
[0095] In a possible implementation, the working node loads the first data processing method into the rule engine, processes the initial data, parses the initial data into a specific data structure, trims redundant fields, converts the field type, and obtains the target data.
[0096] In a possible design, the first data processing method complies with the parsing rules of Groovy syntax. The working node uses the Groovy rule engine to implement various complex parsing using Groovy syntax.
[0097] Based on the above technical solution, the working node collects initial data from the target data source through the first connector. Afterwards, the working node processes the initial data based on the first data processing method to obtain the target data. In this way, by introducing the rule engine, it is possible to flexibly adapt to different collection requirements and complete the collection and processing of data from any source and in any format, greatly reducing the workload of repeated development and improving development efficiency and system maintainability.
[0098] like Figure 4 As shown, a data collection method provided by an embodiment of the present application includes:
[0099] S401: The management node sends information about data collection tasks to the working node.
[0100] The information of the data collection task is used to indicate collecting data from a target data source, and the target data source supports a first interface protocol. The information of the data collection task includes: information of a first connector, and the first connector is configured based on the first interface protocol.
[0101] S402: The working node receives the information of the data collection task sent by the management node.
[0102] S403: The working node obtains the first connector based on the information of the first connector.
[0103] In some embodiments, the data collection system further includes: a connector warehouse, wherein at least one connector is stored in the connector warehouse, and one connector corresponds to one interface protocol. The working node sends a first request message to the connector warehouse, wherein the first request message is used to request to obtain the first connector. In response to the first request message, the connector warehouse sends the first connector to the working node. The working node receives the first connector sent by the connector warehouse.
[0104] S404: The working node obtains target data from the target data source based on the first connector.
[0105] It should be noted that, for the specific description of S401, S402, S403 and S404, reference may be made to the introduction of S201, S202, S203, S204, S301 and S302 in the above embodiments, which will not be repeated here.
[0106] It can be understood that the management node sends the information of the data collection task to the working node. Afterwards, the working node receives the information of the data collection task sent by the management node. In this way, the working node can perform the data collection task based on the information of the data collection task to ensure that the collected data meets the requirements. Then, the working node obtains the first connector based on the information of the first connector. In this way, the connector required for data collection can be quickly and accurately obtained through the information of the first connector. The working node obtains the target data from the target data source based on the first connector. In this way, the working node can directly establish a connection with the target data source without going through the middle layer, thereby improving the efficiency of data collection.
[0107] The following is an introduction to the embodiments of the present application with reference to specific examples. Figure 5As shown, it is an example schematic diagram of a data collection method provided by an embodiment of the present application. The management node receives the connector through the WEB management page and sends the connector to the connector warehouse. The connector warehouse stores the connector. The management node receives the information of the data collection task through the WEB management page, and sends the information of the data collection task to the configuration node. The configuration node stores the information of the data collection task and generates a corresponding task identifier. In response to the collection task trigger operation, the management node sends a task collection message to the Kubernetes platform (i.e., K8S), and the task collection message is used to instruct the working node to perform the data collection task. The Kubernetes platform receives the task collection message, and then the Kubernetes platform starts an empty working node. After the working node is started, it automatically registers with the management node and reports its own survival status through heartbeat. The management node sends a task instruction and a task identifier to the working node. In response to the task instruction, the working node registers with the configuration node and sends an information request message to the configuration node based on the task identifier. The configuration node sends the information of the data collection task corresponding to the task identifier to the working node, and the working node receives the information of the data collection task sent by the configuration node. The information of the data collection task includes the information of the first connector, and the information of the first connector includes the first connection data. Afterwards, the working node sends a first request message to the connector warehouse based on the information of the first connector. The first request message is used to request the first connector. The working node receives the first connector sent by the connector warehouse. The first connector can be a Kafka interface type connector, an FTP interface type connector, or a Socket interface type connector. Afterwards, the working node establishes a connection with the target data source based on the first connector and the first connection data, executes the data collection task, and obtains the initial data from the target data source. The working node loads the first data processing method into the rule engine, processes the initial data, parses the initial data into a specific data structure, trims redundant fields, converts the field type, and obtains the target data.
[0108] The following is an introduction to the embodiments of the present application with reference to specific examples. Figure 6 As shown, it is an example schematic diagram of another data collection method provided by an embodiment of the present application. The working node establishes a connection with the target data source based on the first connector and the first connection data, executes the data collection task, and obtains initial data from the target data source. The working node loads the first data processing method into the rule engine, processes the initial data, and then parses the initial data into a specific data structure (i.e., parsed data), trims redundant fields, converts the field type (i.e., standardized data), and obtains the target data. The working node sends the target data (i.e., output data) to the destination node based on the second connector.
[0109] The above mainly introduces the solution provided by the embodiment of the present application from the perspective of the method. It is understandable that, in order to realize the above functions, the data acquisition device includes a hardware structure and / or software module corresponding to the execution of each function. Those skilled in the art should easily realize that, in combination with the data acquisition method steps of each example described in the embodiment disclosed in the present application, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present application.
[0110] The embodiment of the present application further provides a data acquisition device, which can be a server, a CPU in the server, a data acquisition module in the server, or a client in the server.
[0111] The embodiment of the present application can divide the data acquisition device into functional modules or functional units according to the above method example. For example, each functional module or functional unit can be divided according to each function, or two or more functions can be integrated into one processing module. The above integrated module can be implemented in the form of hardware or in the form of software functional modules or functional units. Among them, the division of modules or units in the embodiment of the present application is schematic, which is only a logical function division, and there may be other division methods in actual implementation.
[0112] The present application embodiment provides a data acquisition device. Figure 7 As shown, the data acquisition device may include: an acquisition module 701 and a processing module 702 .
[0113] The acquisition module 701 is used to acquire information of a data acquisition task, where the data acquisition task is used to indicate that data is collected from a target data source, and the target data source supports a first interface protocol. The information of the data acquisition task includes information of a first connector, where the first connector is configured based on the first interface protocol. The processing module 702 is used to acquire the first connector based on the information of the first connector. The processing module 702 is also used to acquire target data from the target data source based on the first connector.
[0114] Figure 8 8 is a schematic diagram of another data acquisition device according to an exemplary embodiment. The data acquisition device may include a processor 802, and the processor 802 is used to execute application code to implement the data acquisition method in the present application.
[0115] The processor 802 may be a central processing unit (CPU), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits for controlling the execution of the program of the present application.
[0116] like Figure 8 As shown, the data acquisition device may further include a memory 803. The memory 803 is used to store application program codes for executing the solution of the present application, and is controlled to execute by the processor 802.
[0117] The memory 803 may be a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, a random access memory (RAM) or other types of dynamic storage devices that can store information and instructions, or an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical disc, laser disc, optical disc, digital versatile disc, Blu-ray disc, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory 803 may exist independently and be connected to the processor 802 via the bus 804. The memory 803 may also be integrated with the processor 802.
[0118] like Figure 8 As shown, the data acquisition device may further include a communication interface 801, wherein the communication interface 801, the processor 802, and the memory 803 may be coupled to each other, for example, via a bus 804. The communication interface 801 is used to exchange information with other devices, for example, to support information exchange between the data acquisition device and other devices.
[0119] It should be pointed out that Figure 8 The device structure shown in the figure does not constitute a limitation on the data acquisition device, except Figure 8 In addition to the components shown, the data acquisition device may include more or fewer components than shown, or combine certain components, or have a different arrangement of components.
[0120] In actual implementation, the functions implemented by the processing unit can be Figure 8The processor 802 shown calls the program code in the memory 803 to implement.
[0121] The present application also provides a computer-readable storage medium, on which instructions are stored, and when the instructions in the computer-readable storage medium are executed by a processor of a computer device, the computer can execute the data acquisition method provided in the above-mentioned embodiment. For example, the computer-readable storage medium can be a memory 803 including instructions, and the above instructions can be executed by a processor 802 of a computer device to complete the above method. Optionally, the computer-readable storage medium can be a non-temporary computer-readable storage medium, for example, a non-temporary computer-readable storage medium can be a ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage device, etc.
[0122] Fig. 9 A conceptual partial view of a computer program product provided by an embodiment of the present application is schematically shown, where the computer program product includes a computer program for executing a computer process on a computing device.
[0123] In one embodiment, the computer program product is provided using a signal bearing medium 900. The signal bearing medium 900 may include one or more program instructions that, when executed by one or more processors, may provide the above-described Figure 2 , Figure 3 Thus, for example, reference to Figure 2 In the embodiment shown in , one or more features of S201 to S203 may be undertaken by one or more instructions associated with the signal bearing medium 900. In addition, Fig. 9 The program instructions in also describe example instructions.
[0124] In some examples, the signal bearing medium 900 may include a computer readable medium 901, such as, but not limited to, a hard drive, a compact disk (CD), a digital video disk (DVD), a digital tape, a memory, a read-only memory (ROM) or a random access memory (RAM), and the like.
[0125] In some implementations, signal bearing medium 900 may include computer recordable medium 902 such as, but not limited to, memory, read / write (R / W) CD, R / W DVD, and the like.
[0126] In some implementations, signal bearing medium 900 may include communication medium 903 such as, but not limited to, digital and / or analog communication media (eg, fiber optic cables, waveguides, wired communication links, wireless communication links, etc.).
[0127] The signal bearing medium 900 may be communicated by a wireless form of communication medium 903. The one or more program instructions may be, for example, computer executable instructions or logic implemented instructions.
[0128] In some examples, the data collection device may be configured to provide various operations, functions, or actions in response to one or more program instructions via computer-readable medium 901 , computer-recordable medium 902 , and / or communication medium 903 .
[0129] Through the description of the above implementation methods, technical personnel in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.
[0130] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of modules or units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.
[0131] The units described as separate components may or may not be physically separated, and the components shown as units may be one physical unit or multiple physical units, that is, they may be located in one place or distributed in multiple different places. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.
[0132] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.
[0133] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the prior art or the full classification part or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium, including a number of instructions to enable a device (which can be a single-chip microcomputer, chip, etc.) or a processor (processor) to execute the full classification part or part of the steps of each embodiment method of the present application. The aforementioned storage medium includes various media that can store program codes, such as-U disk, mobile hard disk, ROM, RAM, disk or CD.
[0134] The above are only specific implementations of the present application, but the protection scope of the present application is not limited thereto, and any changes or substitutions within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.
Claims
1. A data collection method, characterized in that: Applied to a working node, the method includes: Acquire information of a data collection task, the data collection task is used to instruct to collect data from a target data source, the target data source supports a first interface protocol; the information of the data collection task includes: information of a first connector, the first connector is configured based on the first interface protocol; Based on the information of the first connector, acquiring the first connector; Based on the first connector, target data is acquired from the target data source.
2. The method according to claim 1, characterized in that The acquiring the first connector based on the information of the first connector includes: Sending a first request message to a connector warehouse, wherein the first request message is used to request to obtain the first connector; wherein the connector warehouse stores at least one connector, and one connector corresponds to one interface protocol; Receive the first connector sent by the connector warehouse.
3. The method according to claim 1, characterized in that The information of the data collection task also includes: a first data processing method, the first data processing method is used to convert the format of the data obtained from the target data source; The acquiring target data from the target data source based on the first connector includes: Collecting initial data from the target data source through the first connector; The initial data is processed based on the first data processing method to obtain the target data, where the target data is data in a format required by the data acquisition task.
4. The method according to any one of claims 1 to 3, characterized in that The information of the data collection task also includes: information of a destination node for receiving the target data and information of a second connector, wherein the destination node supports the second connector; the method also includes: Based on the information of the second connector, acquiring the second connector; Based on the second connector, the target data is sent to the destination node.
5. The method according to any one of claims 1 to 3, characterized in that: The information of obtaining the data collection task includes: Receive the task identifier sent by the management node; Sending an information request message to a configuration node, wherein the information request message is used to request information of the data collection task corresponding to the task identifier; Receive the information of the data collection task sent by the configuration node.
6. A data collection method, characterized in that: Applied to a data acquisition system, the data acquisition system includes: a working node and a management node; the method includes: The management node sends information of a data collection task to the working node, where the information of the data collection task is used to indicate collecting data from a target data source, where the target data source supports a first interface protocol; the information of the data collection task includes: information of a first connector, where the first connector is configured based on the first interface protocol; The working node receives the information of the data collection task sent by the management node; The working node obtains the first connector based on the information of the first connector; The working node obtains target data from the target data source based on the first connector.
7. The method according to claim 6, characterized in that The data acquisition system further comprises: a connector warehouse, wherein at least one connector is stored in the connector warehouse, and one connector corresponds to one interface protocol; The working node obtains the first connector based on the information of the first connector, including: The working node sends a first request message to the connector repository, where the first request message is used to request to obtain the first connector; In response to the first request message, the connector repository sends the first connector to the working node; The working node receives the first connector sent by the connector repository.
8. A data acquisition system, characterized in that: The data acquisition system includes: working nodes and management nodes; The management node sends information of a data collection task to the working node, where the information of the data collection task is used to indicate collecting data from a target data source, where the target data source supports a first interface protocol; the information of the data collection task includes: information of a first connector, where the first connector is configured based on the first interface protocol; The working node receives the information of the data collection task sent by the management node; The working node obtains the first connector based on the information of the first connector; The working node obtains target data from the target data source based on the first connector.
9. A data acquisition device, characterized in that: The device comprises an acquisition module and a processing module: The acquisition module is used to acquire information of a data acquisition task, where the data acquisition task is used to instruct to collect data from a target data source, and the target data source supports a first interface protocol; The information of the data collection task includes: information of a first connector, wherein the first connector is configured based on the first interface protocol; The acquisition module is further configured to acquire the first connector based on the information of the first connector; The acquisition module is further configured to acquire target data from the target data source based on the first connector.
10. A data acquisition device, characterized in that: include: Processor and memory; The processor is coupled to the memory; The memory is used to store one or more programs, and one or more of the programs include computer-executable instructions. When the data acquisition device is running, the processor executes the computer-executable instructions stored in the memory to enable the data acquisition device to perform the method as described in any one of claims 1-7.
11. A computer-readable storage medium, wherein instructions are stored in the computer-readable storage medium, characterized in that: When a computer executes the instruction, the computer performs the method according to any one of claims 1 to 7.
12. A computer program product comprising instructions, characterized in that When the instructions are executed by a computing device, the computing device is caused to perform the method according to any one of claims 1 to 7.