Flume-based data collection method and device, computer equipment and storage medium

Abstract

The invention provides a Flume-based data collection method and apparatus, a computer device and a storage medium. The method comprises the steps of receiving to-be-pushed data sent by a data extraction server based on an acquisition component; transmitting the to-be-pushed data acquired by the acquisition component to a preset sinking component based on the transmission channel component, and generating a message producer by the preset sinking component; and based on the message producer, sending the to-be-pushed data to the Rocket MQ message middleware. In the scheme, in the data collection process based on the Flume, the message producer sends the to-be-pushed data to the Rocket MQ message middleware through the preset sinking component, so that the data is pushed to the Rocket MQ from the Flume, and the data can be queried and traced on the Rocket MQ. A distributed message mechanism is realized, and message query and message backtracking can be supported.

Description

Technical Field

[0001] The present invention relates to the technical field of distributed data processing, and in particular to a data collection method, device, computer equipment and storage medium based on Flume. Background Art

[0002] Flume is a distributed, reliable, and highly available massive data aggregation system that supports collecting data from different types of data sources in the system, while providing the ability to simply process the data and sink it to different data receivers.

[0003] The current Flume component only supports Kafka for distributed messaging systems, but Kafka does not support distributed transaction messages, does not support message query mechanisms, and message backtracking is difficult. In business scenarios that require distributed transaction messages, support for message query, and message backtracking, Kafka is difficult to play a role. Summary of the invention

[0004] Based on this, it is necessary to provide a Flume-based data collection method, device, computer equipment and storage medium to address the above technical problems.

[0005] A Flume-based data collection method, including:

[0006] Receive the data to be pushed sent by the data extraction server based on the acquisition component;

[0007] Based on the transmission channel component, the data to be pushed acquired by the collection component is transmitted to the preset sinking component, and the preset sinking component generates a message producer;

[0008] The message producer is used to send the data to be pushed to the RocketMQ message middleware.

[0009] In one embodiment, the step of transmitting the data to be pushed acquired by the collection component to the preset sinking component based on the transmission channel component, and generating a message producer by the preset sinking component includes:

[0010] Based on the transmission channel component, the data to be pushed acquired by the collection component is transmitted to the preset sinking component;

[0011] Acquire a preset processing event in the transmission channel component based on the preset sinking component;

[0012] The message producer is generated by the preset sinking component;

[0013] The step of using the message producer to send the data to be pushed to the RocketMQ message middleware includes:

[0014] Based on the message producer, the data to be pushed and the preset processing event are sent to the RocketMQ message middleware.

[0015] In one of the embodiments, the preset processing event includes converting, filtering or aggregating data.

[0016] In one embodiment, the step of using the message producer to send the data to be pushed to the RocketMQ message middleware includes:

[0017] The message producer is used to send the data to be pushed to the RocketMQ message middleware based on the message queue mechanism.

[0018] In one of the embodiments, the preset sinking component is a RocketMQ Sink data sinking component.

[0019] A Flume-based data collection system, comprising:

[0020] The collection component is used to receive the data to be pushed sent by the data extraction server;

[0021] A transmission channel component, used for transmitting the data to be pushed acquired by the collection component to a preset sinking component, and the preset sinking component generates a message producer;

[0022] A sinking component is preset to use the message producer to send the data to be pushed to the RocketMQ message middleware.

[0023] A Flume-based data collection device, comprising:

[0024] A data receiving module, used for receiving the data to be pushed sent by the data extraction server based on the acquisition component;

[0025] A producer generation module, used for transmitting the data to be pushed acquired by the collection component to a preset sinking component based on the transmission channel component, and the preset sinking component generates a message producer;

[0026] The data sending module is used to use the message producer to send the data to be pushed to the RocketMQ message middleware.

[0027] In one embodiment, the producer generation module includes:

[0028] A sinking unit, configured to transmit the data to be pushed acquired by the collection component to a preset sinking component based on the transmission channel component;

[0029] An event acquisition unit, configured to acquire a preset processing event in the transmission channel component based on the preset sinking component;

[0030] A producer generating unit, configured to generate a message producer from the preset sinking component;

[0031] The data sending module is also used to use the message producer to send the data to be pushed and the preset processing event to the RocketMQ message middleware.

[0032] A computer device comprises a memory and a processor, wherein the memory stores a computer program, and wherein the processor implements the following steps when executing the computer program:

[0033] Receive the data to be pushed sent by the data extraction server based on the acquisition component;

[0034] Based on the transmission channel component, the data to be pushed acquired by the collection component is transmitted to the preset sinking component, and the preset sinking component generates a message producer;

[0035] The message producer is used to send the data to be pushed to the RocketMQ message middleware.

[0036] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the following steps:

[0037] Receive the data to be pushed sent by the data extraction server based on the acquisition component;

[0038] Based on the transmission channel component, the data to be pushed acquired by the collection component is transmitted to the preset sinking component, and the preset sinking component generates a message producer;

[0039] The message producer is used to send the data to be pushed to the RocketMQ message middleware.

[0040] The above-mentioned Flume-based data collection method, device, computer equipment and storage medium, in this solution, based on the Flume data collection process, through the preset sinking component, the message producer sends the data to be pushed to the RocketMQ message middleware, and realizes the data being pushed from Flume to RocketMQ, so that the data can be queried and traced on RocketMQ. Not only a distributed message mechanism is realized, but also message query and message backtracking can be supported. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 A flowchart of a data collection method based on Flume in one embodiment;

[0042] Figure 2 is a structural block diagram of a Flume-based data collection device in one embodiment;

[0043] Figure 3 is an internal structure diagram of a computer device in one embodiment;

[0044] Figure 4 A schematic diagram of the logical structure of a Flume component in an embodiment;

[0045] Figure 5 A schematic diagram of the logical structure of a preset sinking component in an embodiment. DETAILED DESCRIPTION

[0046] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0047] It should be understood that the Agent commonly used in traditional Flume components is an independent Flume data collection process, which mainly includes the following subcomponents:

[0048] Source (collection subcomponent) is used to connect to the data source to obtain data; it has various built-in implementations.

[0049] Sink (sinking subcomponent) is used to transfer data to the next level agent or to the final storage system.

[0050] Channel (transmission channel subcomponent), used to transfer data from source to sink.

[0051] Sink is also called data sinking component. Existing data sinking components mainly include HDFS Sink, Hive Sink, Kafka Sink and other components. Flume components only support Kafka for distributed message systems, but Kafka does not support distributed transaction messages, does not support message query mechanism, and message backtracking is difficult. In business scenarios where distributed transaction messages need to be guaranteed and message query and message backtracking need to be supported, it is difficult for Kafka to play a role. For this reason, a distributed message system sinking component that can support distributed transaction messages, message query, and message backtracking is needed. To solve this problem, the present application adopts the following embodiments to solve it.

[0052] Embodiment 1

[0053] In this embodiment, Figure 1 As shown, a Flume-based data collection method is provided, which includes:

[0054] Step 110: extract the data to be pushed sent by the server based on the data collection component.

[0055] In this embodiment, Figure 4 As shown, the acquisition component is a Source component, which is used to connect with the data source to obtain data. In this embodiment, the data extraction service is the data source, so the acquisition data obtains the data to be pushed from the data extraction server.

[0056] Step 120: Based on the transmission channel component, the data to be pushed acquired by the collection component is transmitted to the preset sinking component, and the preset sinking component generates a message producer.

[0057] In this embodiment, Figure 4 As shown, the transmission channel component is Channel, and the preset sink component is a RocketMQ Sink data sink component; wherein, the transmission channel component is used to transfer the data to be pushed from the collection component to the preset sink component.

[0058] In this embodiment, Figure 5 As shown, the preset sink component RocketMQ sink includes NamedComponent, LifecycleAware, Sink, Configurable, and AbstractSink.

[0059] NamedComponent is an abstract class that represents the basic properties and behaviors of RocketMQ sink components. It defines basic properties such as component name, type, and state, and provides methods to manage and control the life cycle of components.

[0060] LifecycleAware is an interface that indicates that a component implements methods related to lifecycle management. It contains three methods: start(), stop(), and getLifecycleState(), and stipulates that components must implement these methods in order to be started and stopped correctly.

[0061] Sink is a very important component used to take Flume's processing time out of the Channel and transmit it to the next receiver. Sink is usually the final destination of the data stream and can send data to various storage systems or other data consumers.

[0062] Specifically, the Sink component is responsible for the following tasks: Get Flume Events from the Channel and send them to the next target. Process events, such as conversion, filtering, or aggregation. Provide fault tolerance and reliability guarantees for data transmission, such as batch submission mechanisms to ensure that data is not lost.

[0063] AbstractSink is an abstract class that encapsulates some common properties and methods of Sink components. If you need to write a custom Sink component, you can do so by inheriting the AbstractSink class.

[0064] Specifically, the AbstractSink class provides the following functions: Provides a set of default event processing logic, including event transmission, transaction management, etc. Implements lifecycle management related methods, such as start(), stop(), and getLifecycleState() methods. Provides some tool methods, such as the getChannelProcessor() method, which is used to obtain the Channel Processor object in the current context.

[0065] The Configurable class is an interface that provides a method configure(Configuration conf) that is used to configure components with custom properties. The Configurable interface allows developers to use configuration files as input in custom interceptors and channels and convert them into objects that can process information.

[0066] Step 130: Use the message producer to send the data to be pushed to RocketMQ message middleware.

[0067] It should be understood that the preset sink component in this embodiment is RocketMQ sink, and RocketMQ sink supports data transmission with RocketMQ.

[0068] In addition, the preset sinking component will generate a producer, namely RocketMQ Producer, which sends the data transmitted by the transmission channel component to RocketMQ.

[0069] In the above embodiment, based on the data collection process of Flume, data transmission with RocketMQ message middleware can be realized by presetting the sinking component. The message producer sends the data to be pushed to RocketMQ message middleware, and the data is pushed from Flume to RocketMQ, so that the data can be queried and traced on RocketMQ. Not only the distributed message mechanism is realized, but also message query and message backtracking can be supported.

[0070] In one embodiment, the step of transmitting the data to be pushed acquired by the collection component to the preset sinking component based on the transmission channel component, and generating a message producer by the preset sinking component includes: transmitting the data to be pushed acquired by the collection component to the preset sinking component based on the transmission channel component; acquiring the preset processing event in the transmission channel component based on the preset sinking component; generating a message producer by the preset sinking component; the step of sending the data to be pushed to the RocketMQ message middleware by using the message producer includes: sending the data to be pushed and the preset processing event to the RocketMQ message middleware based on the message producer.

[0071] In this embodiment, the preset sinking component includes a Sink component, which is used to obtain a processing event (Flume Event) from the transmission channel component and send it to the next target. In this embodiment, the Sink component of the preset sinking component obtains the processing event from the transmission channel component, and sends the processing event and the data to be pushed to the RocketMQ message middleware. In one embodiment, the preset processing event includes converting, filtering or aggregating the data. In this way, fault tolerance and reliability guarantees for data transmission can be provided, such as batch submission mechanisms to ensure that data is not lost.

[0072] In one embodiment, the step of using the message producer to send the data to be pushed to RocketMQ message middleware includes: using the message producer to send the data to be pushed to RocketMQ message middleware based on a message queue mechanism.

[0073] In this embodiment, RocketMQ message middleware is a queue model message middleware with high performance, high reliability, high real-time and distributed characteristics. Producers, consumers and queues can all be distributed; Producers send messages to some queues in turn, and the queue collection is called a Topic. If the Consumer performs broadcast consumption, one consumer instance consumes all queues corresponding to this Topic. If cluster consumption is performed, multiple Consumer instances consume the queue collection corresponding to this topic on average.

[0074] It should be understood that although Figure 1 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 1 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0075] Embodiment 2

[0076] In this embodiment, Figure 4 As shown, a Flume-based data collection system is provided, including:

[0077] The collection component is used to receive the data to be pushed sent by the data extraction server; in this embodiment, the collection component is Source.

[0078] The transmission channel component is used to transmit the data to be pushed obtained by the collection component to the preset sinking component, and the preset sinking component generates a message producer; in this embodiment, the transmission channel component is a Channel component.

[0079] A sinking component is preset to use the message producer to send the data to be pushed to the RocketMQ message middleware.

[0080] In this embodiment, the preset sink component is a RocketMQ Sink data sink component. The RocketMQ Sink data sink component can realize the transmission of the data to be pushed to RocketMQ, wherein, Figure 5As shown, RocketMQ Sink data sink components include NamedComponent, LifecycleAware, Sink, Configurable, and AbstractSink.

[0081] NamedComponent is an abstract class that represents the basic properties and behaviors of RocketMQ sink components. It defines basic properties such as component name, type, and state, and provides methods to manage and control the life cycle of components.

[0082] LifecycleAware is an interface that indicates that a component implements methods related to lifecycle management. It contains three methods: start(), stop(), and getLifecycleState(), and stipulates that components must implement these methods in order to be started and stopped correctly.

[0083] Sink is a very important component used to take Flume's processing time out of the Channel and transmit it to the next receiver. Sink is usually the final destination of the data stream and can send data to various storage systems or other data consumers.

[0084] Specifically, the Sink component is responsible for the following tasks: Get Flume Events from the Channel and send them to the next target. Process events, such as conversion, filtering, or aggregation. Provide fault tolerance and reliability guarantees for data transmission, such as batch submission mechanisms to ensure that data is not lost.

[0085] AbstractSink is an abstract class that encapsulates some common properties and methods of Sink components. If you need to write a custom Sink component, you can do so by inheriting the AbstractSink class.

[0086] Specifically, the AbstractSink class provides the following functions: Provides a set of default event processing logic, including event transmission, transaction management, etc. Implements lifecycle management related methods, such as start(), stop(), and getLifecycleState() methods. Provides some tool methods, such as the getChannelProcessor() method, which is used to obtain the Channel Processor object in the current context.

[0087] The Configurable class is an interface that provides a method configure(Configuration conf) that is used to configure components with custom properties. The Configurable interface allows developers to use configuration files as input in custom interceptors and channels and convert them into objects that can process information.

[0088] The RocketMQ Sink data sink component implements dynamic configuration of NameServer, Topic, Tag, ProducerId, and GroupId, and receives Flume Channel data streams.

[0089] RocketMQsink data sink component inherits the AbstractSink class and implements Configurable configuration. Configurable is used to configure RocketMQ related information (NameServer, Topic, tag, ProducerId, GroupId), while inheriting AbstractSink is mainly used for integration with Flume.

[0090] Data is output to the Channel through the Source, and then output to the RocketMQ Sink component through the Channel to be sent to the RocketMQ Topic.

[0091] The specific implementation process is as follows:

[0092] #Name the components on this agent

[0093] a1.sources=r1

[0094] a1.sinks=k1

[0095] a1.channels=c1

[0096] #souce

[0097] a1.sources.r1.type=org.apache.flume.source.kafka.KafkaSource

[0098] a1.sources.r1.batchSize = 300

[0099] #a1.sources.r1.batchDurationMillis=20000

[0100] a1.sources.r1.kafka.bootstrap.servers=bootstrap:9092

[0101] a1.sources.r1.kafka.topics=test_topic

[0102] a1.sources.r1.kafka.consumer.group.id=test_group

[0103] a1.sources.r1.kafka.consumer.auto.offset.reset=earliest

[0104] ##################Configure sinks to write to ONS###################

[0105] #define sink ONS

[0106] #Set sink (required)

[0107] a1.sinks.k1.type=com.pcitc.bigdata.sink.RocketMQsink

[0108] #Set PID (required)

[0109] a1.sinks.k1.GROUP_ID=GID_test

[0110] #Set AccessKey (required)

[0111] a1.sinks.k1.AccessKey=AccessKey

[0112] #Set SecretKey (required)

[0113] a1.sinks.k1.SecretKey=SecretKey

[0114] #Set the sending timeout in milliseconds

[0115] a1.sinks.k1.SendMsgTimeoutMillis=100000

[0116] #Set ONS TCP access domain name (required)

[0117] a1.sinks.k1.NAMESRV_ADDR=ons_url

[0118] #Set topic (required)

[0119] a1.sinks.k1.topic=test_ons_topic

[0120] #Set tag

[0121] a1.sinks.k1.tag=*

[0122] #Whether to enable debug logging

[0123] a1.sinks.k1.isDeubg=false

[0124] a1.sinks.k1.layout=%-d{yyyy-MM-dd HH:mm:ss}[%l:%r ms]-[%p]%m%n

[0125] a1.sinks.k1.filename= / data / 1 / flume_jiangsu / ons / log.log

[0126] a1.sinks.k1.datePattern='.'yyyy-MM-dd'.log'

[0127] a1.sinks.k1.maxBackups=1

[0128] a1.sinks.k1.maxFileSize=1

[0129] a1.channels.c1.type=memory

[0130] a1.channels.c1.capacity=1000000

[0131] a1.channels.c1.transactionCapacity=300

[0132] a1.sources.r1.channels=c1

[0133] a1.sinks.k1.channel=c1

[0134] Embodiment 3

[0135] In this embodiment, Figure 2 As shown, a Flume-based data collection device is provided, comprising:

[0136] The data receiving module 210 is used to receive the data to be pushed sent by the data extraction server based on the acquisition component;

[0137] The producer generation module 220 is used to transmit the data to be pushed acquired by the collection component to the preset sinking component based on the transmission channel component, and the preset sinking component generates a message producer;

[0138] The data sending module 230 is used to send the data to be pushed to the RocketMQ message middleware by using the message producer.

[0139] In one embodiment, the producer generation module includes:

[0140] A sinking unit, configured to transmit the data to be pushed acquired by the collection component to a preset sinking component based on the transmission channel component;

[0141] An event acquisition unit, configured to acquire a preset processing event in the transmission channel component based on the preset sinking component;

[0142] A producer generating unit, configured to generate a message producer from the preset sinking component;

[0143] The data sending module is also used to use the message producer to send the data to be pushed and the preset processing event to the RocketMQ message middleware.

[0144] In one embodiment, the preset processing event includes converting, filtering or aggregating data.

[0145] In one embodiment, the data sending module is further used to utilize the message producer to send the data to be pushed to RocketMQ message middleware based on a message queue mechanism.

[0146] In one embodiment, the preset sinking component is a RocketMQ Sink data sinking component.

[0147] For the specific definition of the Flume-based data collection device, please refer to the definition of the Flume-based data collection method above, which will not be repeated here. Each unit in the above-mentioned Flume-based data collection device can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned units can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above-mentioned units.

[0148] Embodiment 4

[0149] In this embodiment, a computer device is provided. Its internal structure diagram can be shown as follows: Figure 3 As shown. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program, and the non-volatile storage medium is deployed with a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with other computer devices deployed with application software. When the computer program is executed by the processor, a Flume-based data collection method is implemented. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covered on the display screen, or a key, trackball or touchpad set on the computer device housing, or an external keyboard, touchpad or mouse, etc.

[0150] Those skilled in the art will understand that Figure 3 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0151] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0152] Receive the data to be pushed sent by the data extraction server based on the acquisition component;

[0153] Based on the transmission channel component, the data to be pushed acquired by the collection component is transmitted to the preset sinking component, and the preset sinking component generates a message producer;

[0154] The message producer is used to send the data to be pushed to the RocketMQ message middleware.

[0155] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0156] Based on the transmission channel component, the data to be pushed acquired by the collection component is transmitted to the preset sinking component;

[0157] Acquire a preset processing event in the transmission channel component based on the preset sinking component;

[0158] The message producer is generated by the preset sinking component;

[0159] Based on the message producer, the data to be pushed and the preset processing event are sent to the RocketMQ message middleware.

[0160] In one embodiment, the preset processing event includes converting, filtering or aggregating data.

[0161] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0162] The message producer is used to send the data to be pushed to the RocketMQ message middleware based on the message queue mechanism.

[0163] In one embodiment, the preset sinking component is a RocketMQ Sink data sinking component.

[0164] Embodiment 5

[0165] In this embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the following steps are implemented:

[0166] Receive the data to be pushed sent by the data extraction server based on the acquisition component;

[0167] Based on the transmission channel component, the data to be pushed acquired by the collection component is transmitted to the preset sinking component, and the preset sinking component generates a message producer;

[0168] The message producer is used to send the data to be pushed to the RocketMQ message middleware.

[0169] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0170] Based on the transmission channel component, the data to be pushed acquired by the collection component is transmitted to the preset sinking component;

[0171] Acquire a preset processing event in the transmission channel component based on the preset sinking component;

[0172] The message producer is generated by the preset sinking component;

[0173] Based on the message producer, the data to be pushed and the preset processing event are sent to the RocketMQ message middleware.

[0174] In one embodiment, the preset processing event includes converting, filtering or aggregating data.

[0175] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0176] The message producer is used to send the data to be pushed to the RocketMQ message middleware based on the message queue mechanism.

[0177] In one embodiment, the preset sinking component is a RocketMQ Sink data sinking component.

[0178] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0179] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0180] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A data collection method based on Flume, characterized in that: include: Receive the data to be pushed sent by the data extraction server based on the acquisition component; Based on the transmission channel component, the data to be pushed acquired by the collection component is transmitted to the preset sinking component, and the preset sinking component generates a message producer; The message producer is used to send the data to be pushed to the RocketMQ message middleware.

2. The method according to claim 1, characterized in that The step of transmitting the data to be pushed acquired by the collection component to the preset sinking component based on the transmission channel component, and generating a message producer by the preset sinking component includes: Based on the transmission channel component, the data to be pushed acquired by the collection component is transmitted to the preset sinking component; Acquire a preset processing event in the transmission channel component based on the preset sinking component; The message producer is generated by the preset sinking component; The step of using the message producer to send the data to be pushed to the RocketMQ message middleware includes: Based on the message producer, the data to be pushed and the preset processing event are sent to the RocketMQ message middleware.

3. The method according to claim 2, characterized in that The preset processing event includes converting, filtering or aggregating data.

4. The method according to claim 1, characterized in that: The step of using the message producer to send the data to be pushed to the RocketMQ message middleware includes: The message producer is used to send the data to be pushed to the RocketMQ message middleware based on the message queue mechanism.

5. The method according to any one of claims 1 to 4, characterized in that: The preset sinking component is the RocketMQSink data sinking component.

6. A Flume-based data collection system, characterized in that: include: The collection component is used to receive the data to be pushed sent by the data extraction server; A transmission channel component, used for transmitting the data to be pushed acquired by the collection component to a preset sinking component, and the preset sinking component generates a message producer; A sinking component is preset to use the message producer to send the data to be pushed to the RocketMQ message middleware.

7. A data collection device based on Flume, characterized in that: include: A data receiving module, used for receiving the data to be pushed sent by the data extraction server based on the acquisition component; A producer generation module, used for transmitting the data to be pushed acquired by the collection component to a preset sinking component based on the transmission channel component, and the preset sinking component generates a message producer; The data sending module is used to use the message producer to send the data to be pushed to the RocketMQ message middleware.

8. The device according to claim 7, characterized in that The producer generation module includes: A sinking unit, configured to transmit the data to be pushed acquired by the collection component to a preset sinking component based on the transmission channel component; An event acquisition unit, configured to acquire a preset processing event in the transmission channel component based on the preset sinking component; A producer generating unit, configured to generate a message producer from the preset sinking component; The data sending module is also used to use the message producer to send the data to be pushed and the preset processing event to the RocketMQ message middleware.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 5 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.

Images