Log collection method, device and equipment of flume component and storage medium

Abstract

The application provides a log collection method and device of a Flume component, equipment and a storage medium. The method comprises the following steps: a source end receives a log request sent by a source port, and sends log information in the log request to a selection end; the selection end queries a log platform according to IP information in the log information, acquires a task ID corresponding to the log information, and acquires a transmission pipeline channel matched with the task ID; the selection end sends the log information to the transmission pipeline channel, so that a sink end at the end of the transmission pipeline channel analyzes the log information, and stores log content acquired after the analysis into a log data table in the sink end, thereby solving the problem of port occupation caused by too many collection tasks, and improving the log collection efficiency.

Description

Technical Field

[0001] This application relates to the field of log collection, and in particular to log collection methods, apparatus, devices and storage media for Flume components. Background Technology

[0002] With the deepening implementation of my country's digital economy development strategy and the continuous improvement of digital infrastructure, the number of various computing devices and application software on the Internet has experienced explosive growth in order to meet the people's urgent needs for various digital services and accelerate the cultivation of new digital economy business forms and models. Simultaneously, because these massive computing devices and application software process a huge number of requests, they generate massive amounts of system logs. These system logs contain a wealth of valuable information and are an important data resource for enterprises and entities. Therefore, in order to fully leverage the value of these system logs, how to collect these massive amounts of system logs has become a problem that must be solved in the process of digital economy development.

[0003] In existing technologies, massive amounts of system logs can be collected using Flume middleware. Flume creates a log collection process using an agent as the unit. This agent can use a custom log source, transmission channel, and destination sink to transmit and collect log data distributed across different types of storage media. Simultaneously, the destination sink can also obtain the collected data from the transmission channel for business operations.

[0004] Currently, most vendors adopt the "one collection task, one agent" approach. This means that when there are multiple collection tasks, the agent corresponding to each collection task needs to be assigned to a different port. Therefore, when there are too many collection tasks, there will be situations where the port is occupied and cannot be collected, and the port needs to be occupied repeatedly, resulting in relatively low log collection efficiency. Summary of the Invention

[0005] This application provides a log collection method, apparatus, device, and storage medium for Flume components to solve the problem in the prior art where, when using Flume to perform log collection tasks, too many collection tasks lead to port occupation, resulting in the inability to collect logs or the need to repeatedly occupy ports, thus causing relatively low log collection efficiency.

[0006] Firstly, this application provides a log collection and processing method based on Flume components, wherein the agent in the Flume component includes: a source, a selector, a channel, and a sink, and the method includes:

[0007] The source terminal receives a log request sent by the source port of the source device, and sends the log information to be collected in the log request to the selector terminal based on the log request.

[0008] The selector queries the log platform based on the IP information in the log information to obtain the task ID corresponding to the log information, and obtains the transmission channel that matches the task ID.

[0009] The selector sends log information to the transmission channel, so that the destination sink at the end of the transmission channel can parse the log information and store the parsed log content into the log data table in the destination sink.

[0010] In the preferred technical solution of the above-mentioned log collection and processing method based on Flume components, the following are included:

[0011] From the log platform, obtain the newly added collection task, which includes: task ID, source port to be collected, destination port to be collected, and log filtering conditions;

[0012] Determine whether the source port to be collected in the collection task is the same as the historical source port in the historical collection task;

[0013] If it is determined that the source port to be collected in the collection task is different from the historical source port in the historical collection task, then a first configuration file is created on the source port to be collected, and a first new transmission channel and a first new destination sink corresponding to the task ID are created in the first configuration file.

[0014] Based on the first configuration file, a first flume-ng process is created on the source port to be collected, and the first flume-ng process is started.

[0015] In the preferred technical solution of the above-mentioned log collection and processing method based on Flume components, the following is also included:

[0016] If it is determined that the source port to be collected in the collection task is the same as any historical port in the historical collection task, then a second new transmission channel and a second new destination sink corresponding to the task ID are added to the second configuration file corresponding to the same source port.

[0017] Based on the second configuration file, start the second flume-ng process on the same source port.

[0018] In the preferred embodiment of the above-mentioned log collection and processing method based on Flume components, the selector queries the log platform based on the IP information in the log information to obtain the task ID corresponding to the IP information, and obtains the transmission channel matching the task ID, including:

[0019] The selector queries the log filtering conditions in each collection task stored in the log platform to obtain the task ID corresponding to the IP information, and uses the transmission channel corresponding to the destination port to be collected that matches the task ID as the transmission channel that matches the task ID.

[0020] In the preferred technical solution of the above-mentioned log collection and processing method based on Flume components, the following is also included:

[0021] When obtaining a paused or terminated task from the log platform, based on the task ID to be paused in the paused task or the task ID to be terminated in the terminated task, delete the transmission channel and destination sink in the configuration file to which the task ID to be paused or terminated belongs.

[0022] In the preferred technical solution of the above-mentioned log collection and processing method based on Flume components, the following is also included:

[0023] If the deleted transport channel is the last channel among all transport channels on the source port corresponding to the deleted transport channel, then delete the created configuration file on the source port and delete the flume-ng process corresponding to the created configuration file.

[0024] In the preferred technical solution of the above-mentioned log collection and processing method based on Flume components, the following is also included:

[0025] When flume-ng detects a change in the configuration file to which the task ID to be paused or the task ID to be terminated belongs, it obtains the new configuration file and rearranges it based on the new configuration file.

[0026] Secondly, this application provides a log collection and processing device based on a Flume component, including: the Flume component, and the Flume component includes an agent, the agent including: a source, a selector, a channel, and a sink;

[0027] The source terminal is used to receive log requests sent by the source port of the source device, and based on the log requests, send the log information to be collected in the log requests to the selector terminal.

[0028] The selector is used to query the log platform based on the IP information in the log information to obtain the task ID corresponding to the log information, and to obtain the transmission channel that matches the task ID.

[0029] The selector is also used to send log information to the transmission channel, so that the destination sink at the end of the transmission channel can parse the log information and store the parsed log content into the log data table in the destination sink.

[0030] Thirdly, embodiments of this application provide an electronic device, including: a processor, and a memory communicatively connected to the processor;

[0031] The memory stores computer-executed instructions;

[0032] The processor executes computer execution instructions stored in the memory to implement the method as described in any of the first aspects.

[0033] Fourthly, embodiments of this application provide a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the method described in any of the first aspects.

[0034] This application provides a log collection method, apparatus, device, and storage medium for the Flume component. The method involves a source receiving a log request from the source port of a source device. Based on the log request, the source sends the log information to be collected to a selector. The selector then queries the log platform based on the IP information in the log information to obtain the task ID corresponding to the log information and the corresponding transmission channel. Finally, the selector sends the log information to the transmission channel, enabling the destination sink at the end of the channel to parse the log information and store the parsed log content in the log data table of the destination sink. Compared to existing technologies that use a "one collection task, one agent" approach for log collection, which requires allocating each agent to a different port when there are multiple collection tasks, resulting in wasted port resources and port congestion when there are too many collection tasks, this method offers a significant advantage. This application addresses the problems of insufficient log collection and redundant port usage, leading to relatively low log collection efficiency. It proposes a Flume component topology and data flow method. By binding a source to a single port to receive log requests from different source devices, it solves the problems of wasted port resources and low log collection efficiency caused by binding a single collection task to a single port. Furthermore, this application uses a selector to query the log platform based on the IP information in the log information to obtain the corresponding task ID. Based on different task IDs, it obtains the matching transmission channel and destination sink, allowing log information with different characteristics to be parsed and collected in different ways. This differentiated parsing improves the scalability of the collection system. Moreover, by selectively classifying and parsing log information instead of using different parsing rules to parse redundant data, the amount of log information processed is greatly reduced, saving system computing resources. Attached Figure Description

[0035] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the specification, serve to explain the principles of this application. The drawings are as follows:

[0036] Figure 1 This is a flowchart illustrating an embodiment of a log collection and processing method based on the Flume component provided in this embodiment.

[0037] Figure 2This is a flowchart illustrating a second embodiment of a log collection and processing method based on the Flume component provided in this embodiment;

[0038] Figure 3 This is a schematic diagram illustrating the principle of the flume-ng process started based on the configuration file when the source port to be collected is initially bound.

[0039] Figure 4 This is a schematic diagram illustrating the principle of the flume-ng process started based on an existing configuration file when the source port to be collected has been bound.

[0040] Figure 5 This is a flowchart illustrating a third embodiment of a log collection and processing method based on the Flume component provided in this embodiment;

[0041] Figure 6 A flowchart illustrating the process of reusing log collection from the same source port;

[0042] Figure 7 This is a flowchart illustrating Embodiment 4 of a log collection and processing method based on the Flume component provided in this embodiment;

[0043] Figure 8 A schematic diagram of the structure of a log collection and processing device based on Flume components, provided in this application embodiment;

[0044] Figure 9 This is a schematic diagram of the structure of an electronic device according to an embodiment of this application.

[0045] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0046] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0047] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with relevant laws, regulations and standards, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0048] With the deepening implementation of my country's digital economy development strategy and the continuous improvement of digital infrastructure, the number of various computing devices and application software on the Internet has experienced explosive growth in order to meet the people's urgent needs for various digital services and accelerate the cultivation of new digital economy business forms and models. Simultaneously, because these massive computing devices and application software process a huge number of requests, they generate massive amounts of system logs. These system logs contain a wealth of valuable information and are an important data resource for enterprises and entities. Therefore, in order to fully leverage the value of these system logs, how to collect these massive amounts of system logs has become a problem that must be solved in the process of digital economy development.

[0049] In existing technologies, massive amounts of system logs can be collected using Flume middleware. Flume creates a log collection process based on an agent. Currently, most vendors adopt the approach of "one collection task, one agent," which means that the agent corresponding to each collection task is assigned to a different port. As a result, when there are too many collection tasks, the port may be occupied, preventing collection, or the port may need to be occupied repeatedly, resulting in relatively low log collection efficiency.

[0050] Based on this, the inventive concept of this application lies in how to provide a new topology for Flume components that can effectively improve log collection efficiency.

[0051] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.

[0052] Figure 1 This is a flowchart illustrating an embodiment of a log collection and processing method based on the Flume component provided in this embodiment. Figure 1 As shown, the method includes the following steps:

[0053] Step S101: The source end receives the log request sent by the source port of the source device, and based on the log request, sends the log information to be collected in the log request to the selector end.

[0054] In this embodiment, the agent module in the Flume component includes: a source, a selector, a channel, and a sink.

[0055] For example, in a client-server (CS) data acquisition scenario, for a data acquisition task that initially binds to a source port of a source device, a configuration file is generated to bind that source port. This configuration file specifies a source and binds a selector. Simultaneously, the data acquisition task binds to a channel and a sink. Subsequent data acquisition tasks bound to the same source port can have a new channel and sink added after the selector. Thus, based on this source port, multiple data acquisition tasks can have their own corresponding channels and sinks. The source port can be a User Datagram Protocol (UDP) port.

[0056] Specifically, for this source port, the source terminal bound to it can receive the log requests it sends. Then, the source terminal sends the log information to be collected in the log request to the selector terminal bound to it.

[0057] Step S102: The selector queries the log platform based on the IP information in the log information to obtain the task ID corresponding to the log information, and obtains the transmission channel that matches the task ID.

[0058] Step S103: The selector sends the log information to the transmission channel, so that the destination sink at the end of the transmission channel can parse the log information and put the parsed log content into the log data table in the destination sink.

[0059] In this embodiment, the selector is bound to multiple transmission channels, and each transmission channel is bound to its corresponding destination sink. In other words, each task corresponds to a transmission channel and a destination sink. Based on this, the selector can selectively assign different tasks to different transmission channels.

[0060] Specifically, the selector can send the received log information to the transmission channel that matches the task ID based on the task ID. Then, the destination sink at the end of the transmission channel will parse the log information and store the parsed log content into the log data table in the destination sink.

[0061] Alternatively, each destination sink can contain corresponding parsing rules for parsing log information. In other words, different destination sinks can contain different parsing rules to parse log information with different characteristics (such as IP information, time information, etc.), thus enabling differentiated parsing of log information with different characteristics. This improves the scalability of the acquisition system. Furthermore, by selectively classifying, parsing, and storing log information, rather than using different parsing rules to parse redundant data, the amount of log information processed is greatly reduced, saving the computing resources used by the system.

[0062] Furthermore, this log data table is the log data table specified by the task ID corresponding to the matched log information to be collected. This log data table is used to store the log content obtained after parsing the log information.

[0063] In this embodiment, the source receives log requests sent by the source port of the source device and, based on the log requests, sends the log information to be collected in the log requests to the selector. Then, the selector queries the log platform according to the IP information in the log information to obtain the task ID corresponding to the log information and obtains the transmission channel that matches the task ID. Finally, the selector sends the log information to the transmission channel so that the sink at the end of the transmission channel can parse the log information and store the parsed log content into the log data table in the sink. Compared to existing technologies that use a "one collection task, one agent" approach for log collection, which wastes port resources by assigning each agent to a different port when there are multiple collection tasks, and leads to port congestion and duplicate port usage when there are too many collection tasks, resulting in relatively low log collection efficiency, this application constructs a collection source, a selection agent, a transmission channel, and a destination sink in the agent module of the Flume component. A port can be bound to both the collection source and the selection agent, and each task is bound to a transmission channel and a destination sink. This allows the selection agent to selectively assign different tasks to different transmission channels and destination sinks, enabling different parsing rules to process log information with different characteristics. This effectively achieves multi-port reuse and allows for targeted parsing of log information with different characteristics, thus improving log collection efficiency.

[0064] Figure 2 This is a flowchart illustrating a second embodiment of a log collection and processing method based on the Flume component provided in this embodiment. Figure 2 As shown, the method includes the following steps:

[0065] Step S201: Obtain the newly added collection task from the log platform. The newly added collection task includes: task ID, source port to be collected, destination port to be collected, and log filtering conditions.

[0066] In this embodiment, the data collection task can be generated in a CS data collection scenario and stored in the log platform. Optionally, the source port to be collected refers to the source port of the source device; the destination port to be collected refers to the destination sink. The log filtering criteria may include: the task ID and IP information corresponding to the data collection task.

[0067] Step S202: Determine whether the source port to be collected in the data acquisition task is the same as the historical source port in the historical data acquisition task. If not, proceed to step S203; if yes, proceed to step S205.

[0068] In this embodiment, when adding a new acquisition task, it is first determined whether the source port to be acquired in the acquisition task is the same as any historical source port in the historical acquisition task, that is, whether the same port is reused. If the source port to be acquired in the acquisition task is not occupied, that is, the same port is not reused, then step S203 is executed. If the source port to be acquired in the acquisition task has been occupied, that is, the same port is reused, then step S205 is executed.

[0069] Step S203: Create the first configuration file on the source port to be collected, and create the first new transmission channel and the first new destination sink corresponding to the task ID in the first configuration file.

[0070] In this embodiment, if the source port to be collected for the task is not occupied, it is equivalent to binding the source port for the first time. Therefore, a first configuration file can be created on the source port to be collected. This first configuration file specifies a source and binds a selector. At the same time, a first new transmission channel and a first new destination sink are created based on the task ID.

[0071] Step S204: Based on the first configuration file, create the first flume-ng process on the source port to be collected, and start the first flume-ng process. End.

[0072] In this embodiment, for example, Figure 3 This is a schematic diagram illustrating the principle of the flume-ng process started based on the configuration file when the source port to be collected is initially bound, as shown below. Figure 3 As shown, once the first configuration file for a new data collection task is created, the first flume-ng process on the source port to be collected will be created and started, based on the data collection source, selector, first new transmission channel, and first new destination sink with corresponding parsing rules configured in the first configuration file.

[0073] Step S205: Add a second new transmission channel and a second new destination sink corresponding to the task ID on the second configuration file corresponding to the same source port.

[0074] In this embodiment, if the source port of the task to be collected is already occupied, a second new transmission channel and a second new destination sink corresponding to the task ID to be collected are added to the second configuration file already configured on the source port (i.e. the same port).

[0075] Step S206: Based on the second configuration file, start a second flume-ng process on the same source port.

[0076] In this embodiment, since the source port has been reused, i.e., a configuration file (i.e., the second configuration file) has been configured, there is no need to generate a new configuration file. That is, to add a second new transmission channel based on the task ID to be collected and a second new destination sink with corresponding parsing rules to the original configuration file (i.e., the second configuration file) and start the original second flume-ng process of the second configuration file.

[0077] thus, Figure 4 This is a schematic diagram illustrating the principle of the flume-ng process started based on an existing configuration file when the source port to be collected has been bound. Figure 4 As shown, subsequent collection tasks bound to the same source port can have their corresponding transmission channel and destination sink created. Thus, after the source collects log information, the selector sends it to the transmission channel of different task IDs according to the different characteristics of the log information (such as IP information), and then sends it to the destination sink corresponding to that task ID, thereby realizing the reuse of the same source port.

[0078] In this embodiment, by determining whether the source port of the newly added collection task is the same as the historical source port in the historical collection task, if the source port of the newly added collection task is not occupied, the collection end (source) is specified in the configuration file to bind to the source port, and a selector is bound to it. Then, a transmission channel and a destination sink are created in the configuration file. If the source port of the newly added collection task is the same as any historical source port in the historical collection task, the transmission channel and destination sink of the collection task are added to the existing configuration file of the source port. This allows different log collection tasks to be collected using the same source port, solving the problems of the source port being occupied and unable to collect, as well as the need to repeatedly occupy the port, thus improving the efficiency of log collection.

[0079] Figure 5 This is a flowchart illustrating a third embodiment of a log collection and processing method based on the Flume component provided in this embodiment. Figure 1Based on the illustrated embodiments, as Figure 5 As shown, one specific implementation of step S102 is as follows:

[0080] Step S301: The selector queries the log filtering conditions of each collection task stored in the log platform to obtain the task ID corresponding to the IP information, and takes the transmission channel corresponding to the destination port to be collected that matches the task ID as the transmission channel that matches the task ID.

[0081] In this embodiment, for example, Figure 6 A flowchart illustrating the process of reusing log collection from the same source port is shown below. Figure 6 As shown, when reusing the same source port, after the source receives the log request from the source device, the source sends the log information to be collected in the log request to the selector. The selector queries the log filtering conditions in each collection task stored in the log platform according to the IP information carried in the log information, thereby obtaining the task ID corresponding to the IP information carried in the log information. The transmission channel and destination sink that match the ID are used as the transmission channel and destination sink of the log information to transmit the log information. Then, the sink at the end of the channel parses the log information and stores it in the log data table to be stored.

[0082] In this embodiment, the selector interacts with the log platform. When the selector sends the IP information accompanying the log information to the log platform, the log platform queries the stored log filtering conditions for each collection task based on the IP information carried by the log platform to obtain the task ID corresponding to the IP information. The transmission channel corresponding to the destination port to be collected that matches the task ID is then used as the transmission channel matching the task ID. Due to the specific data link structure of the Flume component and the capabilities of its selector, log information can be parsed only once. Compared to the original Flume multiplexing structure and log information processing, this application can selectively send log information to different transmission channels, rather than simply sending log information to all transmission channels as in the original Flume multiplexing scheme. This allows log information with different characteristics to be parsed in different ways, improving the scalability of the log collection system. Furthermore, this application uses the selector to perform differentiated parsing of log information with different characteristics. By selectively classifying and storing log information, rather than using different parsing rules to parse redundant data, the amount of log information processed is greatly reduced, saving the computing resources used by the system.

[0083] Figure 7 This is a flowchart illustrating a fourth embodiment of a log collection and processing method based on the Flume component, which is based on the above embodiments, such as... Figure 7 As shown, the method may further include the following steps:

[0084] Step S401: When obtaining a paused task or a terminated task from the log platform, based on the task ID to be paused in the paused task or the task ID to be terminated in the terminated task, delete the transmission channel and destination sink in the configuration file to which the task ID to be paused or terminated belongs.

[0085] Optionally, the method may further include:

[0086] Step S402: If the deleted transmission channel is the last channel among all transmission channels on the source port corresponding to the deleted transmission channel, then delete the created configuration file on the source port and delete the flume-ng process corresponding to the created configuration file.

[0087] In this embodiment, if there are no other data acquisition tasks executing on the port matched by the task ID to be paused in the paused task or the task to be terminated in the terminated task, that is, if the deleted transmission channel is the last channel among all transmission channels on the source port corresponding to the deleted transmission channel, then the configuration file already created on that source port is deleted, and the flume-ng process corresponding to the configuration file already created is also deleted.

[0088] Alternatively, the method may further include:

[0089] When flume-ng detects a change in the configuration file of a task ID to be paused or terminated, it retrieves the new configuration file and rearranges the program based on the new configuration file.

[0090] In this embodiment, if there are other data acquisition tasks executing on the port matched by the task ID to be paused in the paused task or the task to be terminated in the terminated task, only the transmission pipe channel corresponding to the task ID in the configuration file on that source port is deleted. When flume-ng detects that the configuration file to which the task ID to be paused or terminated belongs has been modified, it will automatically rearrange the data according to the new configuration file.

[0091] The following is combined Figure 8 A detailed description is provided of a log collection and processing device based on Flume components provided in the embodiments. Figure 8 A schematic diagram of an embodiment of a log collection and processing device based on Flume components provided in this application; as shown below. Figure 8 As shown, the device includes a Flume component, and the Flume component includes an agent, which includes a source 81, a selector 82, a channel 83, and a sink 84.

[0092] The acquisition terminal (source81) receives log requests from the source port of the source device and, based on the log requests, sends the log information to be collected to the selection terminal (selector). The selection terminal (selector82) queries the log platform based on the IP information in the log information to obtain the task ID corresponding to the log information and the transmission channel matching the task ID. The selection terminal (selector81) also sends the log information to the transmission channel, enabling the destination sink at the end of the transmission channel to parse the log information and store the parsed log content in the log data table of the destination sink.

[0093] This embodiment of the log collection and processing device based on the Flume component can perform the above-described... Figure 1 , Figure 2 , Figure 5 as well as Figure 7 The implementation principles and effects of any of the method embodiments are similar, and will not be described again here.

[0094] Figure 9 This is a schematic diagram illustrating the structure of an electronic device according to an embodiment of this application. See also... Figure 9 The electronic device provided in this embodiment includes at least one processor 91 and a memory 92. The processor 91 and the memory 92 are connected via a bus 93.

[0095] The specific implementation process of processor 91 can be found in the above method embodiments, and its implementation principle and technical effect are similar. It will not be repeated here.

[0096] In the above Figure 9 In the illustrated embodiments, it should be understood that the processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this invention can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules within the processor.

[0097] The memory may include random access memory (RAM) and may also include non-volatile memory (NVM), such as at least one disk storage device.

[0098] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, the buses shown in the accompanying drawings are not limited to a single bus or a single type of bus.

[0099] This application also provides a computer-readable storage medium storing computer-executable instructions thereon, which, when executed by a processor, implement the above-described log collection and processing method based on Flume components.

[0100] The aforementioned computer-readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The computer-readable storage medium can be any available medium accessible to a general-purpose or special-purpose computer.

[0101] The aforementioned computer-readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The computer-readable storage medium can be any available medium accessible to a general-purpose or special-purpose computer.

[0102] An exemplary readable storage medium is coupled to a processor, enabling the processor to read information from and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can reside in an Application Specific Integrated Circuit (ASIC). Alternatively, the processor and the readable storage medium can exist as discrete components in the device.

[0103] The division of units is merely a logical functional division; in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices, or units, and may be electrical, mechanical, or other forms.

[0104] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0105] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0106] If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0107] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily essential to this application.

[0108] It should be further noted that although the steps in the flowchart are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowchart may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0109] It should be understood that the above-described device embodiments are merely illustrative, and the device of this application can also be implemented in other ways. For example, the division of units / modules in the above embodiments is only a logical functional division, and there may be other division methods in actual implementation. For example, multiple units, modules, or components may be combined, or integrated into another system, or some features may be ignored or not executed.

[0110] Furthermore, unless otherwise specified, the functional units / modules in the various embodiments of this application can be integrated into one unit / module, or each unit / module can exist physically separately, or two or more units / modules can be integrated together. The integrated units / modules described above can be implemented in hardware or as software program modules.

[0111] When integrated units / modules are implemented in hardware, the hardware can be digital circuits, analog circuits, etc. The physical implementation of the hardware structure includes, but is not limited to, transistors, memristors, etc. Unless otherwise specified, the processor can be any suitable hardware processor, such as a CPU, GPU, FPGA, DSP, and ASIC, etc. Unless otherwise specified, the storage unit can be any suitable magnetic or magneto-optical storage medium, such as Resistive Random Access Memory (RRAM), Dynamic Random Access Memory (DRAM), Static Random Access Memory (SRAM), Enhanced Dynamic Random Access Memory (EDRAM), High-Bandwidth Memory (HBM), Hybrid Memory Cube (HMC), etc.

[0112] If the integrated unit / module is implemented as a software program module and sold or used as an independent product, it can be stored in a computer-readable storage device (CMD). Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a memory and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned memory includes various media capable of storing program code, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard drive, magnetic disk, or optical disk.

[0113] In the above embodiments, the descriptions of each embodiment have their own emphasis. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments. The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as these combinations of technical features do not contradict each other, they should be considered within the scope of this specification.

[0114] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this application are indicated by the following claims.

[0115] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. A log collection and processing method based on Flume components, characterized in that, The agent module in the Flume component includes: a source, a selector, a channel, and a sink. The method includes: Get new collection tasks from the log platform. The new collection tasks include: task ID, source port to be collected, destination port to be collected, and log filtering conditions. Determine whether the source port to be collected in the collection task is the same as the historical source port in the historical collection task; If it is determined that the source port to be collected in the collection task is different from the historical source port in the historical collection task, then a first configuration file is created on the source port to be collected, and a first new transmission channel and a first new destination sink corresponding to the task ID are created in the first configuration file; based on the first configuration file, a first flume-ng process is created on the source port to be collected, and the first flume-ng process is started. If it is determined that the source port to be collected in the collection task is the same as any historical port in the historical collection task, then a second new transmission channel and a second new destination sink corresponding to the task ID are added to the second configuration file corresponding to the same source port; based on the second configuration file, the second flume-ng process on the same source port is started. The source terminal receives a log request sent by the source port of the source device, and sends the log information to be collected in the log request to the selector terminal based on the log request. The selector queries the log platform based on the IP information in the log information to obtain the task ID corresponding to the log information, and obtains the transmission channel that matches the task ID. The selector sends log information to the transmission channel, so that the sink at the end of the transmission channel can parse the log information and store the parsed log content into the log data table in the sink.

2. The method according to claim 1, characterized in that, The selector queries the log platform based on the IP information in the log information to obtain the task ID corresponding to the IP information, and obtains the transmission channel matching the task ID, including: The selector queries the log filtering conditions in each collection task stored in the log platform to obtain the task ID corresponding to the IP information, and uses the transmission channel corresponding to the destination port to be collected that matches the task ID as the transmission channel that matches the task ID.

3. The method according to any one of claims 1 to 2, characterized in that, Also includes: When obtaining a paused or terminated task from the log platform, based on the task ID to be paused in the paused task or the task ID to be terminated in the terminated task, delete the transmission channel and destination sink in the configuration file to which the task ID to be paused or terminated belongs.

4. The method according to claim 3, characterized in that, Also includes: If the deleted transport channel is the last channel among all transport channels on the source port corresponding to the deleted transport channel, then delete the created configuration file on the source port and delete the flume-ng process corresponding to the created configuration file.

5. The method according to claim 3, characterized in that, Also includes: When flume-ng detects a change in the configuration file to which the task ID to be paused or terminated belongs, it obtains the new configuration file and rearranges it based on the new configuration file.

6. A log collection and processing device based on Flume components, characterized in that, The device includes: the Flume component, and the Flume component includes an agent, the agent including: a source, a selector, a channel, and a sink; wherein, The judgment end is used to obtain new collection tasks from the log platform. The new collection task includes: task ID, source port to be collected, destination port to be collected, and log filtering conditions; it determines whether the source port to be collected in the collection task is the same as the historical source port in the historical collection tasks; if it is determined that the source port to be collected in the collection task is different from the historical source ports in the historical collection tasks, a first configuration file is created on the source port to be collected, and a first new transmission channel and a first new destination sink corresponding to the task ID are created in the first configuration file; based on the first configuration file, a first flume-ng process is created on the source port to be collected, and the first flume-ng process is started; if it is determined that the source port to be collected in the collection task is the same as any historical port in the historical collection tasks, a second new transmission channel and a second new destination sink corresponding to the task ID are added to the second configuration file corresponding to the same source port; based on the second configuration file, a second flume-ng process is started on the same source port. The source terminal is used to receive log requests sent by the source port of the source device, and based on the log requests, send the log information to be collected in the log requests to the selector terminal. The selector is used to query the log platform based on the IP information in the log information to obtain the task ID corresponding to the log information, and to obtain the transmission channel that matches the task ID. The selector is also used to send log information to the transmission channel, so that the destination sink at the end of the transmission channel can parse the log information and store the parsed log content into the log data table in the destination sink.

7. An electronic device, characterized in that, include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory to implement the method as described in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of claims 1 to 5.

Images