Log collection method and device, electronic equipment and storage medium
The nginx logs are obtained and processed through the target Filebeat, and stored in a relational database through the transition interface, solving the problem of difficulty in log collection, operation and maintenance deployment and complex configuration, and achieving lightweight log collection and readability.
Patent Information
- Application Number
- CN202311789859.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-12-22
- Publication Date
- 2025-06-24
AI Technical Summary
In the prior art, log collection has problems such as operation and maintenance deployment and complex configuration.
The nginx log of the target node is obtained through the target Filebeat and sent to the transition interface. The transition interface inputs the log into a relational database for storage.
It realizes a lightweight log collection method, reduces the workload of operation and maintenance deployment and the overhead of server resources, and makes nginx logs readable.
Smart Images
Figure CN120196500A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the technical field of data processing, and in particular, to a method and device for log collection, an electronic device, and a storage medium. Background Art
[0002] In recent years, due to the continuous growth of data in the Internet field, each company has faced the need to process massive amounts of data. Data analysis within a department mainly serves the operation and maintenance of each department of the company, and data analysis is mainly based on the logs generated by the servers of each department (such as apache logs, nginx logs, etc.). Therefore, log collection has become an essential operation before data analysis.
[0003] In related technologies, generally, big data analysis components such as flume are used to process nginx logs and perform log collection. However, big data analysis components such as flume must rely on other big data analysis components, which increases the difficulty of private deployment and makes the configuration complex; in addition, these analysis components for processing data streams have diverse data processing and transformation functions, which will make them more resource-consuming, that is, for the same function, more computer resources (such as CPU, memory) are required. Especially in the scenario of private deployment, hardware resources are more precious. Therefore, it is very important to reduce the consumption of hardware resources to achieve the goal.
[0004] Therefore, in related technologies, there are technical problems of large operation and maintenance deployment difficulty and complex configuration in order to achieve log collection. Summary of the Invention
[0005] This application provides a method and device for log collection, an electronic device, and a storage medium, so as to at least solve the technical problems of large operation and maintenance deployment difficulty and complex configuration in order to achieve log collection in related technologies.
[0006] According to one aspect of the embodiments of this application, a method for log collection is provided, including:
[0007] The target Filebeat obtains the target nginx log corresponding to the target node;
[0008] The target Filebeat sends the target nginx log to the transition interface;
[0009] The transition interface inputs the target nginx log into a relational database for storage.
[0010] Optionally, in the method as described above, the target Filebeat obtains the target nginx log corresponding to the target node, including:
[0011] The target Filebeat obtains at least one original nginx log of the target node;
[0012] The target Filebeat aggregates the at least one original nginx log according to a target time duration granularity to obtain the aggregated target nginx log.
[0013] Optionally, as in the foregoing method, before the target Filebeat aggregates the at least one original nginx log according to a target time duration granularity to obtain the aggregated target nginx log, the method further includes:
[0014] Obtain the write capacity of the relational database;
[0015] Determine a target frequency for initiating a write operation on the relational database according to the write capacity;
[0016] Determine the target time duration granularity according to the target frequency.
[0017] Optionally, as in the foregoing method, the target Filebeat aggregates the at least one original nginx log according to a target time duration granularity to obtain the aggregated target nginx log, including:
[0018] Determine preset fields that need to be stored in the relational database;
[0019] The target Filebeat extracts data from each original nginx log according to the preset fields to obtain each specified nginx log corresponding to the preset fields;
[0020] Aggregate all specific nginx logs in all specified nginx logs to obtain the aggregated target nginx log, where the specific nginx log is a specified nginx log whose generation time of the corresponding original nginx log is within a target time period, and the target time period is a time period with a duration corresponding to the target time duration granularity.
[0021] Optionally, as in the foregoing method, after the transition interface inputs the target nginx log into the relational database for storage, the method further includes:
[0022] The transition interface obtains success information fed back by the relational database, where the success information is used to indicate that the target nginx log is successfully stored in the relational database.
[0023] Optionally, as in the method described above, after obtaining the success information fed back by the relational database at the transition interface, the method further includes:
[0024] The transition interface sends the success information to the target Filebeat.
[0025] Optionally, as in the method described in any of the preceding items, the method further includes:
[0026] The transition interface obtains all historical nginx logs within a historical time period, where the historical time period is a time period corresponding to a specified duration granularity;
[0027] The transition interface aggregates all the historical nginx logs to obtain the second-aggregated nginx logs of the specified duration granularity;
[0028] The transition interface inputs the second-aggregated nginx logs into the relational database for storage.
[0029] According to another aspect of the embodiments of the present application, there is also provided a log collection device, including:
[0030] A Filebeat module, configured to obtain target nginx logs corresponding to a target node through a target Filebeat;
[0031] The Filebeat module is further configured to send the target nginx logs to a transition interface through the target Filebeat;
[0032] The transition interface inputs the target nginx logs into a relational database for storage.
[0033] According to yet another aspect of the embodiments of the present application, there is also provided an electronic device, including a processor, a communication interface, a memory, and a communication bus. Among them, the processor, the communication interface, and the memory complete mutual communication through the communication bus; among them, the memory is used to store a computer program; the processor is configured to execute the method steps in any of the above embodiments by running the computer program stored on the memory.
[0034] According to yet another aspect of the embodiments of the present application, there is also provided a computer-readable storage medium, in which a computer program is stored. Among them, the computer program is configured to execute the method steps in any of the above embodiments when running.
[0035] In the embodiment of the present application, the target Filebeat is used to obtain the target nginx log corresponding to the target node, thereby providing a lightweight log collection method, which can effectively reduce the workload of operation and maintenance deployment and can effectively reduce the overhead of server resources; and after the target Filebeat sends the target nginx log to the transition interface, the transition interface inputs the target nginx log into the relational database for storage; thus, the target nginx log can finally be read from the relational database, making the nginx log readable; in summary, the method of this embodiment achieves the technical effects of effectively reducing the workload of operation and maintenance deployment, effectively reducing the overhead of server resources, and making the nginx log readable; and thus solves the technical problem in the related art that it is difficult to deploy operation and maintenance and the configuration is complex in order to achieve log collection. BRIEF DESCRIPTION OF THE DRAWINGS
[0036] The accompanying drawings herein are incorporated into the specification and form a part of the specification, showing embodiments consistent with the present application, and are used together with the specification to explain the principles of the present application.
[0037] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the accompanying drawings required for use in the description of the embodiments or the prior art. Obviously, for those of ordinary skill in the art, other drawings can also be obtained based on these drawings without creative efforts.
[0038] Figure 1 It is a schematic diagram of the hardware environment of an optional log collection method according to an embodiment of the present application;
[0039] Figure 2 It is a schematic flowchart of an optional log collection method according to an embodiment of the present application;
[0040] Figure 3 It is a schematic diagram of another optional log collection method according to an embodiment of the present application;
[0041] Figure 4 It is a schematic flowchart of an optional log collection method according to an application example of the present application;
[0042] Figure 5 It is a structural block diagram of an optional log collection device according to an embodiment of the present application;
[0043] Figure 6 It is a structural block diagram of an optional electronic device according to an embodiment of the present application. DETAILED DESCRIPTION OF THE EMBODIMENTS
[0044] To enable those skilled in the art to better understand the solution of this application, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of this application. Obviously, the described embodiments are only a part of the embodiments of this application, rather than all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by those of ordinary skill in the art without making creative efforts shall fall within the protection scope of this application.
[0045] It should be noted that the terms "first", "second", etc. in the specification and claims of this application and the above-mentioned drawings are used to distinguish similar objects, and do not necessarily need to describe a specific order or sequence. It should be understood that such data can be interchanged under appropriate circumstances so that the embodiments of this application described here can be implemented in an order other than those illustrated or described here. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device that includes a series of steps or units does not necessarily have to be limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or are inherent to these processes, methods, products or devices.
[0046] First, some nouns or terms that appear during the description of the embodiments of this application are applicable to the following explanations:
[0047] 1. Object storage: It is a technology used to store and manage large-scale unstructured data. Different from traditional file systems or block storage, object storage organizes data into objects, and each object has a unique identifier (usually a URL or key), and contains the data itself and the associated metadata. Object storage is usually used to store and manage large-scale data in cloud computing environments, such as images, videos, audios, documents, etc. It provides high scalability, durability and reliability, enabling data to be stored and accessed in a distributed system.
[0048] 2. Privatization: Privatization deployment refers to deploying software, applications or services in a private environment, rather than using the infrastructure of a public cloud or a managed service provider. In privatization deployment, the organization owns and controls the entire deployment process, including aspects such as hardware, network, storage and software. Privatization deployment is usually used by organizations with high requirements for data security, compliance and control. Through privatization deployment, the organization can manage and maintain the infrastructure by itself, ensuring that data is stored and processed in a controlled environment. In addition, privatization deployment can also meet some special requirements, such as specific network configurations, performance requirements or integration requirements, etc.
[0049] According to one aspect of the embodiments of the present application, a log collection method is provided. Optionally, in this embodiment, the above log collection method may be applied to, for example, Figure 1 the hardware environment composed of a terminal 1402 and a server 1404 as shown. As Figure 1 shown, the server 1404 is connected to the terminal 1402 through a network, and can be used to provide services (such as game services, application services, etc.) for the terminal or the client installed on the terminal. A database can be set up on the server or independently of the server to provide data storage services for the server 1404.
[0050] The above network may include, but is not limited to, at least one of the following: wired network, wireless network. The above wired network may include, but is not limited to, at least one of the following: wide area network, metropolitan area network, local area network. The above wireless network may include, but is not limited to, at least one of the following: WIFI (Wireless Fidelity), Bluetooth. The terminal is not limited to a PC, mobile phone, tablet computer, etc.
[0051] The log collection method of the embodiments of the present application may be executed by the server, or may be executed by the terminal, or may also be jointly executed by the server and the terminal. Among them, when the terminal executes the log collection method of the embodiments of the present application, it may also be executed by the client installed on it.
[0052] Taking the execution of the log collection method in this embodiment by the server as an example, Figure 2 a log collection method provided by the embodiments of the present application includes the following steps:
[0053] Step S101, the target Filebeat obtains the target nginx log corresponding to the target node.
[0054] The log collection method in this embodiment may be applied to the scenario of log collection during the private deployment of big data, or may also be the scenario of log collection during the public deployment of big data. In the embodiments of the present application, the above log collection method is described by taking the log collection during the private deployment of big data as an example. For other types of scenarios, the above log collection method is equally applicable without contradiction.
[0055] Optionally, the target nginx log may be one or more log data obtained from the log file output by the service deployed on the target node.
[0056] Filebeat is a very lightweight log collection tool.
[0057] The target Filebeat can process the log file output by the service on the target node (for example, filtering, screening, etc.) to obtain the target nginx log.
[0058] Step S102: The target Filebeat sends the target nginx log to the transition interface.
[0059] Specifically, after obtaining the target nginx log, the target Filebeat can send the target nginx log to the transition interface by initiating a post request to the transition interface.
[0060] Step S103: The transition interface inputs the target nginx log into a relational database for storage.
[0061] Specifically, after obtaining the target nginx log, the transition interface can input the target nginx log into a relational database (such as MySQL object storage, Oracle database, Microsoft SQLServer database, etc.) by performing a write operation; and store the target nginx log in the relational database.
[0062] The method of this embodiment uses the target Filebeat to obtain the target nginx log corresponding to the target node, thus providing a lightweight log collection method, which can effectively reduce the workload of operation and maintenance deployment and the overhead of server resources; and after the target Filebeat sends the target nginx log to the transition interface, the transition interface inputs the target nginx log into a relational database for storage; thus, finally, the target nginx log can be read from the relational database, making the nginx log readable; in summary, the method of this embodiment achieves the technical effects of effectively reducing the workload of operation and maintenance deployment, effectively reducing the overhead of server resources, and making the nginx log readable; and thus solves the technical problem in the related art that the operation and maintenance deployment is difficult and the configuration is complex to achieve log collection.
[0063] As Figure 3 shown, as an alternative embodiment, for the method as described above, step S101 where the target Filebeat obtains the target nginx log corresponding to the target node includes the following steps:
[0064] Step S201: The target Filebeat obtains at least one original nginx log of the target node.
[0065] Specifically, in general, Filebeat is deployed on a single node and is used to collect nginx logs of each node. Therefore, the original nginx logs can be customized nginx logs generated by the target node according to the business types deployed on the target node. For example, when the business type is an upload business, the original nginx logs corresponding to the upload business include the bucket dimension, the unique identifier of the bucket, and the upload volume upload in the bucket dimension.
[0066] Step S202: The target Filebeat aggregates at least one original nginx log according to the target time granularity to obtain the aggregated target nginx log.
[0067] Specifically, after the target Filebeat obtains at least one original nginx log, the at least one original nginx log can be aggregated, and in this embodiment, the aggregation is performed according to the target time granularity to obtain the target nginx log.
[0068] The target time granularity can be determined by Filebeat or a module dedicated to determining the time granularity in the following manner: obtaining the write capacity of the relational database; determining the target frequency of initiating write operations on the relational database according to the write capacity; and determining the target time granularity according to the target frequency.
[0069] That is to say, according to the write capacity of the relational database, the corresponding target frequency is determined, and then the corresponding target time granularity is determined based on the target frequency.
[0070] The write capacity of the relational database can be the maximum load that the relational database can accept for write operations. And in general, for example, for a MySQL relational database, its write capacity is relatively poor. Therefore, if high-frequency write operations are performed on it, it will cause the MySQL relational database to be abnormal.
[0071] For example, the corresponding relationship between the write capacity and the frequency can be set in advance, and then the target frequency (that is, how often to perform a write operation on the relational database) is determined according to the corresponding relationship and the write capacity of the relational database. Finally, the target time granularity can be determined according to the target frequency.
[0072] In this embodiment, by aggregating at least one original nginx log according to the target time granularity, the aggregated target nginx log is obtained, so that the number of write operations initiated on the relational database can be reduced, and thus the load on the relational database can be reduced.
[0073] As an alternative embodiment, in the method as described above, in step S202, the target Filebeat aggregates at least one original nginx log according to the target time granularity to obtain the aggregated target nginx log, which includes the following steps:
[0074] Determine the preset fields that need to be stored in the relational database; the target Filebeat extracts data from each original nginx log according to the preset fields to obtain each specified nginx log corresponding to the preset fields; all the specific nginx logs in all the specified nginx logs are aggregated to obtain the aggregated target nginx log, where the specific nginx log is the specified nginx log whose corresponding original nginx log is generated within the target time period, and the target time period is a time period with a duration corresponding to the target time granularity.
[0075] That is to say, after the target Filebeat can extract the specified nginx log corresponding to the preset fields from the original nginx log according to the preset fields required by the relational database, it is necessary to determine all the specific nginx logs whose corresponding generation times are within the target time period among all the specified nginx logs, and finally aggregate the specific nginx logs to obtain the target nginx log.
[0076] For example, when the target time granularity is 1 minute, the moment when the target nginx log was last aggregated can be determined, and then the minute after the last moment is determined as the target time period. The specified nginx log obtained after processing the original nginx log whose generation time is within this target time period is determined as the specific nginx log; finally, all the specific nginx logs are aggregated to obtain the aggregated target nginx log corresponding to the target time period and with the target time granularity.
[0077] And generally, there is no identical log content between the previously generated target nginx log and the currently generated target nginx log.
[0078] As an alternative embodiment, in the method as described above, after the target nginx log is input into the relational database for storage through the transition interface, the method further includes:
[0079] The transition interface obtains the success information fed back by the relational database, where the success information is used to indicate that the target nginx log has been successfully stored in the relational database.
[0080] Further, after the transition interface obtains the success information fed back by the relational database, the transition interface can also send the success information to the target Filebeat.
[0081] Optionally, when the target Filebeat fails to obtain the success information corresponding to the target nginx log within a preset duration, it can send the target nginx log to the transition interface again, so that the transition interface writes the target nginx log into the relational database again.
[0082] As an optional embodiment, like the method of the previous item, the method includes the following steps:
[0083] The transition interface obtains all historical nginx logs within a historical time period, where the historical time period is a time period corresponding to the duration of the specified duration granularity;
[0084] The transition interface aggregates all historical nginx logs to obtain the second-aggregated nginx log of the specified duration granularity;
[0085] The transition interface inputs the second-aggregated nginx log into the relational database for storage.
[0086] Specifically, the historical time period can be a time period after the last second-aggregation operation performed by the transition interface (i.e., the operation performed before the historical time period and used to generate the previous second-aggregated nginx log) and with a duration corresponding to the duration of the specified duration granularity.
[0087] Specifically, when the time corresponding to the target nginx log is within this historical event segment, all historical nginx logs include the target nginx log.
[0088] Furthermore, the second-aggregation operation is an operation of aggregating the nginx logs that have been aggregated according to the target duration granularity indicated in steps S201 - S202 again.
[0089] The transition interface can cache all historical nginx logs in this historical time period, and after determining that the time since the last second-aggregation reaches the duration corresponding to the specified duration granularity, trigger the execution of the operation of aggregating all historical nginx logs to obtain the second-aggregated nginx log of the specified duration granularity. Finally, the transition interface inputs the second-aggregated nginx log into the relational database for storage.
[0090] Through the method of this embodiment, the transition interface can perform second-aggregation on the logs from Filebeat with different duration granularities, thus meeting the requirements of different services.
[0091] Such as Figure 4As shown below, an application example of any of the foregoing embodiments is provided:
[0092] Filebeat is used to split (i.e., extract data) and aggregate (at a few - minute granularity) the customized nginx logs; and send them to the transition interface. The transition node can store the aggregated logs (i.e., insert node minuterdata) of each node into a relational database. In addition, inside the transition interface, secondary aggregation of the productized target nginx logs aggregated by Filebeat is completed to obtain the secondary - aggregated nginx logs, and then they are stored in mysql (i.e., the relational database). By this method, directly compile Filebeat into a binary executable file, and at the same time configure the corresponding Filebeat.yml to achieve it. The deployment and operation and maintenance are simple. In addition, the status of the deployed Filebeat can be monitored by the way of opening ports, avoiding data loss due to the false death of a single - node Filebeat for some reasons.
[0093] It should be noted that, for the foregoing method embodiments, for the sake of simple description, they are all expressed as a series of action combinations. However, those skilled in the art should know that this application is not limited by the described action sequence, because according to this application, some steps can be in other sequences or performed simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to this application.
[0094] Through the description of the above - mentioned implementation manners, those skilled in the art can clearly understand that the method according to the above - mentioned embodiments can be implemented by means of software plus a necessary general - purpose hardware platform. Of course, it can also be implemented by hardware, but in many cases the former is a better implementation manner. Based on such an understanding, the technical solution of this application, in essence, or the part that makes a contribution to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM (Read - Only Memory), RAM (Random Access Memory), magnetic disk, optical disk), and includes several instructions to enable a terminal device (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods described in various embodiments of this application.
[0095] According to another aspect of the embodiments of this application, a log collection device for implementing the above - mentioned log collection method is also provided. Figure 5 is a structural block diagram of an optional log collection device according to the embodiments of this application, as Figure 5 shown, the device may include:
[0096] The Filebeat module 1 is used to obtain the target nginx log corresponding to the target node through the target Filebeat;
[0097] The Filebeat module 1 is also used to send the target nginx log to the transition interface through the target Filebeat;
[0098] The transition interface 2 inputs the target nginx log into the relational database 3 for storage.
[0099] It should be noted that the Filebeat module 1 in this embodiment can be used to execute the above steps S101 and S102, and the transition interface 2 in this embodiment can be used to execute the above step S103.
[0100] The device in this embodiment, in addition to including the above modules, may also include modules for executing any method in any embodiment of the foregoing log collection method.
[0101] It should be noted here that the examples and application scenarios implemented by the above modules and the corresponding steps are the same, but are not limited to the content disclosed in the above embodiments. It should be noted that the above modules, as part of the device, can run in the Figure 1 hardware environment as shown, and can be implemented by software or by hardware, where the hardware environment includes a network environment.
[0102] According to another aspect of the embodiments of the present application, an electronic device for implementing the above log collection method is further provided. The electronic device may be a server, a terminal, or a combination thereof.
[0103] According to another embodiment of the present application, an electronic device is further provided, including: as Figure 6 shown, the electronic device may include: a processor 1501, a communication interface 1502, a memory 1503, and a communication bus 1504. Among them, the processor 1501, the communication interface 1502, and the memory 1503 communicate with each other through the communication bus 1504.
[0104] The memory 1503 is used to store a computer program;
[0105] The processor 1501, when executing the program stored on the memory 1503, implements the following steps:
[0106] Step S101, the target Filebeat obtains the target nginx log corresponding to the target node.
[0107] Step S102, the target Filebeat sends the target nginx log to the transition interface.
[0108] In step S103, the transition interface inputs the target nginx log into a relational database for storage.
[0109] Optionally, in this embodiment, the above communication bus may be a PCI (Peripheral Component Interconnect) bus, an EISA (Extended Industry Standard Architecture) bus, or the like. The communication bus may be divided into an address bus, a data bus, a control bus, etc. For the sake of simplicity, only a thick line is used to represent it in the figure, but it does not mean that there is only one bus or one type of bus. The communication interface is used for communication between the above electronic device and other devices.
[0110] The memory may include a random access memory (RAM), and may also include a non-volatile memory (NVM), such as at least one disk memory. Optionally, the memory may also be at least one storage device located far from the aforementioned processor.
[0111] As an example, the above memory 1503 may but is not limited to include the Filebeat module 1 and the transition interface 2 in the above log collection device. In addition, it may also include but is not limited to other module units in the above log collection device, which will not be elaborated in this example.
[0112] The above processor may be a general-purpose processor, which may include but is not limited to: a CPU (Central Processing Unit), an NP (Network Processor), etc.; it may also be a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.
[0113] The embodiment of the present application also provides a computer-readable storage medium, where the storage medium includes a stored program, and when the program runs, it executes the method steps of the above method embodiment.
[0114] Optionally, in this embodiment, the above storage medium may include, but is not limited to: various media such as USB flash drives, ROMs, RAMs, mobile hard disks, magnetic disks, or optical discs that can store program codes.
[0115] The serial numbers of the embodiments of the present application are only for description and do not represent the superiority or inferiority of the embodiments.
[0116] If the integrated unit in the above embodiment is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in the above computer-readable storage medium. Based on such an understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art, or all or part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing one or more computer devices (which can be personal computers, servers, or network devices, etc.) to execute all or part of the steps of the methods described in various embodiments of the present application.
[0117] In the above embodiments of the present application, the descriptions of the various embodiments have their own emphases. For the parts not detailed in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.
[0118] In the several embodiments provided by the present application, it should be understood that the disclosed client can be implemented in other ways. Among them, the device embodiments described above are only illustrative. For example, the division of the units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed couplings or direct couplings or communication connections to each other can be through some interfaces. The indirect couplings or communication connections of the units or modules can be in an electrical or other form.
[0119] The units described as separate components may or may not be physically separated. The components displayed as units may or may not be physical units, that is, they can be located in one place or distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution provided in this embodiment.
[0120] In addition, the functional units in each embodiment of the present application can be integrated into one processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above integrated units can be implemented in the form of hardware or in the form of software functional units.
[0121] The above are only the preferred embodiments of the present application. It should be noted that for those of ordinary skill in the art, without departing from the principle of the present application, several improvements and refinements can be made, and these improvements and refinements should also be regarded as the protection scope of the present application.
Claims
1. A log collection method, characterized in that, including: The target Filebeat obtains the target nginx log corresponding to the target node; The target Filebeat sends the target nginx log to the transition interface; The transition interface inputs the target nginx log into a relational database for storage.
2. The method according to claim 1, characterized in that, The target Filebeat obtains the target nginx log corresponding to the target node, including: The target Filebeat obtains at least one original nginx log of the target node; The target Filebeat aggregates the at least one original nginx log according to the target duration granularity to obtain the aggregated target nginx log.
3. The method according to claim 2, wherein Before the target Filebeat aggregates the at least one original nginx log according to the target duration granularity to obtain the aggregated target nginx log, the method further includes: Obtaining the write capacity of the relational database; Determining the target frequency of initiating a write operation on the relational database according to the write capacity; Determining the target duration granularity according to the target frequency.
4. The method according to claim 2, characterized in that, The target Filebeat aggregates the at least one original nginx log according to the target duration granularity to obtain the aggregated target nginx log, including: Determining preset fields to be stored in the relational database; The target Filebeat extracts data in each original nginx log according to the preset fields to obtain each specified nginx log corresponding to the preset fields; The target Filebeat aggregates all specific nginx logs in all specified nginx logs to obtain the aggregated target nginx log, where the specific nginx log is the specified nginx log whose generation time of the corresponding original nginx log is within the target time period, and the target time period is a time period with a duration corresponding to the target duration granularity.
5. The method according to claim 2, wherein After the transition interface inputs the target nginx log into a relational database for storage, the method further includes: The transition interface obtains the success information fed back by the relational database, where the success information is used to indicate that the target nginx log is successfully stored in the relational database.
6. The method according to claim 5, wherein After the transition interface obtains the success information fed back by the relational database, the method further includes: The transition interface sends the success information to the target Filebeat.
7. The method according to any one of claims 1 to 6, characterized in that, The method further includes: The transition interface obtains all historical nginx logs within the historical time period, where the historical time period is a time period corresponding to the specified duration granularity; The transition interface aggregates all the historical nginx logs to obtain the second-aggregated nginx log of the specified duration granularity; The transition interface inputs the second-aggregated nginx log into the relational database for storage.
8. A log collection device, characterized in that, including: The Filebeat module is used to obtain the target nginx logs corresponding to the target nodes through the target Filebeat; The Filebeat module is further used to send the target nginx logs to the transition interface through the target Filebeat; The transition interface inputs the target nginx logs into a relational database for storage.
9. An electronic device, comprising a processor, a communication interface, a memory, and a communication bus, wherein, The processor, the communication interface, and the memory complete mutual communication through the communication bus, characterized in that The memory is used to store computer programs; The processor is used to execute the method according to any one of claims 1 to 7 by running the computer program stored on the memory.
10. A computer-readable storage medium, characterized in that, A computer program is stored in the storage medium, wherein the computer program is set to execute the method according to any one of claims 1 to 7 when running.