Method and system for realizing log collection based on front end and rear end

Abstract

The invention provides a log collection method and system based on a front end and a rear end, and the method comprises the steps: creating and configuring a theme of a kraft card for a virtual machine, configuring a collector and a processor at the same time, and enabling the collector and the processor to be respectively connected to the related configuration of the theme corresponding to the kraft card; appointing a catalog corresponding to the collector monitoring log file so as to collect log data under the catalog, and sending the log data to the kraft card; monitoring the log data stored in the kraft card by using the processor, formatting the log data monitored by the processor, and outputting the log data to a search engine; the log data of the search engine is subjected to distributed log storage, so that a back end obtains the log data after the distributed log storage, the log data obtained by the back end is previewed and analyzed through a front-end visual interface, and the requirement of most projects for collecting logs of a software system and a multi-service cluster link can be met.

Description

Technical Field

[0001] The present invention belongs to the field of log technology, and in particular relates to a log collection method and system based on a front end and a back end. Background Art

[0002] Log centers usually use ELK (Elasticsearch, Logstash, Kibana), but this combination has a big problem in terms of resource usage. Go is well-known for its high performance and concurrency, making it very suitable for developing distributed systems. Go's goroutine and channel mechanisms simplify concurrent programming and make it more efficient. In addition, Go also has the characteristics of fast compilation, efficient operation, rich standard libraries, and good cross-platform support, which makes it an ideal choice for developing distributed log collection systems.

[0003] For business software systems with huge log volumes, the log throughput can reach nearly 10,000 logs per second during peak hours, and the index building speed can reach thousands of logs per second during idle hours. However, existing log collection methods often cannot meet the needs of most projects and multi-service cluster link log collection. Summary of the invention

[0004] In order to solve the above technical problems, the present invention provides a log collection method and system based on the front-end and the back-end, which are used to solve the technical problems of the background technology.

[0005] On the one hand, the invention provides the following technical solution, a method for implementing log collection based on the front end and the back end, the method comprising:

[0006] Create and configure a Kafka topic for the virtual machine, configure a collector and a processor, and connect the collector and the processor to the relevant configuration of the topic corresponding to the Kafka respectively;

[0007] Specify the directory corresponding to the log file monitored by the collector so as to collect the log data in the directory and send the log data to the Kafka;

[0008] Using the processor to monitor the log data stored in the Kafka, formatting the log data monitored by the processor, and outputting the log data to a search engine;

[0009] The log data of the search engine is stored in a distributed log format so that the backend can obtain the log data after the distributed log format is stored, and the log data obtained by the backend can be previewed and analyzed through a frontend visual interface.

[0010] Compared with the prior art, the beneficial effects of the present application are: by distributing the log data of the search engine for log storage, so that the backend obtains the log data after the distributed log storage, and previewing and analyzing the steps of the log data obtained by the backend through the front-end visual interface, the log collection method has the advantages of many characteristics of the Go language, especially its high performance and resource saving characteristics, thereby meeting the needs of most projects and multi-service cluster link log collection.

[0011] Furthermore, after the step of configuring the collector and the processor, the method further includes:

[0012] A first configuration file is defined, and the first configuration file specifies a default output path of the log, so that the collector monitors and collects the specified log.

[0013] Furthermore, the method further comprises:

[0014] Configure the Kafka proxy address to ensure that the collector and the processor can communicate with Kafka.

[0015] Furthermore, the method further comprises:

[0016] Define a second configuration file, and customize the output path of the log corresponding to the second configuration file, wherein the second configuration file includes the API address for log reporting.

[0017] Furthermore, after the step of customizing the output path of the log corresponding to the second configuration file, the method further includes:

[0018] Based on the API address, the backend obtains the log data monitored by the collector, and performs distributed log storage on the log data obtained by the backend, while previewing and analyzing the log data obtained by the backend through a front-end visual interface.

[0019] Furthermore, the virtual machine includes a Tomcat application server and a Jar package of a Java application;

[0020] The number of the Kafkas corresponds to the number of the processors.

[0021] In a second aspect, the invention provides the following technical solution: a log collection system based on a front-end and a back-end, the system comprising:

[0022] A configuration module is used to create and configure a Kafka topic for the virtual machine, configure a collector and a processor, and connect the collector and the processor to the relevant configuration of the topic corresponding to the Kafka respectively;

[0023] A storage module, used to specify a directory corresponding to the log file monitored by the collector, so as to collect the log data in the directory and send the log data to the Kafka;

[0024] An engine module, configured to use the processor to monitor the log data stored in the Kafka, format the log data monitored by the processor, and output the log data to a search engine;

[0025] The analysis module is used to perform distributed log storage on the log data of the search engine so that the backend obtains the log data after the distributed log storage, and previews and analyzes the log data obtained by the backend through a front-end visual interface.

[0026] In a third aspect, the invention provides the following technical solution: a computer comprising a memory, a processor, and a computer program stored in the memory and executable on the processor; when the processor executes the computer program, the log collection method based on the front-end and the back-end is implemented as described above.

[0027] In a fourth aspect, the invention provides the following technical solution: a storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the log collection method based on the front end and the back end as described above. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0029] Figure 1 A flow chart of a log collection method based on the front end and the back end provided in the first embodiment of the present invention;

[0030] Figure 2 A structural block diagram of a log collection system based on a front-end and a back-end provided in a second embodiment of the present invention;

[0031] Figure 3 A schematic diagram of the hardware structure of a computer provided in the third embodiment of the present invention.

[0032] The embodiments of the present invention will be further described below with reference to the accompanying drawings. DETAILED DESCRIPTION

[0033] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout are the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the embodiments of the present invention, and should not be construed as limiting the present invention.

[0034] In the description of the embodiments of the present invention, it should be understood that the terms "length", "width", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside" and "outside" etc., indicating orientations or positional relationships, are based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the embodiments of the present invention and simplifying the description, and do not indicate or imply that the referred device or element must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as a limitation on the present invention.

[0035] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of the embodiments of the present invention, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0036] Embodiment 1

[0037] In the first embodiment of the present invention, see Figure 1 , a log collection method based on the front end and the back end, comprising the following steps S01 to S04:

[0038] S01, creating and configuring a Kafka topic for the virtual machine, configuring a collector and a processor, and connecting the collector and the processor to the relevant configurations of the topic corresponding to the Kafka respectively;

[0039] In this embodiment, a Kafka topic is first created for a virtual machine (including a Tomcat application server and a running Java application Jar package). Next, Filebeat (log collector) and Logstash (log processor) are configured to ensure that they can correctly connect to the topic corresponding to Kafka to achieve the transmission and processing of log data.

[0040] It is worth noting that the number of Kafka topics corresponds to the number of Logstash instances. This design ensures that each Logstash instance can effectively listen to and process log data from the corresponding Kafka topic, thereby improving the efficiency and reliability of log processing.

[0041] S02, specifying the collector to monitor the directory corresponding to the log file, so as to collect the log data in the directory and send the log data to the Kafka;

[0042] Specifically, the method further includes:

[0043] Configure the Kafka proxy address to ensure that the collector and the processor can communicate with Kafka.

[0044] In this embodiment, first, a log directory is specified for Filebeat, that is, the log file storage location that Filebeat needs to monitor is clarified so that it can collect the log files in these directories in real time. Secondly, in order to ensure that Logstash can smoothly send the collected log data to the Kafka cluster, the broker address (also known as the proxy address) of the Kafka cluster is configured for the ELK system. This step is crucial, as it ensures that log data can be efficiently and reliably transmitted between Logstash and the Kafka cluster.

[0045] S03, using the processor to monitor the log data stored in the Kafka, formatting the log data monitored by the processor, and outputting the log data to a search engine;

[0046] In this embodiment, the search engine is Elasticsearch (ES).

[0047] S04, performing distributed log storage on the log data of the search engine so that the backend obtains the log data after the distributed log storage, and previews and analyzes the log data obtained by the backend through a front-end visual interface.

[0048] Specifically, after the step of configuring the collector and the processor, the method further includes:

[0049] A first configuration file is defined, and the first configuration file specifies a default output path of the log, so that the collector monitors and collects the specified log.

[0050] In this embodiment, a general logback log configuration file (first configuration file) is used to manage log output. By default, the log will be output to the directory / data / applog / ${appName}, where ${appName} is a placeholder representing the name of the application. Users can also customize the output path of the log through the configuration item logging.path.log in the application configuration file. In addition, in order to facilitate users to view and analyze log data, a visual interface based on Vue (front-end framework) is also provided. Through this interface, users can intuitively preview the log content, so as to better understand the operating status and log information of the system.

[0051] Specifically, the method further includes:

[0052] Define a second configuration file, and customize the output path of the log corresponding to the second configuration file, wherein the second configuration file includes the API address for log reporting.

[0053] More specifically, after the step of customizing the output path of the log corresponding to the second configuration file, the method further includes:

[0054] Based on the API address, the backend obtains the log data monitored by the collector, and performs distributed log storage on the log data obtained by the backend, while previewing and analyzing the log data obtained by the backend through a front-end visual interface.

[0055] In this embodiment, the specific description of the AIP configuration process for actively reporting service logs is as follows:

[0056] (1) Use the common logback log configuration file (second configuration file): Logs are output to the / data / applog / ${appName} directory by default, where ${appName} is a dynamic variable representing the name of the current application. Users can customize the log output path, dependencies, and API addresses in the application configuration file.

[0057] (2) Preview log content through the Vue visual interface.

[0058] In the specific implementation, the Docker base image is used. The base image contains SkyWalking, Filebeat, and the default startup script start.sh. The start.sh script is responsible for starting Filebeat and starting the application by executing app.jar. The FROM instruction in the Dockerfile specifies the source of the base image, that is, xx.xx.xx.xx:20404 / jichujx-f42ea0 / filebeat-7.12.0:2024010205.

[0059] Dockerfile instruction: FROM is an instruction in Dockerfile used to specify the base image.

[0060] Basic image: xx.xx.xx.xx:20404 / jichujx-f42ea0 / filebeat-7.12.0:2024010205 represents the full name of the Docker basic image, including the warehouse address, image name and tag (or version number). The private warehouse address is used here.

[0061] SkyWalking: SkyWalking is an application performance monitoring system used to track and monitor the performance and health status of services. Including SkyWalking in the Docker image means that you want to monitor the application in a containerized environment.

[0062] Filebeat: Filebeat is part of the ElasticStack and is used to collect and forward log data. Including Filebeat in the base image indicates that you want to automatically collect logs in the container and send them to the specified log processing system.

[0063] In summary, a log collection method based on the front-end and back-end has the following effects:

[0064] 1. Implemented using Golang, giving full play to Golang's advantages such as high performance and resource saving.

[0065] 2. The log throughput reaches nearly 10,000 records per second, and the indexing speed can reach thousands of records per second during idle time, meeting the needs of most projects.

[0066] 3. It supports multi-keyword full-text search function and has a built-in Chinese word segmenter, providing millisecond-level query response speed and smooth and imperceptible operation.

[0067] 4. A built-in log query management interface based on Vue is provided, the interface is simple and elegant, and the operation habits are in line with user intuition.

[0068] 5. Supports switch control through personalized environment variables, and provides automatic maintenance function of log warehouse, which is more flexible and convenient to use.

[0069] 6. Provide Docker images and support containerized deployment, which greatly facilitates deployment and migration.

[0070] 7. Provide Java project log collection package to achieve closed-loop support for Java projects.

[0071] 8. Provide Golang project log collection package to achieve closed-loop support for Golang projects.

[0072] 9. Support multi-service cluster mode deployment to provide high availability of services and data redundancy guarantees.

[0073] 10. The coupling between systems is extremely low, and various systems can be easily connected, which is applicable to both large state-owned enterprise projects and small projects.

[0074] Embodiment 2

[0075] like Figure 2 As shown, in a second embodiment of the present invention, a log collection system based on a front-end and a back-end is provided, and the system includes:

[0076] The configuration module 10 is used to create and configure a Kafka topic for the virtual machine, configure a collector and a processor, and connect the collector and the processor to the relevant configuration of the topic corresponding to the Kafka respectively;

[0077] The storage module 20 is used to specify the directory corresponding to the log file monitored by the collector so as to collect the log data in the directory and send the log data to the Kafka;

[0078] The engine module 30 is used to use the processor to monitor the log data stored in the Kafka, format the log data monitored by the processor, and output it to the search engine;

[0079] The analysis module 40 is used to perform distributed log storage on the log data of the search engine so that the backend obtains the log data after the distributed log storage, and previews and analyzes the log data obtained by the backend through a front-end visual interface.

[0080] The log collection system based on the front-end and back-end provided in the embodiment of the present invention has the same implementation principle and technical effects as those in the aforementioned method embodiment. For the sake of brief description, for matters not mentioned in the system embodiment, reference can be made to the corresponding contents in the aforementioned method embodiment.

[0081] Embodiment 3

[0082] like Figure 3 As shown, in the third embodiment of the present invention, the embodiment of the present invention provides the following technical solution, a computer, including a memory 202, a processor 201, and a computer program stored in the memory 202 and executable on the processor 201, and the processor 201 implements the log collection method based on the front-end and the back-end as described above when executing the computer program.

[0083] Specifically, the processor 201 may include a central processing unit (CPU), or an application specific integrated circuit (ASIC), or may be configured to implement one or more integrated circuits of the embodiments of the present application.

[0084] Among them, the memory 202 may include a large-capacity memory for resources or instructions. By way of example and not limitation, the memory 202 may include a hard disk drive (HDD), a floppy disk drive, a solid-state drive (SSD), a flash memory, an optical disk, a magneto-optical disk, a magnetic tape, or a universal serial bus (USB) drive, or a combination of two or more of these. In appropriate cases, the memory 202 may include a removable or non-removable (or fixed) medium. In appropriate cases, the memory 202 may be inside or outside the resource processing device. In a specific embodiment, the memory 202 is a non-volatile memory. In a specific embodiment, the memory 202 includes a read-only memory (ROM) and a random access memory (RAM). Where appropriate, the ROM may be a mask-programmed ROM, a programmable ROM (Programmable Read-Only Memory, PROM for short), an erasable PROM (Erasable Programmable Read-Only Memory, EPROM for short), an electrically erasable PROM (Electrically Erasable Programmable Read-Only Memory, EEPROM for short), an electrically alterable ROM (Electrically Alterable Read-Only Memory, EAROM for short) or a flash memory (FLASH) or a combination of two or more of these. Under appropriate circumstances, the RAM can be a static random access memory (Static Random-Access Memory, abbreviated as SRAM) or a dynamic random access memory (Dynamic Random Access Memory, abbreviated as DRAM), wherein the DRAM can be a fast page mode dynamic random access memory (Fast Page Mode Dynamic Random Access Memory, abbreviated as FPMDRAM), an extended resource output dynamic random access memory (Extended Date Out Dynamic Random Access Memory, abbreviated as EDODRAM), a synchronous dynamic random access memory (Synchronous Dynamic Random-Access Memory, abbreviated as SDRAM), etc.

[0085] The memory 202 may be used to store or cache various resource files required for processing and / or communication, as well as possible computer program instructions executed by the processor 201 .

[0086] The processor 201 implements the log collection method based on the front end and the back end by reading and executing the computer program instructions stored in the memory 202 .

[0087] In some embodiments, the computer may further include a communication interface 203 and a bus 200. Figure 3 As shown, the processor 201, the memory 202, and the communication interface 203 are connected via a bus 200 and communicate with each other.

[0088] The communication interface 203 is used to implement communication between the modules, devices, units and / or devices in the embodiment of the present application. The communication interface 203 can also implement resource communication with other components such as: external devices, image / resource acquisition devices, resource libraries, external storage, and image / resource processing workstations.

[0089] The bus 200 includes hardware, software, or both, and couples the components of the computer to each other. The bus 200 includes, but is not limited to, at least one of the following: a resource bus (DataBus), an address bus (AddressBus), a control bus (ControlBus), an expansion bus (ExpansionBus), and a local bus (LocalBus). By way of example and not limitation, the bus 200 may include an Accelerated Graphics Port (AGP) or other graphics bus, an Extended Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB) bus, a HyperTransport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an InfiniBand interconnect, a Low Pin Count (LPC) bus, or a 100,000 Pin Count bus. Bus, memory bus, Micro Channel Architecture (MCA) bus, Peripheral Component Interconnect (PCI) bus, PCI-Express (PCI-X) bus, Serial Advanced Technology Attachment (SATA) bus, Video Electronics Standards Association Local Bus (VLB) bus or other suitable bus or a combination of two or more of these. Where appropriate, bus 200 may include one or more buses. Although the embodiments of the present application describe and illustrate specific buses, the present application considers any suitable bus or interconnect.

[0090] Embodiment 4

[0091] In the fourth embodiment of the present invention, in combination with the above-mentioned log collection method based on the front-end and the back-end, the embodiment of the present invention provides the following technical solution, a storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the above-mentioned log collection method based on the front-end and the back-end is implemented.

[0092] Those skilled in the art will appreciate that the logic and / or steps described in the flowchart or otherwise herein, for example, can be considered as a sequenced resource list of executable instructions for implementing logical functions, and can be specifically implemented in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purposes of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses.

[0093] More specific examples of readable media (non-exhaustive list) include the following: an electrical connection with one or more wirings (electronic device), a portable computer disk case (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and editable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be a paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, deciphering or processing in another suitable manner as necessary, and then stored in a computer memory.

[0094] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, a plurality of steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or a combination thereof: a discrete logic circuit having a logic gate circuit for implementing a logic function for a resource signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0095] The technical features of the above-described embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above-described embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0096] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A log collection method based on front-end and back-end, characterized in that: The method comprises: Create and configure a Kafka topic for the virtual machine, configure a collector and a processor, and connect the collector and the processor to the relevant configuration of the topic corresponding to the Kafka respectively; Specify the directory corresponding to the log file monitored by the collector so as to collect the log data in the directory and send the log data to the Kafka; Using the processor to monitor the log data stored in the Kafka, formatting the log data monitored by the processor, and outputting the log data to a search engine; The log data of the search engine is stored in a distributed log format so that the backend can obtain the log data after the distributed log format is stored, and the log data obtained by the backend can be previewed and analyzed through a frontend visual interface.

2. The log collection method based on the front end and the back end according to claim 1 is characterized in that: After the step of configuring the collector and the processor, the method further comprises: A first configuration file is defined, and the first configuration file specifies a default output path of the log, so that the collector monitors and collects the specified log.

3. The log collection method based on the front-end and the back-end according to claim 1 is characterized in that: The method further comprises: Configure the Kafka proxy address to ensure that the collector and the processor can communicate with Kafka.

4. The log collection method based on the front-end and the back-end according to claim 1 is characterized in that: The method further comprises: Define a second configuration file, and customize the output path of the log corresponding to the second configuration file, wherein the second configuration file includes the API address for log reporting.

5. The log collection method based on the front-end and the back-end according to claim 4 is characterized in that: After the step of customizing the output path of the log corresponding to the second configuration file, the method further includes: Based on the API address, the backend obtains the log data monitored by the collector, and performs distributed log storage on the log data obtained by the backend, while previewing and analyzing the log data obtained by the backend through a front-end visual interface.

6. The log collection method based on the front-end and the back-end according to claim 1 is characterized in that: The virtual machine includes a Tomcat application server and a Jar package of a Java application; The number of the Kafkas corresponds to the number of the processors.

7. A log collection system based on front-end and back-end, characterized in that: The system comprises: A configuration module is used to create and configure a Kafka topic for the virtual machine, configure a collector and a processor, and connect the collector and the processor to the relevant configuration of the topic corresponding to the Kafka respectively; A storage module, used to specify a directory corresponding to the log file monitored by the collector, so as to collect the log data in the directory and send the log data to the Kafka; An engine module, configured to use the processor to monitor the log data stored in the Kafka, format the log data monitored by the processor, and output the log data to a search engine; The analysis module is used to perform distributed log storage on the log data of the search engine so that the backend obtains the log data after the distributed log storage, and previews and analyzes the log data obtained by the backend through a front-end visual interface.

8. A computer comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the log collection method based on the front end and the back end is implemented as described in any one of claims 1 to 6.

9. A storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by the processor, the log collection method based on the front end and the back end is implemented as described in any one of claims 1 to 6.

Images