A log viewing method, apparatus, device and storage medium

Abstract

The application provides a log viewing method, device, equipment and storage medium. The application comprises the following steps: obtaining a log of a Spark driver program end, the log comprising standard output data, standard error output data and other data, wherein the Spark driver program end is a Driver program of Spark in a Cluster mode; sending the standard output data in the log to Kafka through a first thread and a first byte stream in a console; sending the standard error output data in the log to Kafka through a second thread and a second byte stream in the console; sending the other data in the log to Kafka through a Java log framework; receiving a log collection instruction; and determining and displaying target display log content from Kafka according to the log collection instruction. The method of the application can collect logs and display target display log content in the logs on a page when the Driver program of Spark is in the Cluster mode, so that the logs can be intuitively and conveniently viewed.

Description

Technical Field

[0001] This application relates to the field of information technology, and in particular to a log viewing method, apparatus, device, and storage medium. Background Technology

[0002] Apache Spark is a fast and general-purpose computing engine designed for large-scale data processing. It has now formed a rapidly developing and widely used ecosystem. When a Spark job runs, a driver process is required to communicate with the Spark cluster, responsible for submitting tasks, retrieving results, and controlling task execution. Cloud deployment is a common deployment model for current web applications. In this model, the application's available memory resources are very limited. However, the Spark driver generally requires a significant amount of memory because it needs to retrieve execution results and temporarily store data required by the Spark cluster. In this scenario, it is clearly unsuitable for the virtual machine or container deploying the web application to provide driver resources. Therefore, web applications are more suitable for submitting Spark jobs in cluster mode.

[0003] In cluster mode, viewing Spark logs is a challenge for users. In existing technologies, users need to access the Spark UI or log in to the host where the Spark driver is located to view the logs.

[0004] Therefore, currently, when deploying web applications in the cloud and submitting Spark jobs in cluster mode, viewing logs is inconvenient and not intuitive. Summary of the Invention

[0005] This application provides a log viewing method, apparatus, device, and storage medium to address the inconvenience and lack of intuitiveness in viewing logs when web applications are deployed in the cloud and Spark jobs are submitted in cluster mode.

[0006] Firstly, this application provides a method for viewing logs, including:

[0007] Retrieve logs from the Spark driver, including standard output data, standard error output data, and other data. The Spark driver is the Spark Driver program in Cluster mode.

[0008] Send the standard output data from the log to Kafka via the first thread and the first byte stream in the console;

[0009] The standard error output data in the log is sent to Kafka via the second thread and the second byte stream in the console;

[0010] Send other data from the logs to Kafka using the Java logging framework;

[0011] Receive log collection instructions;

[0012] Based on the log collection instructions, determine and display the target log content from Kafka.

[0013] In this application, the standard output data in the log is sent to Kafka via the first thread and the first byte stream in the console, including:

[0014] Create a new first thread and a first byte stream, where the first byte stream is used to relocate standard output data to the standard output byte stream;

[0015] When the first thread reads the first byte stream status and finds it to be non-empty, the first thread sends the first byte stream to Kafka until the first thread reads the first byte stream status and finds it to be empty.

[0016] In this application, standard error data from the logs is sent to Kafka via a second thread and a second byte stream in the console, including:

[0017] Create a second thread and a second byte stream, where the second byte stream is used to relocate the standard error output to the standard error output byte stream;

[0018] When the second thread reads the status of the second byte stream and finds it to be non-empty, the second thread sends the second byte stream to Kafka until the second thread reads the status of the second byte stream and finds it to be empty.

[0019] In this application, other data from the logs is sent to Kafka via a Java logging framework, including:

[0020] Determine the Kafka send function and Java logging framework;

[0021] Based on Kafka's send function and the Java logging framework, other data in the logs are sent to Kafka via the Java logging framework.

[0022] In this application, before receiving the log collection instruction, the method further includes:

[0023] Based on the preset framework, logs in Kafka are parsed into target structure logs, where the target display log content can be obtained through collection commands.

[0024] In this application, after parsing the logs in Kafka into the target structure logs according to the preset framework, the method further includes:

[0025] The target structure log is stored in the database so that the page can retrieve the target structure log from the database in response to user actions.

[0026] In this application, the target log content is determined and displayed from Kafka according to the log collection instructions, including:

[0027] Based on the log collection instructions, determine the log filtering conditions and log output format;

[0028] Determine the initial log content from Kafka based on the log filtering criteria;

[0029] Adjust the initial log content according to the log output format to obtain the final log content;

[0030] Display log content.

[0031] In this application, the log content is displayed, including:

[0032] The log content is sent to the server, which then sends the log content to the websocket on the page for display.

[0033] Secondly, this application provides a log viewing device, comprising:

[0034] The acquisition module is used to acquire logs from the Spark driver. The logs include standard output data, standard error output data, and other data. The Spark driver is the Spark Driver program in Cluster mode.

[0035] The first sending module is used to send the standard output data in the log to Kafka through the first thread and the first byte stream in the console;

[0036] The second sending module is used to send the standard error output data in the log to Kafka through the second thread and the second byte stream in the console;

[0037] The third sending module is used to send other data in the log to Kafka via the Java logging framework;

[0038] The receiving module is used to receive log collection commands;

[0039] The determination module is used to determine and display the target log content from Kafka based on the log collection instructions.

[0040] Thirdly, this application provides an electronic device, including: a processor, and a memory communicatively connected to the processor;

[0041] The memory stores instructions that the computer executes;

[0042] The processor executes computer execution instructions stored in memory to implement the method in this application.

[0043] Fourthly, this application provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the method of this application.

[0044] This application provides a log viewing method, apparatus, device, and storage medium. It acquires logs from a Spark driver, including standard output data, standard error output data, and other data. The Spark driver is a Spark Driver program in Cluster mode. The method involves sending the standard output data from the logs to Kafka via a first thread and a first byte stream in the console; sending the standard error output data from the logs to Kafka via a second thread and a second byte stream in the console; sending other data from the logs to Kafka via a Java logging framework; receiving log collection instructions; and determining and displaying the target log content from Kafka according to the log collection instructions. By collecting logs when the Spark driver is a Spark Driver program in Cluster mode, sending all logs to Kafka, obtaining the target log content from Kafka through collection instructions, and then displaying the target log content on a page, log data can be viewed intuitively and conveniently. Attached Figure Description

[0045] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0046] Figure 1 A flowchart illustrating a log viewing method provided in an embodiment of this application;

[0047] Figure 2 A flowchart illustrating another log viewing method provided in an embodiment of this application;

[0048] Figure 3 This is a schematic diagram of the structure of the log viewing device provided in the embodiments of this application;

[0049] Figure 4 This is a schematic diagram of the electronic device structure provided in an embodiment of this application.

[0050] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0051] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0052] Currently, viewing Spark logs in cluster mode presents a challenge for users. Existing technologies, on the one hand, require users to access the Spark UI or log in to the host where the Spark driver resides to view logs, making it inconvenient. On the other hand, web applications typically do not allow users to directly access the Spark cluster page or log in to the host, thus preventing them from viewing job execution logs. Furthermore, Spark jobs require multiple hosts to handle different parts, resulting in Spark tasks being relatively dispersed. In short, viewing logs is inconvenient and unintuitive.

[0053] This application provides a log viewing method, apparatus, device, and storage medium. It acquires logs from a Spark driver, including standard output data, standard error output data, and other data. The Spark driver is a Spark Driver program in Cluster mode. The method involves sending the standard output data from the logs to Kafka via a first thread and a first byte stream in the console; sending the standard error output data from the logs to Kafka via a second thread and a second byte stream in the console; sending other data from the logs to Kafka via a Java logging framework; receiving log collection instructions; and determining and displaying the target log content from Kafka according to the log collection instructions. Standard output and standard error output data were originally only printed to the console. Using a first thread and a first byte stream, standard output data from the console can be sent to Kafka without affecting its printing. Similarly, using a second thread and a second byte stream, standard error output data can be sent to Kafka without affecting its printing. Simultaneously, other log data is sent to Kafka via the Java logging framework, allowing Kafka to retrieve the logs. Spark jobs require multiple hosts to handle different parts, resulting in scattered tasks. Therefore, a relay system like Kafka is needed to integrate logs from all different hosts. Since logs contain a lot of information, it can affect users' ability to view important or closely monitored data. Therefore, log collection is necessary to identify and display target log content from Kafka. This allows for convenient and intuitive log viewing in this embodiment.

[0054] The log viewing method provided in this embodiment can be executed by a server. The server can be any server. This embodiment does not impose any particular restriction on the implementation method of the execution subject, as long as the execution subject can obtain the logs from the Spark driver, including standard output data, standard error output data, and other data. The Spark driver is a Spark Driver program in Cluster mode. The method involves sending the standard output data from the logs to Kafka via a first thread and a first byte stream in the console; sending the standard error output data from the logs to Kafka via a second thread and a second byte stream in the console; sending other data from the logs to Kafka via a Java logging framework; receiving log collection instructions; and determining and displaying the target log content from Kafka according to the log collection instructions.

[0055] Kafka is a high-throughput distributed publish-subscribe messaging system that can handle all action streams from consumers on a website. These actions (web browsing, searching, and other user actions) are a key element of many social functions on the modern web. This data is often handled through log processing and log aggregation due to throughput requirements. It's a viable solution for log data and offline analytics systems like Hadoop, but with the constraint of requiring real-time processing. Kafka aims to unify online and offline message processing through Hadoop's parallel loading mechanism, and also to provide real-time messaging through clustering.

[0056] Spark is a distributed computing platform, a computing framework written in Scala, and a fast, general-purpose, and scalable big data analysis engine based on memory.

[0057] A log is a record of a system's or software's completed processes for future reference. It doesn't have a fixed format; it's usually a text file that can be opened with Notepad to view its contents. However, it could be in other formats, in which case opening it directly would result in gibberish. The purpose of most logs can be understood from their filenames, such as uninstall.log or error.log. The former is typically generated during software installation and provided to the uninstaller, while the latter usually records errors and other information that occurred during software operation.

[0058] Figure 1 This is a flowchart illustrating a log viewing method provided in an embodiment of this application. Figure 1 As shown, the execution subject of this method can be a server or other servers; this embodiment does not impose any special restrictions here. Figure 1 As shown, the method includes:

[0059] S101. Obtain the logs from the Spark driver. The logs include standard output data, standard error output data, and other data. The Spark driver is the Spark Driver program in Cluster mode.

[0060] Standard output is formally called stdout, and standard error is formally called stderr. The system outputs to the user through two channels: general information is output from stdout, and some error information is output from stderr. In Java, these correspond to System.out and System.err, respectively.

[0061] Other data refers to the data in the Spark driver's logs other than standard output and standard error output.

[0062] S102. Send the standard output data in the log to Kafka through the first thread and the first byte stream in the console.

[0063] In this embodiment of the application, sending the standard output data in the log to Kafka via the first thread and the first byte stream in the console includes:

[0064] Create a new first thread and a first byte stream, where the first byte stream is used to relocate standard output data to the standard output byte stream;

[0065] When the first thread reads the first byte stream status and finds it to be non-empty, the first thread sends the first byte stream to Kafka until the first thread reads the first byte stream status and finds it to be empty.

[0066] The first byte stream is used to relocate the standard output data to the standard output byte stream, so that the first thread can send the first byte stream to Kafka. When the first thread reads that the status of the first byte stream is not empty, it sends the first byte stream to Kafka. Through the first byte stream and the first thread, Kafka finally obtains the standard output data in the log.

[0067] When a log is generated and contains standard output data, the first byte stream is non-empty.

[0068] When executing a Spark job, multiple hosts are needed to run it, resulting in logs on each host. Because these logs are scattered and difficult to view centrally, each host needs to send its logs to Kafka as an intermediary. This makes it easier for subsequent pages to retrieve and display the logs.

[0069] S103. Send the standard error output data in the log to Kafka through the second thread and the second byte stream in the console.

[0070] In this embodiment of the application, sending standard error data from the log to Kafka via a second thread and a second byte stream in the console includes:

[0071] Create a second thread and a second byte stream, where the second byte stream is used to relocate the standard error output to the standard error output byte stream;

[0072] When the second thread reads the status of the second byte stream and finds it to be non-empty, the second thread sends the second byte stream to Kafka until the second thread reads the status of the second byte stream and finds it to be empty.

[0073] The second byte stream is used to relocate the standard output data to the standard output byte stream, so that the second thread can send the second byte stream to Kafka. If the second thread reads that the second byte stream is not empty, it sends the second byte stream to Kafka. Through the second byte stream and the second thread, Kafka finally obtains the standard output data in the log.

[0074] When a log is generated and contains standard output data, the second byte stream is non-empty.

[0075] Standard output data and standard error output data are sent to Kafka concurrently, so a first thread is needed to send standard output data and a second thread is needed to send standard error output data.

[0076] S104. Send other data in the log to Kafka via the Java logging framework.

[0077] In this embodiment of the application, sending other data from the log to Kafka via the Java logging framework includes:

[0078] Determine the Kafka send function and Java logging framework;

[0079] Based on Kafka's send function and the Java logging framework, other data in the logs are sent to Kafka via the Java logging framework.

[0080] Among them, the Java logging framework can be log4j (Logging for Java, a logging library for the Java programming language).

[0081] By overriding the AppenderSkeleton class in log4j, implementing its append method, and calling the send function provided by Kafka, other logs can be sent to Kafka.

[0082] S105, Receive log collection command.

[0083] In this embodiment of the application, before receiving the log collection instruction, the method further includes:

[0084] Based on the preset framework, logs in Kafka are parsed into target structure logs, and the target display log content in the target structure logs can be obtained through collection commands.

[0085] The preset framework can be the Flink framework, and the target structure can be a Flink table. Here, the logs in Kafka can be mapped to Flink stream tables through the Flink Connector so that subsequent collection commands can be used for collection.

[0086] In this embodiment of the application, after parsing the logs in Kafka into logs of the target structure according to a preset framework, the method further includes:

[0087] The target structure log is stored in the database so that the page can retrieve the target structure log from the database in response to user actions.

[0088] The page can only display logs generated at the current time, which are time-sensitive; if you want to be able to search for historical logs, you need to store the logs in the database, and then the page will retrieve the logs from the database and display them on the page.

[0089] S106. Based on the log collection instructions, determine and display the target log content from Kafka.

[0090] The log includes a timestamp, application project, module, task tag, host, log source, thread ID, log level, and log content. The timestamp is the time the log was sent. The application project is a tag that identifies which project generated the log; this can be defined by the user program. The module is also a tag that identifies the module that generated the log; this can also be defined by the user program. The task tag is the tag for the Spark task. The host is the hostname of the host that generated the log. The log source is the location where the log was generated, usually the Java class name. The thread ID is incorrect and can be the thread name. For example, a log could be: 2021.01.01 01:01:01.001 (timestamp), admin (application project), user (module), job1 (task tag), pc-001 (host), cn.chinaunicom.jh.spark.JobSubmit (log source), pool-1-thread-7 (thread name), info (log level), and "Task executed successfully!" (log content).

[0091] In this embodiment of the application, determining and displaying the target log content from Kafka according to the log collection instruction includes:

[0092] Based on the log collection instructions, determine the log filtering conditions and log output format;

[0093] Determine the initial log content from Kafka based on the log filtering criteria;

[0094] Adjust the initial log content according to the log output format to obtain the final log content;

[0095] Display log content.

[0096] The log collection command can be an SQL command used to filter the logs that the user is interested in. The filtering criteria can be any one of the following: timestamp, application project, module, task tag, host, log source, thread ID, log level, or log content. The filtering condition is the log portion the user needs. For example, to filter logs with the DEBUG level and print them by timestamp and log content, the collection command would be: `select timestamp, message from log where level = 'DEBUG'`. The target log content displayed would be: 2023-11-14 17:00:00.001, which is one log entry. To filter logs from application project 'xxx' and print them by timestamp, level, and log content, the collection command would be: `select timestamp, level, message from log where application = 'xxx'`. The target log content displayed would be: 2023-11-14 17:00:00.001, INFO, which is one log entry.

[0097] In this embodiment of the application, the log content is displayed, including:

[0098] The log content is sent to the server, which then sends the log content to the websocket on the page for display.

[0099] To ensure the page can display log content in real time, a WebSocket connection is established between the page and the server. The server then sends the read content to the page in real time via the WebSocket for display.

[0100] The log viewing method provided in this application embodiment enables centralized and real-time viewing of logs, reduces the complexity of the log viewing process, and allows for intuitive and quick log viewing.

[0101] Figure 2 This is a flowchart illustrating another log viewing method provided in an embodiment of this application. Figure 2 As shown, the execution subject of this method can be a server or other servers; this embodiment does not impose any special restrictions here. Figure 2 As shown, the method includes:

[0102] The S201 Spark driver collects logs and sends them to the server. The server then sends the logs to the console and log4j. The logs include: timestamp, application project, module, task tag, host, log source, thread ID, log level, and log content. Logs sent to the console include standard output and standard error output; all other logs are input to log4j, referring to non-standard output and standard error output.

[0103] In Spark driver S202, for logs output to the console, the driver relocates them to a byte stream. If the byte stream is not empty, the logs are sent to Kafka and printed to the console. The method for relocating logs output to the console and printing them to the console is as follows: First, a first object and a second object are created. The first object stores standard output and then prints it to the console, while the second object stores standard error output and then prints it to the console. The logs consist of standard output and standard error. Second, a first byte stream and a second byte stream are created. The first byte stream is used to relocate standard output to the first byte stream, and the second byte stream is used to relocate standard error output to the second byte stream. Third, a first thread and a second thread are created. The first thread checks if the first byte stream is not empty before sending it to Kafka, and the second thread checks if the second byte stream is not empty before sending it to Kafka. Once the log reading is complete, the byte stream data from both the first and second objects are printed.

[0104] S203 and the Spark driver forward logs output to log4j via the send function in Kafka and log4j, and then print the logs to the console.

[0105] S204, Kafka sends logs to the server.

[0106] S205. The server parses the logs into the target data structure.

[0107] The target data structure can be in Flink format; the server can use the Flink framework to parse the logs into Flink tabular logs.

[0108] S206. The server stores the parsed target data structure logs in the database for easy access at any time.

[0109] S207. The server responds to the user's filtering instruction and filters out the target data in the parsed log.

[0110] The filtering command can use SQL statements and can output data in a specific format; the target data can be timestamp, application project, module, task tag, host, log source, thread ID, log level, or log content.

[0111] S208. The server sends the target data to the page for display.

[0112] The server can send the target data to the page via WebSocket.

[0113] The alternative log viewing method provided in this application embodiment enables centralized and real-time log viewing, reduces the complexity of the log viewing process, collects Spark job logs into Kafka, and then saves them to a database or server and provides a query page to ensure that web users can easily view Spark logs.

[0114] Figure 3 This is a structural example diagram of the log viewing device provided in an embodiment of this application. Figure 3 As shown, the log viewing device 30 includes: an acquisition module 301, a first sending module 302, a second sending module 303, a third sending module 304, a receiving module 305, and a determining module 306. Wherein:

[0115] The acquisition module 301 is used to acquire logs from the Spark driver, including standard output data, standard error output data, and other data. The Spark driver is the Spark Driver program in Cluster mode.

[0116] The first sending module 302 is used to send the standard output data in the log to Kafka through the first thread and the first byte stream in the console;

[0117] The second sending module 303 is used to send the standard error output data in the log to Kafka through the second thread and the second byte stream in the console;

[0118] The third sending module 304 is used to send other data in the log to Kafka through the Java logging framework;

[0119] Receiver module 305 is used to receive log collection instructions;

[0120] The determination module 306 is used to determine and display the target log content from Kafka according to the log collection instructions.

[0121] In this embodiment of the application, the first sending module 302 may also be specifically used for:

[0122] Create a new first thread and a first byte stream, where the first byte stream is used to relocate standard output data to the standard output byte stream;

[0123] When the first thread reads the first byte stream status and finds it to be non-empty, the first thread sends the first byte stream to Kafka until the first thread reads the first byte stream status and finds it to be empty.

[0124] In this embodiment of the application, the first sending module 302 may also be specifically used for:

[0125] Create a second thread and a second byte stream, where the second byte stream is used to relocate the standard error output to the standard error output byte stream;

[0126] When the second thread reads the status of the second byte stream and finds it to be non-empty, the second thread sends the second byte stream to Kafka until the second thread reads the status of the second byte stream and finds it to be empty.

[0127] In this embodiment of the application, the third sending module 304 may also be specifically used for:

[0128] Determine the Kafka send function and Java logging framework;

[0129] Based on Kafka's send function and the Java logging framework, other data in the logs are sent to Kafka via the Java logging framework.

[0130] In this embodiment of the application, the receiving module 305 may also be specifically used for:

[0131] Based on the preset framework, logs in Kafka are parsed into target structure logs, and the target display log content in the target structure logs can be obtained through collection commands.

[0132] In this embodiment of the application, the receiving module 305 may also be specifically used for:

[0133] The target structure log is stored in the database so that the page can retrieve the target structure log from the database in response to user actions.

[0134] In this embodiment of the application, the determining module 306 can also be specifically used for:

[0135] Based on the log collection instructions, determine the log filtering conditions and log output format;

[0136] Determine the initial log content from Kafka based on the log filtering criteria;

[0137] Adjust the initial log content according to the log output format to obtain the final log content;

[0138] Display log content.

[0139] In this embodiment of the application, the determining module 306 can also be specifically used for:

[0140] The log content is sent to the server, which then sends the log content to the websocket on the page for display.

[0141] Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Figure 4 As shown, the electronic device 40 includes:

[0142] The electronic device 40 may include a processor 401 with one or more processing cores, a memory 402 with one or more computer-readable storage media, a communication component 403, and other components. The processor 401, memory 402, and communication component 403 are connected via a bus 404.

[0143] In the specific implementation process, at least one processor 401 executes computer execution instructions stored in memory 402, causing at least one processor 401 to execute the log viewing method described above.

[0144] The specific implementation process of processor 401 can be found in the above method embodiments, and its implementation principle and technical effect are similar. It will not be repeated here.

[0145] In the above Figure 4 In the illustrated embodiments, it should be understood that the processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this invention can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules within the processor.

[0146] The memory may include random access memory (RAM) and may also include non-volatile memory (NVM), such as at least one disk storage device.

[0147] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, the buses shown in the accompanying drawings are not limited to a single bus or a single type of bus.

[0148] In some embodiments, a computer program product is also provided, including a computer program or instructions that, when executed by a processor, implement the steps in any of the above-described log viewing methods.

[0149] For details on the implementation of each of the above operations, please refer to the previous examples, which will not be repeated here.

[0150] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be performed by instructions, or by instructions controlling related hardware. These instructions can be stored in a computer-readable storage medium and loaded and executed by a processor.

[0151] Therefore, embodiments of this application provide a computer-readable storage medium storing a plurality of instructions that can be loaded by a processor to execute the steps in any of the log viewing methods provided in embodiments of this application.

[0152] The storage medium may include: read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0153] According to one aspect of this application, a computer program product or computer program is provided, the computer program product or computer program including computer instructions stored in a computer-readable storage medium.

[0154] Since the instructions stored in the storage medium can execute the steps in any of the log viewing methods provided in the embodiments of this application, the beneficial effects that any of the log viewing methods provided in the embodiments of this application can achieve can be realized. For details, please refer to the previous embodiments, which will not be repeated here.

[0155] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this application are indicated by the following claims.

[0156] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. A method for viewing logs, characterized in that, The method includes: Obtain logs from the Spark driver, including standard output data, standard error output data, and other data, wherein the Spark driver is a Spark Driver program in Cluster mode; The standard output data in the log is sent to Kafka through the first thread and the first byte stream in the console. The first byte stream is used to relocate the standard output data to the standard output byte stream. When the first thread reads the status of the first byte stream and it indicates that the first byte stream is not empty, the first thread sends the first byte stream to Kafka until the first thread reads the status of the first byte stream and it indicates that the first byte stream is empty. The standard error output data in the log is sent to Kafka through the second thread and the second byte stream in the console. The second byte stream is used to relocate the standard error output to the standard error output byte stream. When the second thread reads the status of the second byte stream and it indicates that the second byte stream is not empty, the second thread sends the second byte stream to Kafka until the second thread reads the status of the second byte stream and it indicates that the second byte stream is empty. The remaining data in the logs are sent to Kafka via the Java logging framework; Based on the preset framework, the logs in Kafka are parsed into logs of the target structure; Receive log collection instructions; According to the log collection instruction, determine the log filtering conditions and log output format; according to the log filtering conditions, determine the initial log content from the target structure log; according to the log output format, adjust the initial log content to obtain the log content; and display the log content.

2. The method according to claim 1, characterized in that, The step of sending other data from the log to Kafka via the Java logging framework includes: Determine the Kafka send function and Java logging framework; Based on Kafka's send function and the Java logging framework, the other data in the logs are sent to Kafka via the Java logging framework.

3. The method according to claim 1, characterized in that, After parsing the logs in Kafka into the target structure logs according to the preset framework, the method further includes: The target structure log is stored in a database so that the page can retrieve the target structure log from the database in response to user actions.

4. The method according to claim 1, characterized in that, Display log content, including: The log content is sent to the server, so that the server sends the log content to the websocket on the page and displays it.

5. A log viewing device, characterized in that, include: The acquisition module is used to acquire logs from the Spark driver, which include standard output data, standard error output data, and other data. The Spark driver is the Spark Driver program in Cluster mode. The first sending module is used to send the standard output data in the log to Kafka through the first thread and the first byte stream in the console. The first byte stream is used to relocate the standard output data to the standard output byte stream. When the first thread reads the status of the first byte stream to indicate that the first byte stream is not empty, the first thread sends the first byte stream to Kafka until the first thread reads the status of the first byte stream to indicate that the first byte stream is empty. The second sending module is used to send the standard error output data in the log to Kafka through the second thread and the second byte stream in the console. The second byte stream is used to relocate the standard error output to the standard error output byte stream. When the second thread reads the status of the second byte stream and indicates that the second byte stream is not empty, the second thread sends the second byte stream to Kafka until the second thread reads the status of the second byte stream and indicates that the second byte stream is empty. The third sending module is used to send other data in the log to Kafka via the Java logging framework; The receiving module is used to parse logs in Kafka into logs of the target structure according to a preset framework; and to receive log collection instructions. The determination module is used to determine log filtering conditions and log output format according to the log collection instruction; determine initial log content from the target structure log according to the log filtering conditions; adjust the initial log content according to the log output format to obtain log content; and display the log content.

6. An electronic device, characterized in that, include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory to implement the method as described in any one of claims 1 to 4.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of claims 1 to 4.

Images