Log backward-reading method, log processing method and related device

Abstract

The invention discloses a log backward-reading method and a related device, and relates to the field of data processing, and the method comprises the following steps: obtaining a log backward-reading instruction; analyzing the log read-back instruction to obtain read-back log configuration parameters; determining a target log required to be read back from the log file based on the read-back log configuration parameter; and calling and taking the target log as response data of the log read-back instruction. According to the scheme, when the log is read back, the target log needing to be read back this time is determined and called based on the read-back log configuration parameters in the log read-back instruction, and then the determined target log is used as the response data of the log read-back instruction to be fed back to the sender of the log read-back instruction, so that the scheme does not need to feed back an overall log file, and the log read-back efficiency is improved. When the log file is read back, only the target log matched with the read-back log configuration parameter needs to be fed back, the data size of the target log is smaller than the data size of the whole log file, the data interaction amount is reduced, and then the problems of bandwidth and storage redundancy existing in an existing log reading mode are solved.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a log readback method, a log processing method, and related apparatus. Background Technology

[0002] "Log rereading" typically refers to reviewing or analyzing previously recorded log data within a system or application. This log data may contain useful information about system activity, user behavior, error messages, warnings, debugging information, and more. Log rereading is a common step in system maintenance, troubleshooting, and performance optimization.

[0003] In the existing logs, logs from various business operations are mixed together and stored as a whole in the system. When a business needs the logs, the entire log needs to be transmitted. This log reading method has problems with high bandwidth requirements and storage redundancy. Summary of the Invention

[0004] In view of the above problems, this application provides a log back reading method, a log processing method, and related apparatus to achieve accurate log back reading, thereby reducing the amount of data interaction during the log reading process and reducing bandwidth redundancy and storage redundancy. The specific solution is as follows:

[0005] The first aspect of this application provides a log readback method, including:

[0006] Obtain log readback command;

[0007] The log readback command is parsed to obtain the log readback configuration parameters;

[0008] The target log to be read back is determined from the log file based on the readback log configuration parameters.

[0009] The target log is retrieved and used as the response data for the log readback command.

[0010] Optionally, in the above log readback method, the log readback configuration parameters include at least one or more of the following: timestamp, process ID, thread ID, log level, and / or tag.

[0011] Optionally, in the above log readback method, retrieving and using the target log as response data for the log readback instruction includes:

[0012] Determine whether the complete target log can be retrieved from the locally stored log file;

[0013] When the complete target log can be retrieved from the locally stored log file, the target log is retrieved from the locally stored log file and used as the response data for the log readback command;

[0014] When the complete target log cannot be retrieved from the locally stored log file, the target log is retrieved from the locally stored log file and recorded as the first target log; the target log is retrieved from the log file of the target terminal that matches the log readback command and recorded as the second target log; the first target log and the second target log are used as the response data for the log readback command.

[0015] Optionally, in the above log readback method, after retrieving the target log from the log file of the target terminal matching the log readback instruction and recording it as the second target log, the method further includes:

[0016] The second target log is stored in the log file stored locally.

[0017] Optionally, in the above log readback method, after storing the second target log in the locally stored log file, the method further includes:

[0018] Determine whether the storage duration of the second target log has reached the set duration. If the set duration has been reached, delete the second target log from the locally stored log file.

[0019] A log processing method, comprising:

[0020] Extract the timestamps and / or variable contents of the log files to be processed;

[0021] The timestamp and / or variable content corresponding to the log to be processed are stored in an index dictionary as log configuration parameters of the log to be processed. The index dictionary stores the mapping relationship between the log configuration parameters and the encoded message, where the encoded message is the encoded message of the log to be processed.

[0022] Optionally, the above log processing methods also include:

[0023] When the system referenced by the log processing method is the Android system, the process ID and thread ID of the log to be processed are stored as log configuration parameters in the index dictionary.

[0024] Optionally, the above log processing method, before extracting the timestamp and / or variable content of the log to be processed from the log file, further includes:

[0025] The acquired logs to be processed are compressed using a streaming log structured compression method to obtain a log file to be processed. A log readback device includes:

[0026] The instruction acquisition unit is used to acquire log readback instructions;

[0027] The parsing unit is used to parse the log readback command to obtain the log readback configuration parameters;

[0028] The target log identification unit is used to determine the target log to be read back from the log file based on the readback log configuration parameters.

[0029] A response unit is used to retrieve the target log and use it as response data for the log readback instruction.

[0030] A computer program product includes computer-readable instructions that, when executed on an electronic device, cause the electronic device to implement any of the log readback methods described above.

[0031] A log processing apparatus, comprising:

[0032] Adjust the extraction unit to extract timestamps and / or variable content from the logs to be processed;

[0033] The dictionary processing unit is used to store the timestamp and / or variable content corresponding to the log to be processed as log configuration parameters of the log to be processed in an index dictionary. The index dictionary stores the mapping relationship between the log configuration parameters and the encoded message, and the encoded message is the encoded message of the log to be processed.

[0034] An electronic device includes at least one processor and a memory connected to the processor, wherein:

[0035] The memory is used to store computer programs;

[0036] The processor is used to execute the computer program to enable the electronic device to perform log back read as described in any of the above.

[0037] A computer storage medium carrying one or more computer programs, which, when executed by an electronic device, enable the electronic device to implement the log readback method or the log processing method described in any one of the preceding claims.

[0038] A third aspect of this application provides a computer program product, including computer-readable instructions that, when executed on an electronic device, cause the electronic device to implement the log readback method of the first aspect or any implementation thereof, or the log processing method described in any one of the above.

[0039] A fourth aspect of this application provides an electronic device, including at least one processor and a memory connected to the processor, wherein:

[0040] The memory is used to store computer programs;

[0041] The processor is used to execute the computer program so that the electronic device can implement the log readback method of the first aspect or any implementation thereof, or the log processing method of any one of the above.

[0042] By employing the above technical solution, the solution provided in this application, when reading back logs, can determine and retrieve the target log to be read back based on the log readback configuration parameters in the log readback command, and then feed back the determined target log as the response data of the log readback command to the sender of the log readback command. It can be seen that when responding to the log readback command, this solution does not need to feed back the entire log file, but only the target log that matches the log readback configuration parameters. The data volume of the target log is smaller than the data volume of the entire log file, which reduces the amount of data interaction and thus solves the problems of large bandwidth requirements and storage redundancy in existing log reading methods. Attached Figure Description

[0043] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and the originals and elements are not necessarily drawn to scale.

[0044] Figure 1 A schematic diagram of an implementation system architecture for the log readback method provided in this application embodiment;

[0045] Figure 2 This application provides a schematic diagram of a terminal structure as an embodiment.

[0046] Figure 3 A schematic diagram of a server structure provided in an embodiment of this application;

[0047] Figure 4 A flowchart illustrating a log readback method provided in an embodiment of this application;

[0048] Figure 5 A flowchart illustrating a log readback method provided in another embodiment of this application;

[0049] Figure 6 This is a schematic diagram of a log file compression process disclosed in an embodiment of this application;

[0050] Figure 7 This is a schematic flowchart of a log processing method disclosed in an embodiment of this application;

[0051] Figure 8This is a schematic diagram of the structure of a log readback device disclosed in an embodiment of this application;

[0052] Figure 9 This is a schematic diagram of the structure of a log processing device disclosed in an embodiment of this application;

[0053] Figure 10 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0054] The embodiments of this application are described below with reference to the accompanying drawings. The terminology used in the implementation section of this application is for explaining specific embodiments only and is not intended to limit the scope of this application.

[0055] The embodiments of this application will now be described with reference to the accompanying drawings. Those skilled in the art will recognize that, with technological advancements and the emergence of new scenarios, the technical solutions provided in the embodiments of this application are equally applicable to similar technical problems.

[0056] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such terms are interchangeable where appropriate; this is merely a way of distinguishing objects with the same attributes in the embodiments of this application. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion, so that a process, method, system, product, or apparatus that comprises a series of elements is not necessarily limited to those elements, but may include other elements not explicitly listed or inherent to those processes, methods, products, or apparatuses.

[0057] The log readback method provided in this application can be applied to, for example... Figure 1 The system architecture shown is as follows: Figure 1 A schematic diagram of a system architecture is shown. The system may include a terminal 100 and a server 200. The server 200 may include one or more servers (…). Figure 1 (The example includes a server), and the server 200 can provide the method provided in the embodiments of this application to one or more terminals.

[0058] The terminal 100 may be equipped with an application for performing log back reading. The application and the webpage can provide an interface. The terminal 100 can receive relevant parameters input by the user on the user interaction interface and send the parameters to the server 200. The server 200 can obtain the processing result based on the received parameters and return the processing result to the terminal 100.

[0059] It should be understood that in some optional implementations, the terminal 100 can also complete the action of obtaining the processing result based on the received parameters on its own, without the need for the server to cooperate. This application embodiment is not limited to this.

[0060] The following description Figure 1 The product form of the mid-terminal 100;

[0061] The terminal 100 in this application embodiment can be a mobile phone, tablet computer, wearable device, vehicle device, augmented reality (AR) / virtual reality (VR) device, laptop computer, ultra-mobile personal computer (UMPC), netbook, personal digital assistant (PDA), etc., and this application embodiment does not impose any restrictions on it.

[0062] Figure 2 A schematic diagram of an optional hardware structure for terminal 100 is shown.

[0063] refer to Figure 2 As shown, the terminal 100 may include a radio frequency unit 110, a memory 120, an input unit 130, a display unit 140, a camera 150 (optional), an audio circuit 160 (optional), a speaker 161 (optional), a microphone 162 (optional), a headphone jack 163 (optional), a processor 170, an external interface 180, a power supply 190, and other components. Those skilled in the art will understand that... Figure 2 These are merely examples of terminals or multi-functional devices and do not constitute a limitation on terminals or multi-functional devices. They may include more or fewer components than shown in the illustration, or combine certain components, or use different components.

[0064] The input unit 130 can be used to receive input numerical or character information, and to generate key signal inputs related to user settings and function control of the portable multi-functional device. Specifically, the input unit 130 may include a touchscreen 131 (optional) and / or other input devices 132. The touchscreen 131 can collect touch operations performed by the user on or near it (such as operations performed by the user using fingers, knuckles, styluses, or any suitable object on or near the touchscreen), and drive the corresponding connection devices according to a pre-set program. The touchscreen can detect the user's touch actions, convert the touch actions into touch signals and send them to the processor 170, and can receive and execute commands sent by the processor 170; the touch signal includes at least touch point coordinate information. The touchscreen 131 can provide an input interface and an output interface between the terminal 100 and the user. In addition, various types of touchscreens, such as resistive, capacitive, infrared, and surface acoustic wave, can be used to implement the touchscreen. Besides the touchscreen 131, the input unit 130 may also include other input devices. Specifically, other input devices 132 may include, but are not limited to, one or more of the following: physical keyboard, function keys (such as volume control buttons, power buttons, etc.), trackball, mouse, joystick, etc.

[0065] Among them, the input device 132 can receive input data, etc.

[0066] The display unit 140 can be used to display information input by the user or information provided to the user, various menus of the terminal 100, interactive interfaces, file display, and / or playback of any multimedia file. In this embodiment, the display unit 140 can be used to display user interaction interfaces, processing results, etc.

[0067] The memory 120 can be used to store instructions and data. The memory 120 may primarily include an instruction storage area and a data storage area. The data storage area can store various types of data, such as multimedia files and text. The instruction storage area can store software units such as operating systems, applications, and instructions required for at least one function, or subsets or extended sets thereof. It may also include non-volatile random access memory. It provides the processor 170 with hardware, software, and data resources for managing the computing device, supporting control software and applications. It is also used for storing multimedia files, as well as storing running programs and applications.

[0068] The processor 170 is the control center of the terminal 100. It connects various parts of the terminal 100 via various interfaces and lines. By running or executing instructions stored in the memory 120 and calling data stored in the memory 120, it performs various functions and processes data of the terminal 100, thereby controlling the terminal device as a whole. Optionally, the processor 170 may include one or more processing units; preferably, the processor 170 may integrate an application processor and a modem processor, wherein the application processor mainly handles the operating system, user interface, and applications, and the modem processor mainly handles wireless communication. It is understood that the modem processor may not be integrated into the processor 170. In some embodiments, the processor and memory can be implemented on a single chip; in some embodiments, they can also be implemented separately on independent chips. The processor 170 can also be used to generate corresponding operation control signals, send them to the corresponding components of the computing processing device, read and process data in the software, especially read and process data and programs in the memory 120, so that the various functional modules therein perform corresponding functions, thereby controlling the corresponding components to act according to the instructions.

[0069] The memory 120 can be used to store software code related to the executable log readback method, and the processor 170 can execute the steps of the log readback method, and can also schedule other units (such as the above-mentioned input unit 130 and display unit 140) to achieve the corresponding functions.

[0070] The radio frequency unit 110 (optional) can be used for receiving and transmitting signals during information transmission or calls. For example, it can receive downlink information from the base station and process it for the processor 170; additionally, it can transmit uplink data to the base station. Typically, the RF circuit includes, but is not limited to, an antenna, at least one amplifier, a transceiver, a coupler, a low-noise amplifier (LNA), a duplexer, etc. Furthermore, the radio frequency unit 110 can also communicate wirelessly with network devices and other devices. This wireless communication can use any communication standard or protocol, including but not limited to Global System for Mobile communication (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Long Term Evolution (LTE), email, Short Messaging Service (SMS), etc.

[0071] In this embodiment of the application, the radio frequency unit 110 can send data to the server 200 and receive the processing results sent by the server 200.

[0072] It should be understood that the radio frequency unit 110 is optional and can be replaced with other communication interfaces, such as a network port.

[0073] The terminal 100 also includes a power supply 190 (such as a battery) that supplies power to various components. Preferably, the power supply can be logically connected to the processor 170 through a power management system, thereby enabling functions such as charging, discharging, and power consumption management through the power management system.

[0074] Terminal 100 also includes an external interface 180, which can be a standard Micro USB interface or a multi-pin connector, which can be used to connect terminal 100 to other devices for communication or to connect a charger to charge terminal 100.

[0075] Although not shown, terminal 100 may also include a flash, a wireless fidelity (WiFi) module, a Bluetooth module, sensors with various functions, etc., which will not be described in detail here. Some or all of the methods described below can be applied to, for example... Figure 2 In the terminal 100 shown.

[0076] The following description Figure 1 The product form of the mid-range server 200;

[0077] Figure 3 A structural diagram of a server 200 is provided, as follows: Figure 3 As shown, server 200 includes bus 201, processor 202, communication interface 203, and memory 204. Processor 202, memory 204, and communication interface 203 communicate with each other via bus 201.

[0078] Bus 201 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of representation, Figure 3 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0079] The processor 202 can be any one or more of the following processors: central processing unit (CPU), graphics processing unit (GPU), microprocessor (MP), or digital signal processor (DSP).

[0080] Memory 204 may include volatile memory, such as random access memory (RAM). Memory 204 may also include non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid state drive (SSD).

[0081] The memory 204 can be used to store software code related to the log readback method, and the processor 202 can execute the steps of the chip's log readback method, or schedule other units to achieve the corresponding functions.

[0082] It should be understood that the aforementioned terminal 100 and server 200 can be centralized or distributed devices. The processors (e.g., processor 170 and processor 202) in the aforementioned terminal 100 and server 200 can be hardware circuits (such as application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), general-purpose processors, digital signal processors (DSPs), microprocessors or microcontrollers, etc.) or combinations of these hardware circuits. For example, the processor can be a hardware system with instruction execution capabilities, such as a CPU or DSP, or a hardware system without instruction execution capabilities, such as an ASIC or FPGA, or a combination of the aforementioned hardware systems without instruction execution capabilities and hardware systems with instruction execution capabilities.

[0083] This application provides a log readback method, which can be applied to... Figure 1 Taking the computer device in the diagram as an example, this computer device could specifically be terminal 110 or server 120 in the diagram above. (Refer to...) Figure 4 The log readback method specifically includes the following steps:

[0084] Step S401: Obtain the log readback command.

[0085] Step S402: Parse the log readback command to obtain the log readback configuration parameters.

[0086] Step S403: Determine the target log to be read back from the log file based on the readback log configuration parameters.

[0087] Step S404: Retrieve the target log and use it as the response data for the log readback command.

[0088] In this embodiment, log backreading refers to rereading or processing previous log records to achieve a certain purpose, such as data consistency restoration, event replay, performance analysis, etc. Log backreading is typically required in scenarios including: fault analysis and problem localization, performance optimization, behavior verification, auditing and compliance checks, security analysis and risk identification, disaster recovery and data recovery, etc. When a user needs to backread logs, a log backreading instruction is generated. This log backreading instruction may include log backreading configuration parameters, which can be entered by the user through an interactive interface. These parameters specify the logs to be backread; that is, by parsing the log backreading configuration parameters, it can be determined which logs in the log file the user needs to backread.

[0089] In this embodiment, the readback log configuration parameters can be pre-stored in a dictionary, and an index can be established between the readback log configuration parameters and the original log. Based on the index relationship, the target log to be read back can be quickly determined based on the readback log configuration parameters.

[0090] Taking the Android car infotainment system as an example, the log format of the Android car infotainment system is as follows:

[0091] 01-01 12:00:00.000 1572 1572 I / ActivityManager(1): Start proc 1572:com.example.app / u0a0 for activity com.example.app / .MainActivity;

[0092] 01-01 12:00:01.000 1572 1572 D / MyActivity( 1572): onCreate() calledwith Bundle;

[0093] 01-01 12:00:02.000 906 912 I / SurfaceFlinger( 906): Graphic bufferallocation for surface 0xb7a0d4d8, size 1080x1920;

[0094] 01-01 12:00:03.000 1572 1572 I / MyActivity( 1572): onResume() called;

[0095] 01-01 12:00:04.000 1572 1572 D / MyService( 1572): onStartCommand()called with Intent { act=com.example.app.ACTION_START_SERVICE};

[0096] 01-01 12:00:05.000 903 903 W / System(1572): Class not found forresource;

[0097] 01-01 12:00:06.000 1572 1572 E / MyActivity( 1572): Error: Unable toopen content provider;

[0098] 01-01 12:00:07.000 1572 1572 F / ActivityManager(1): Error reportingcrash of process 1572: com.example.app;

[0099] The contents of each part of the log entry are represented as follows:

[0100] Timestamp: 01-01 12:00:00.000 indicates the log time and the timestamp accurate to milliseconds.

[0101] Process ID: For example, 1572 represents the process ID that generated the log.

[0102] Thread ID: 1572 indicates the thread ID that generated the log (in this example, the process ID and thread ID are the same, indicating that the log comes from the main thread).

[0103] Log levels: I for Info, D for Debug, W for Warning, E for Error, and F for Fatal.

[0104] Tags such as ActivityManager, MyActivity, SurfaceFlinger, MyService, and System are tags for components or classes that generate logs.

[0105] Log content: The following text is the specific content of the log message.

[0106] As shown in the log above, if it is necessary to analyze the MyActivity log in the TAG business, simply use the log configuration parameters of the MyActivity log as the log readback configuration parameters. At this time, the target log to be read back can be determined as MyActivity based on the log readback configuration parameters, and there is no need to read back other redundant logs. However, in the existing system, all the logs above need to be sent to the reader.

[0107] In this embodiment, after determining the target log to be read back from the log file based on the log readback configuration parameters, data is extracted from the log file to obtain the target log, and then the obtained target log is fed back to the sender of the log readback command as the response data of the log readback command.

[0108] As can be seen from the above solution, the technical solution disclosed in this application, when reading back logs, can determine and retrieve the target log to be read back based on the log readback configuration parameters in the log readback command, and then feed back the determined target log as the response data of the log readback command to the sender of the log readback command. It can be seen that when responding to the log readback command, this solution does not need to feed back the entire log file, but only the target log that matches the log readback configuration parameters. The data volume of the target log is smaller than the data volume of the entire log file, which reduces the amount of data interaction and thus solves the problems of bandwidth and storage redundancy in the existing log reading method.

[0109] In one embodiment of this application, the specific type of the readback log configuration parameter can be set by the user according to their needs, as long as the required target log can be found and determined. For example, the readback log configuration parameter may include any one or more of the following: timestamp, process ID, thread ID, log level and / or tag.

[0110] Optionally, this method can be applied to cloud devices or on-device devices. However, in real-world scenarios, log files may be partially stored in the cloud and partially stored on the device. Therefore, devices using this method may only be able to find part of the target log or may not be able to find the target log at all. For this situation, please refer to [link to relevant documentation]. Figure 5 The step of retrieving and using the target log as response data for the log readback instruction includes:

[0111] Step S501: Determine whether the complete target log can be retrieved from the locally stored log file.

[0112] Step S502: When the complete target log can be retrieved from the locally stored log file, the target log is retrieved from the locally stored log file and used as the response data for the log readback instruction.

[0113] Step S503: When the complete target log cannot be retrieved from the locally stored log file; the target log retrieved from the locally stored log file is recorded as the first target log; the target log retrieved by the target terminal matched by the log readback command is recorded as the second target log, the second target log is obtained from the log file in the target terminal, and the log obtained from the locally stored log file and the target log retrieved by the target terminal are used as the response data of the log readback command.

[0114] In this embodiment, "local" refers to the local machine of the device applying the log readback method. In this solution, after determining the target log, it can be first determined whether the complete target log can be retrieved locally. For example, if the target log is some logs between time period A and time period C, the local machine may only store the target logs between time period B and time period C, while the target logs corresponding to time period A and time period B are stored on the device. In this case, only a portion of the target logs can be retrieved from the log files stored locally on the device using this method.

[0115] In this embodiment, when the log readback method is applied to a cloud device, step S501 specifically involves determining whether the complete target log can be retrieved from the log file stored locally on the cloud device. When the log readback method is applied to a user device, step S501 specifically involves determining whether the complete target log can be retrieved from the log file stored locally on the user device. If the complete target log can be retrieved from the locally stored log file, it can be retrieved locally and the target log can be used to respond to the log readback instruction.

[0116] In this implementation, when it is determined that not all target logs are stored locally, all target logs stored in the local log file are first extracted. Then, the log file of the target terminal matching the log readback command is read, and the remaining target logs are retrieved from that target terminal's log file. The target logs obtained from both methods are used together as the response data for the log readback command. When the log readback method is applied to a cloud device, the target terminal refers to the user device; when the log readback method is applied to the user device, the target terminal refers to the cloud device.

[0117] Optionally, in the above embodiments, after obtaining the first target log and the second target log, the first target log and the second target log can be merged and de-redundanted, and the merged and de-redundanted first target log and the second target log can be used as the response data for the log readback instruction. Alternatively, only the first target log and the second target log can be de-redundanted, and the de-redundanted first target log and the second target log can be used to respond to the log readback instruction separately.

[0118] Optionally, when the target terminal captures the target logs, under normal circumstances, after responding to the log readback command, these logs need to be deleted to avoid them occupying local memory. In this solution, considering that the user may collect this part of the target logs again in the near future, the second target logs obtained by the target terminal can be stored in a locally stored log file first. After the second target logs are stored in the locally stored log file, a timer is started to determine whether the timer has reached a set duration. When the timer reaches a set duration (e.g., 1 hour, 6 hours, 12 hours, or 24 hours), it indicates that the user will no longer need to obtain the second target logs in the near future. At this time, the second target logs can be deleted from the locally stored log file to free up local space. This method allows the readback data to be retained for a certain period of time, making it convenient for users to use directly, avoiding repeated readbacks from the vehicle end, and saving network bandwidth.

[0119] In the technical solutions disclosed in the above embodiments of this application, when "determining the target log to be read back from the log file based on the readback log configuration parameters," log searching is involved. Existing log search tools widely used in the technology can index logs to provide fast search performance, but the size of the index is on the same order of magnitude as the original log size. Furthermore, in traditional solutions, when processing log files, log archiving and compression are required based on the log type. Commonly used log archiving and compression tools can provide high compression ratios, but searching archived logs is a slow and complex process because the log file must first be decompressed. In contrast, compared to traditional compression tools, the compression ratio of structured log compression is significantly higher than all commonly used compression methods. The compression ratio of structured log compression is three times that of current Zip compression, and it also has search performance that other compression methods do not possess. Therefore, the log file in this solution can be a log file compressed using structured log compression.

[0120] Corresponding to the log readback method described above, this application also discloses a log processing method, which belongs to the log structure compression method, see [link to relevant documentation]. Figure 7 The method may include:

[0121] Step S701: Extract the timestamps and / or variable contents of the logs to be processed from the log files to be processed.

[0122] Step S702: Store the timestamp and / or variable content corresponding to the log to be processed in the index dictionary as the log configuration parameters of the log to be processed.

[0123] In this embodiment, the structured log compression method mainly consists of two stages: encoding and compression. The encoding stage in this scheme is implemented through steps S701 and S702. A log to be processed can typically be divided into the following parts: timestamp, variables, and static text content. The static text is the encoded log message, and the variables are the modified content extracted from the encoded log content. Simultaneously, a text matching algorithm is used to identify the variable content in the log. In this scheme, the timestamp and / or the variable content can be used as log configuration parameters for the log to be processed, depending on the requirements. The variable content may include the log level and / or tag corresponding to the log to be processed. Of course, if the application system of the log processing method is an Android system, the variable content may also include the process ID and thread ID.

[0124] In this embodiment, after determining the log configuration parameters of the log to be processed, the log configuration parameters are stored in an index dictionary. The index dictionary stores the mapping relationship between the log configuration parameters and the encoded message, where the encoded message is the encoded message of the log to be processed. Then, after obtaining the log readback configuration parameters, the index dictionary can be matched and searched according to these parameters to determine the corresponding encoded message, and thus the target log corresponding to that encoded message can be determined.

[0125] Furthermore, regarding the compression stage: if, during the processing of the logs to be processed, the buffered logs exceed a preset value, a preset compression method (e.g., Lzma) can be used to compress the logs to be processed and the index dictionary. In this embodiment, the device applying the log processing method can be an Android device or a device operating another operating system. Taking the Android system as an example, since the log file needs to be compressed using a structured log compression method, in order to reduce disk writes, a streaming structured log compression method can be used first to compress the obtained logs to be processed, generating a log file to be processed. The integration of the structured log compression method on the Android device can be divided into two parts. That is, the first part integrates the streaming structured log compression method, and the other part integrates the standard structured log compression method. The first part is used to integrate the streaming structured log compression method during the log file generation stage, that is, the logs to be processed are compressed using the streaming structured log compression method to obtain the log file to be processed. In other words, before obtaining the log readback instruction, it is necessary to obtain the log file to be processed after compressing the logs using the streaming structured log compression method. The difference between streaming log structured compression and standard log structured compression is that streaming compression can compress logs in real time, without waiting for the log to be completely written to the file before compression. That is, whenever a new log entry is written, it can be encoded and compressed directly. For example... Figure 6The Android Log refers to the pending logs of the Android system, while Other Logs refer to pending logs from other systems. Streaming compression reduces disk writes, and files compressed using streaming structured compression, like those compressed using standard structured compression, offer direct search capabilities. However, compared to standard structured compression, streaming compression lacks an index dictionary, resulting in lower compression ratios and search performance. After obtaining the pending log file compressed using streaming structured compression, it can be further processed using standard structured compression. This involves compressing the log file again using standard structured compression and building an index dictionary based on the log file's configuration parameters and content. This process can be implemented in the cloud or on the user device. When implemented on the user device, after compressing the log file using standard structured compression and building the index dictionary based on the log file's configuration parameters and content, the user device must upload the compressed log file and the constructed index dictionary to the cloud device.

[0126] The above describes a log readback method provided by an embodiment of this application. The following describes the apparatus for performing the above log readback method.

[0127] Please see Figure 8 , Figure 8 This is a schematic diagram of a log readback device provided in an embodiment of this application. Figure 8 As shown, the log readback device includes:

[0128] Instruction acquisition unit 1, corresponding to the above method, is used to acquire log readback instructions;

[0129] Parsing unit 2, corresponding to the above method, is used to parse the log readback command to obtain the log readback configuration parameters;

[0130] The target log identification unit 3, corresponding to the above method, is used to determine the target log to be read back from the log file based on the readback log configuration parameters.

[0131] Response unit 4, corresponding to the above method, is used to retrieve and use the target log as response data for the log readback instruction.

[0132] Corresponding to the above method, when the response unit 4 retrieves and uses the target log as response data for the log readback instruction, it is specifically used for:

[0133] Determine whether the complete target log can be retrieved from the locally stored log file;

[0134] When the complete target log can be retrieved from the locally stored log file, the target log is retrieved from the locally stored log file and used as the response data for the log readback command;

[0135] When the complete target log cannot be retrieved from the locally stored log file, the target log is retrieved from the locally stored log file and recorded as the first target log; the target log is retrieved from the log file of the target terminal that matches the log readback command and recorded as the second target log; the first target log and the second target log are used as the response data for the log readback command.

[0136] Corresponding to the above method, the above device also includes a storage unit, used to retrieve the target log from the log file of the target terminal matched by the log readback instruction, record it as the second target log, and then store the second target log in the log file of the local storage.

[0137] Corresponding to the above method, the storage unit is also used to: determine whether the storage duration of the second target log has reached the set duration, and when the set duration is reached, delete the second target log from the log file stored locally.

[0138] A log processing device, see Figure 9 The device includes:

[0139] Adjust extraction unit 5 to extract the timestamps and / or variable contents of the log files to be processed.

[0140] The dictionary processing unit 6 is used to store the timestamp and / or variable content corresponding to the log to be processed as log configuration parameters of the log to be processed in an index dictionary. The index dictionary stores the mapping relationship between the log configuration parameters and the encoded message, and the encoded message is the encoded message of the log to be processed.

[0141] Corresponding to the above method, the log processing device based on log structure compression may further include a streaming log structure compression unit, used to compress the acquired logs to be processed using streaming log structure compression to obtain the log files to be processed.

[0142] This application also provides an electronic device in its embodiments. (See reference...) Figure 10The diagram illustrates a structural schematic suitable for implementing the electronic device in the embodiments of this application. The electronic device in the embodiments of this application may include, but is not limited to, fixed terminals such as mobile phones, laptops, PDAs (personal digital assistants), PADs (tablet computers), desktop computers, etc. Figure 10 The electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0143] like Figure 10 As shown, the electronic device may include a processing unit (e.g., a central processing unit, a graphics processing unit, etc.) 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage device 608 into a random access memory (RAM) 603. When the electronic device is powered on, the RAM 603 also stores various programs and data required for the operation of the electronic device. The processing unit 601, ROM 602, and RAM 603 are interconnected via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0144] Typically, the following devices can be connected to I / O interface 605: input devices 606 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 607 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 608 including, for example, memory cards, hard drives, etc.; and communication devices 609. Communication device 609 allows electronic devices to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 10 Electronic devices with various devices are shown, but it should be understood that it is not required to implement or have all of the devices shown. More or fewer devices may be implemented or have instead.

[0145] This application also provides a computer program product including computer-readable instructions, which, when executed on an electronic device, cause the electronic device to implement any of the log readback method and log processing method provided in this application.

[0146] This application also provides a computer-readable storage medium that carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement any of the log readback methods and log processing methods provided in this application.

[0147] It should also be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. In addition, in the device embodiment drawings provided in this application, the connection relationship between modules indicates that they have a communication connection, which can be implemented as one or more communication buses or signal lines.

[0148] Through the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware, or it can be implemented by special-purpose hardware including application-specific integrated circuits, special-purpose CPUs, special-purpose memory, special-purpose components, etc. Generally, any function performed by a computer program can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be diverse, such as analog circuits, digital circuits, or special-purpose circuits. However, for this application, software program implementation is more often the preferred implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a readable storage medium, such as a computer floppy disk, USB flash drive, mobile hard disk, ROM, RAM, magnetic disk, or optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, training equipment, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0149] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product.

[0150] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, training device, or data center to another website, computer, training device, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device or data center that integrates one or more available media. The available media may be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state drives (SSDs)).

Claims

1. A log readback method characterized by, The method comprises: obtaining a log readback instruction; obtaining a readback log configuration parameter based on the log readback instruction; determining a target log required to be readback from a log file based on the readback log configuration parameter; calling and taking the target log as response data of the log readback instruction.

2. The log readback method of claim 1, wherein, The readback log configuration parameter at least comprises a combination of any one or more of a timestamp, a process ID, a thread ID, a log level and / or a label.

3. The log readback method of claim 1, wherein, The calling and taking the target log as response data of the log readback instruction comprises: judging whether the complete target log can be called from a locally stored log file; when the complete target log can be called from the locally stored log file, calling the target log from the locally stored log file and taking the target log as response data of the log readback instruction; when the complete target log cannot be called from the locally stored log file, calling the target log from a log file of a target terminal matched with the log readback instruction and recording the target log as a first target log; calling the target log from a log file of a target terminal matched with the log readback instruction and recording the target log as a second target log; taking the first target log and the second target log as response data of the log readback instruction.

4. The log readback method of claim 3, wherein, After calling the target log from a log file of a target terminal matched with the log readback instruction and recording the target log as a second target log, the method further comprises: storing the second target log in the locally stored log file.

5. The log readback method of claim 4, wherein, After storing the second target log in the locally stored log file, the method further comprises: judging whether a storage duration of the second target log reaches a set duration, and when the storage duration reaches the set duration, deleting the second target log from the locally stored log file.

6. A log processing method characterized by, The method comprises: extracting a timestamp and / or variable content of a log to be processed in a log file to be processed; storing the timestamp and / or variable content of the log to be processed as a log configuration parameter of the log to be processed in an index dictionary, the index dictionary storing a mapping relationship between the log configuration parameter and an encoded message, the encoded message being an encoded message of the log to be processed.

7. The log processing method of claim 6, wherein, The method further comprises: when a reference system of the log processing method is an Android system, storing a process number and a thread number of the log to be processed as a log configuration parameter in the index dictionary.

8. The log processing method of claim 7, wherein, The method further comprises: judging whether a total number of the logs to be processed is greater than a preset value, and if the total number is greater than the preset value, compressing the log to be processed and the index dictionary by using a preset compression mode.

9. The log processing method of claim 7, wherein, Before extracting the timestamp and / or variable content of the log to be processed in the log file to be processed, the method further comprises: compressing the log to be processed obtained by using a streaming log structured compression mode to obtain the log file to be processed.

10. A log readback apparatus characterized by comprising: The method comprises: an instruction obtaining unit configured to obtain a log readback instruction; an analysis unit configured to analyze the log readback instruction to obtain a readback log configuration parameter; a target log identifying unit configured to determine a target log required to be readback from a log file based on the readback log configuration parameter; a response unit configured to call and take the target log as response data of the log readback instruction.

11. A log processing apparatus characterized by comprising: The method comprises: The adjusting extracting unit is configured to extract a timestamp and / or variable content of a to-be-processed log in a to-be-processed log file. The dictionary processing unit is configured to store the timestamp and / or variable content of the to-be-processed log as a log configuration parameter of the to-be-processed log in an index dictionary, and the index dictionary stores a mapping relationship between the log configuration parameter and an encoded message, and the encoded message is an encoded message of the to-be-processed log.

12. A computer program product, characterised in that, Computer readable instructions are included, when the computer readable instructions are run on an electronic device, the electronic device implements the log back-reading method of any one of claims 1 to 5, or the log processing method of any one of claims 6-9.

13. An electronic device, comprising: The memory is configured to store a computer program; The processor is configured to execute the computer program, so that the electronic device can implement the log back-reading method of any one of claims 1 to 5, or the log processing method of any one of claims 6-9. The storage medium carries one or more computer programs, when the one or more computer programs are executed by an electronic device, the electronic device can implement the log back-reading method of any one of claims 1 to 5, or the log processing method of any one of claims 6-9.

14. A computer storage medium, characterized in that ​

Images