Log processing methods, devices, electronic equipment and storage media

Abstract

This disclosure relates to a log processing method, apparatus, electronic device, and storage medium, belonging to the field of data processing. The method includes: generating a request log in response to a first service request; obtaining response data from the request log; when the data volume of the response data exceeds a set threshold, generating a response ID mapped to the response data, and recording the mapping relationship between the response data and the response ID in a mapping log; replacing the response data with the response ID and writing it into a regular log; writing the regular log into a regular log file; and writing the mapping log into a large log file; wherein data volumes exceeding a data volume threshold are considered large logs, and data volumes less than the data volume threshold are considered regular logs. This method can, to some extent, improve the storage problems caused by excessively large log data and the difficulties in log querying caused by excessively large log data.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the field of data processing, and in particular, to a log processing method and device, electronic equipment and storage medium. BACKGROUND

[0002] With the vigorous development of Internet technology, logs used to record the description of dates, times, users and actions related operations are still essential tools, but when performing computer operations, complex and lengthy data often occurs, resulting in too large log data. Excessive log data will cause performance problems for log storage servers, thereby affecting log queries. SUMMARY

[0003] To overcome the problems in the related art, the present disclosure provides a log processing method and device, electronic equipment and storage medium.

[0004] According to a first aspect of an embodiment of the present disclosure, a log processing method is provided, applied to a terminal, and the method comprises:

[0005] In response to a first service generation request log, response data of the request log is obtained;

[0006] When the data volume of the response data is greater than a set threshold, a response ID mapped with the response data is generated, and a mapping relationship between the response data and the response ID is recorded in a mapping log;

[0007] The response ID is written into a normal log instead of the response data;

[0008] The normal log is written into a normal log file, and the mapping log is written into a large log file; wherein data with a data volume greater than a data volume threshold is a large log, and data with a data volume less than the data volume threshold is a normal log.

[0009] Optionally, the method further comprises:

[0010] When the response data is less than the set threshold, the response data is written into the normal log, and then the normal log is written into the normal log file.

[0011] According to a second aspect of an embodiment of the present disclosure, a log processing method is provided, applied to a server, and the method comprises:

[0012] A normal log file and a large log file are obtained;

[0013] When there is an update in the large log file, a mapping log message containing the updated mapping log in the large log file is sent to a first database to store the updated mapping log in the first database.

[0014] Send a regular log message to the second database, the regular log message containing regular logs from the regular log file, to store the regular logs in the second database;

[0015] Specifically, when the response data in the request log of the first service is less than a set threshold, the response data is written into the ordinary log; when the response data is greater than the set threshold, the response ID corresponding to the response data is written into the ordinary log instead of the response data. The mapping log is used to record the mapping relationship between the response data and the response ID in the request log of the first service.

[0016] Optionally, the method further includes:

[0017] When there are no updates in the large log file, a normal log message is sent to the second database to store the normal log message in the second database.

[0018] Optionally, the method further includes:

[0019] In response to an instruction to display the first log through a visualization platform, determine whether the first log contains response data, wherein the first log is any ordinary log in the second database;

[0020] If the first log contains a response ID, retrieve the response data corresponding to the response ID in the first log from the mapping log in the first database;

[0021] After replacing the response ID in the first log with the corresponding response data, the first log is displayed.

[0022] Optionally, the method further includes:

[0023] If the first log contains response data, display the first log.

[0024] Optionally, the ordinary log file and the large log file are obtained through the Filebeat collector, the ordinary log message and the mapped log message are Kafka messages, the first database is an Elasticsearch database, the second database is a MongoDB database, the first database and the second database listen to the ordinary log message and the mapped log message through Logstash, and the visualization platform is Kibana.

[0025] According to a third aspect of the present disclosure, a log processing apparatus is provided for use in a terminal, the apparatus comprising:

[0026] The data acquisition module is used to obtain the response data of the request log in response to the first business request log;

[0027] The generation module is used to generate a response ID that maps to the response data when the amount of response data exceeds a set threshold. The mapping relationship between the response data and the response ID is recorded in the mapping log.

[0028] The first writing module is used to write the response ID into a regular log instead of the response data.

[0029] The second writing module is used to write the ordinary logs to an ordinary log file and to write the mapped logs to a large log file; wherein the data volume is greater than the data volume threshold and the data volume is less than the data volume threshold and the logs are ordinary logs.

[0030] According to a fourth aspect of the present disclosure, a log processing apparatus is provided, applied to a server, the apparatus comprising:

[0031] The file acquisition module is used to acquire both regular log files and large log files.

[0032] The first sending module is used to send a mapping log message to the first database when there is an update in the large log file. The mapping log message contains the updated mapping log in the large log file so as to store the updated mapping log in the first database.

[0033] The second sending module is used to send ordinary log messages to the second database. The ordinary log messages contain ordinary logs from the ordinary log files, so as to store the ordinary logs in the second database.

[0034] Specifically, when the response data in the request log of the first service is less than a set threshold, the response data is written into the ordinary log; when the response data is greater than the set threshold, the response ID corresponding to the response data is written into the ordinary log instead of the response data. The mapping log is used to record the mapping relationship between the response data and the response ID in the request log of the first service.

[0035] According to a fifth aspect of the present disclosure, a log processing apparatus is provided, comprising: a processor; and a memory for storing processor-executable instructions;

[0036] The processor is configured to execute the executable instructions to implement the log processing method described in any of the embodiments of the first aspect above.

[0037] According to a sixth aspect of the present disclosure, a log processing apparatus is provided, comprising: a processor; and a memory for storing processor-executable instructions;

[0038] The processor is configured to execute the executable instructions to implement the log processing method described in any of the embodiments of the second aspect above.

[0039] According to a seventh aspect of the present disclosure, a computer-readable storage medium is provided that stores computer program instructions thereon, which, when executed by a processor, implement the steps of the log processing method described in the first aspect of the present disclosure.

[0040] According to an eighth aspect of the present disclosure, a computer-readable storage medium is provided that stores computer program instructions thereon, which, when executed by a processor, implement the steps of the log processing method described in the second aspect of the present disclosure.

[0041] The technical solutions provided by the embodiments of this disclosure may include the following beneficial effects:

[0042] In the above technical solution, in response to the first service request log, the response data of the request log is obtained; when the data volume of the response data exceeds a set threshold, a response ID mapped to the response data is generated, and the mapping relationship between the response data and the response ID is recorded in the mapping log; the response ID replaces the response data and is written to the ordinary log; the ordinary log is written to the ordinary log file, and the mapping log is written to the large log file; where the data volume is greater than the data volume threshold, it is the large log, and the data volume is less than the data volume threshold, it is the ordinary log. Through the above technical solution, when the data volume of the response data exceeds the set threshold, the response ID mapped to the response data is used to replace the response data and recorded in the log, which to some extent improves the storage problem caused by excessively large data in the log, as well as the log query difficulty caused by excessively large log data.

[0043] Other features and advantages of this disclosure will be described in detail in the following detailed description section. Attached Figure Description

[0044] The accompanying drawings are provided to further illustrate the present disclosure and form part of the specification. They are used together with the following detailed description to explain the present disclosure, but do not constitute a limitation thereof. In the drawings:

[0045] Figure 1 This is a flowchart illustrating a log processing method according to an exemplary embodiment.

[0046] Figure 2 This is a flowchart illustrating a log processing method according to an exemplary embodiment.

[0047] Figure 3 This is a flowchart illustrating yet another log processing method according to an exemplary embodiment.

[0048] Figure 4 This is a block diagram illustrating a log processing apparatus according to an exemplary embodiment.

[0049] Figure 5 This is a block diagram illustrating a log processing apparatus according to an exemplary embodiment.

[0050] Figure 6 This is a block diagram illustrating an electronic device according to an exemplary embodiment.

[0051] Figure 7 This is a block diagram illustrating another electronic device according to an exemplary embodiment. Detailed Implementation

[0052] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0053] It is understood that in this disclosure, "multiple" refers to two or more, and other quantifiers are similar. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, and B alone. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. The singular forms "a," "the," and "the" are also intended to include the plural forms unless the context clearly indicates otherwise.

[0054] It is further understood that the terms "first," "second," etc., are used to describe various types of information, but this information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another, and do not indicate a specific order or degree of importance. In fact, expressions such as "first," "second," etc., are completely interchangeable.

[0055] It is further understood that although operations are described in a specific order in the accompanying drawings in the embodiments of this disclosure, this should not be construed as requiring these operations to be performed in the specific order or serial order shown, or requiring all of the shown operations to be performed to obtain the desired result. In certain environments, multitasking and parallel processing may be advantageous.

[0056] It should be noted that all actions involving the acquisition of signals, information, or data in this application are carried out in compliance with the relevant data protection laws and policies of the country where the application is located, and with the authorization granted by the owner of the relevant device.

[0057] Figure 1 This is a flowchart illustrating a log processing method according to an exemplary embodiment, applied to a terminal, such as... Figure 1 As shown, the method includes the following steps.

[0058] In step S101, in response to the first service generating request log, the response data of the request log is obtained.

[0059] Understandably, this first business function can be business code used to fulfill user needs. When a user initiates a request, a request log containing relevant data about that request can be generated. For example, the request log may include the request initiation time, request headers, and request body. Correspondingly, when the request is processed, corresponding response data will be generated to fulfill the user's needs and obtain response data. The response data may include the time the response was generated, the access path, the size of the returned response data, response headers, and response body.

[0060] In step S102, when the amount of response data exceeds a set threshold, a response ID is generated that maps to the response data, and the mapping relationship between the response data and the response ID is recorded in the mapping log.

[0061] Understandably, response data may contain information such as the time the response was generated, the access path, the size of the returned response data, response headers, and response body. Therefore, when the amount of data in the response data exceeds a set threshold, a response ID that maps to that response data can be generated, and the mapping relationship between the response ID and the response data can be recorded in the mapping log.

[0062] In step S103, the response ID is written to the regular log instead of the response data.

[0063] In step S104, ordinary logs are written to ordinary log files, and mapped logs are written to large log files; where the data volume is greater than the data volume threshold, the logs are large logs, and the data volume is less than the data volume threshold, the logs are ordinary logs.

[0064] Understandably, since the response ID data volume is relatively small, replacing the response data with the response ID in the log will not cause storage bottlenecks due to excessively large log data, thus avoiding performance issues. Regular logs and mapped logs are written to their respective log files. Each mapped log contains at least one response data point with a data volume greater than or equal to a set threshold, along with a response ID that maps to that response data, and the mapping relationship between the response data and the response ID. Multiple mapped logs can be stored in a large log file. Each regular log can contain at least one response ID.

[0065] In the above technical solution, in response to the first service request log, the response data of the request log is obtained. When the data volume of the response data exceeds a set threshold, a response ID mapped to the response data is generated, and the mapping relationship between the response data and the response ID is recorded in the mapping log. The response ID replaces the response data and is written to the ordinary log. The ordinary log is written to the ordinary log file, and the mapping log is written to the large log file. Among them, the data volume exceeding the data volume threshold is the large log, and the data volume less than the data volume threshold is the ordinary log. Through the above technical solution, when the data volume of the response data exceeds the set threshold, the response ID mapped to the response data is used to replace the response data and recorded in the log. Since the data volume of the response ID is much smaller than the data volume of the response data, it can improve the storage problem caused by excessively large data in the log and the log query difficulty caused by excessively large log data to a certain extent.

[0066] Optionally, the method may further include: when the response data is less than a set threshold, writing the response data to a regular log and then writing the regular log to a regular log file.

[0067] It is understandable that when the response data is less than the set threshold, the response data can be written to a regular log and then the regular log can be written to a regular log file. Therefore, each regular log can store at least one response ID and / or at least one response data with a data volume less than the set threshold.

[0068] Figure 2 This is a flowchart illustrating a log processing method according to an exemplary embodiment, such as... Figure 2 As shown, this method is applied to a server, and the method also includes the following steps.

[0069] In step S201, the ordinary log file and the large log file are obtained.

[0070] Understandably, servers can use Filebeat (a log collector) to retrieve both regular and large log files from the terminal. Filebeat is a member of Beats, a lightweight log collector that comes with pre-built modules containing the configurations needed to collect, parse, enrich, and visualize data in various log file formats. For example, when retrieving log files, Filebeat starts a harvester program for each file, continuously reading the log file. Log files collected by different harvester programs are sent to a queue, which can be memory-based or disk-based; Filebeat defaults to a memory-based queue. Whenever data in the queue reaches a certain size or exceeds a set time (default 1 second), it is consumed from the queue by registered clients and sent to the configured backend. Examples of suitable backends include Kafka, a distributed publish-subscribe messaging system with high throughput, and Elasticsearch (also known as ES database), a distributed real-time search and data analytics engine.

[0071] In step S202, when there is an update in the large log file, a mapping log message is sent to the first database. The mapping log message contains the updated mapping log in the large log file, so as to store the updated mapping log in the first database.

[0072] It's understandable that updates in a large log file can be interpreted as updates to the mapped logs within that file. When new mapped logs appear in the large log file, the mapped log message can be sent to the primary database. For example, after Filebeat retrieves the large log file, it packages the newly appearing mapped logs and sends them to Kafka. Upon receiving the message containing the updated mapped logs, Kafka uses the topic as a classifier. After Kafka receives the message containing the updated mapped logs, it sends the updated mapped log message out through the corresponding topic. Logstash, an open-source data collection engine with real-time pipeline capabilities, can be used to listen for Kafka log messages. When Logstash on the server detects messages in a Kafka topic, it sends the messages to the corresponding location. For example, messages containing updated mapped logs can be sent to the primary database, where the updated mapped logs can be stored. This primary database can be a database within Elasticsearch.

[0073] In step S203, a regular log message is sent to the second database. This regular log message contains regular logs from a regular log file, so that the regular logs are stored in the second database.

[0074] Specifically, when the response data in the request log of the first service is less than a set threshold, the response data is written to the ordinary log. When the response data is greater than the set threshold, the response ID corresponding to the response data is written to the ordinary log instead of the response data. This mapping log is used to record the mapping relationship between the response data and the response ID in the request log of the first service.

[0075] For example, similar to step S202, after Filebeat obtains the ordinary log file, it packages the updated ordinary log and sends it to Kafka. After Kafka receives the message containing the updated ordinary log, the topic can act as a classifier. After Kafka obtains the message containing the updated ordinary log, it sends the message of the updated ordinary log out through the corresponding topic. When the Logstash server listens for the message of the topic in Kafka, it will send the message to the corresponding location. For example, the message containing the updated ordinary log can be sent to the second database to store the updated ordinary log in the second database. The second database can be a MongoDB database (MongoDB is a database based on distributed file storage).

[0076] In the above technical solution, ordinary log files and large log files are obtained; when there is an update in the large log file, a mapping log message is sent to the first database, which contains the updated mapping log in the large log file, so as to store the updated mapping log in the first database; an ordinary log message is sent to the second database, which contains the ordinary log in the ordinary log file, so as to store the ordinary log in the second database. Through the above technical solution, the mapping log and ordinary log are stored in different databases, which can facilitate log querying while reducing the storage pressure on the database.

[0077] Optionally, the method may further include:

[0078] When there are no updates in the large log file, a regular log message is sent to the second database to store the regular log message in the second database.

[0079] It is understandable that when there is no updated mapping log in the large log file, a normal log message can be sent to the second database. In other words, whether there is no updated mapping log in the large log file or there is an updated mapping log in the large log file, a normal log message can be sent to the second database and stored in the second database.

[0080] Figure 3 This is a flowchart illustrating a log processing method according to an exemplary embodiment, such as... Figure 3 As shown, the method also includes the following steps.

[0081] In step S301, in response to the instruction to display the first log through the visualization platform, it is determined whether the first log contains response data, wherein the first log is any ordinary log in the second database.

[0082] In step S302, if the first log contains a response ID, the response data corresponding to the response ID in the first log is obtained from the mapping log in the first database.

[0083] In step S303, the first log is displayed after the corresponding response data replaces the response ID in the first log.

[0084] Understandably, when logs need to be displayed, a visualization platform can be used. This platform could be Kibana (Kibana is an open-source analytics and visualization platform designed to work with Elasticsearch databases, allowing interaction with data stored in Elasticsearch, such as searching and viewing data). When querying logs, Kibana can index the desired logs by name and time and display detailed log content. Before displaying the logs, if the first log contains a response ID, the response data corresponding to that response ID is retrieved from the mapping logs stored in the first database. This response data replaces the original response ID and is then displayed.

[0085] Optionally, the method may further include:

[0086] If the first log contains response data, display the first log.

[0087] With the above technical solution, when displaying logs through a visualization platform, the data in the logs can be mapped to response data based on the response ID. This allows the display of response data that is greater than or equal to a set threshold. When the data in the logs is response data that is less than the set threshold, the response data can be displayed directly. This improves the flexibility of log display and the convenience of log querying to a certain extent, and avoids the performance bottleneck problem caused by excessively large log data.

[0088] Figure 4This is a block diagram illustrating a log processing apparatus according to an exemplary embodiment, applied to a terminal, such as... Figure 4 As shown, the log processing device 400 includes...

[0089] The data acquisition module 401 is used to obtain the response data of the request log generated by the first business.

[0090] The generation module 402 is used to generate a response ID that maps to the response data when the amount of response data exceeds a set threshold. The mapping relationship between the response data and the response ID is recorded in the mapping log.

[0091] The first write module 403 is used to write the response ID instead of the response data into the ordinary log.

[0092] The second writing module 404 is used to write ordinary logs to ordinary log files and to write mapped logs to large log files; where the data volume is greater than the data volume threshold is the large log and the data volume is less than the data volume threshold is the ordinary log.

[0093] Optionally, the log processing device 400 may include:

[0094] The second writing module is used to write the response data to a regular log and then write the regular log to a regular log file when the response data is less than a set threshold.

[0095] In the above technical solution, in response to the first service request log, the response data of the request log is obtained; when the data volume of the response data exceeds a set threshold, a response ID mapped to the response data is generated, and the mapping relationship between the response data and the response ID is recorded in the mapping log; the response ID replaces the response data and is written to the ordinary log; the ordinary log is written to the ordinary log file, and the mapping log is written to the large log file; where the data volume is greater than the data volume threshold, it is the large log, and the data volume is less than the data volume threshold, it is the ordinary log. Through the above technical solution, when the data volume of the response data exceeds the set threshold, the response ID mapped to the response data is used to replace the response data and recorded in the log, which to some extent improves the storage problem caused by excessively large data in the log, as well as the log query difficulty caused by excessively large log data.

[0096] Figure 5 This is a block diagram illustrating a log processing apparatus according to an exemplary embodiment, including an application server, such as... Figure 5 As shown, the log processing device 500 includes...

[0097] File acquisition module 501 is used to acquire ordinary log files and large log files;

[0098] The first sending module 502 is used to send a mapping log message to the first database when there is an update in the large log file. The mapping log message contains the updated mapping log in the large log file so as to store the updated mapping log in the first database.

[0099] The second sending module 503 is used to send a normal log message to the second database. The normal log message contains the normal log in the normal log file so as to store the normal log in the second database.

[0100] Specifically, when the response data in the request log of the first service is less than a set threshold, the response data is written to the ordinary log. When the response data is greater than the set threshold, the response ID corresponding to the response data is written to the ordinary log instead of the response data. This mapping log is used to record the mapping relationship between the response data and the response ID in the request log of the first service.

[0101] Optionally, the log processing device 500 may include:

[0102] The data determination module is used to determine whether the first log contains response data in response to an instruction to display the first log through a visualization platform. The first log is any ordinary log in the second database.

[0103] The acquisition module is used to retrieve response data corresponding to the response ID in the first log from the mapping log in the first database, provided that the first log contains a response ID.

[0104] The first display module is used to replace the response ID in the first log with the corresponding response data and then display the first log.

[0105] Optionally, the log processing device 500 may include:

[0106] The second display module is used to display the first log if the first log contains response data.

[0107] In the above technical solution, ordinary log files and large log files are obtained; when there is an update in the large log file, a mapping log message is sent to the first database, which contains the updated mapping log in the large log file, so as to store the updated mapping log in the first database; an ordinary log message is sent to the second database, which contains the ordinary log in the ordinary log file, so as to store the ordinary log in the second database. Through the above technical solution, the mapping log and ordinary log are stored in different databases, which can facilitate log querying while reducing the storage pressure on the database.

[0108] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0109] Figure 6 This is a block diagram illustrating an electronic device 600 according to an exemplary embodiment, such as... Figure 6 As shown, the electronic device 600 may include a processor 601 and a memory 602. The electronic device 600 may also include one or more of a multimedia component 603, an input / output (I / O) interface 604, and a communication component 605.

[0110] The processor 601 controls the overall operation of the electronic device 600 to complete all or part of the steps in the log processing method described above. The memory 602 stores various types of data to support the operation of the electronic device 600. This data may include, for example, instructions for any application or method operating on the electronic device 600, and application-related data such as contact data, sent and received messages, pictures, audio, video, etc. The memory 602 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read-Only Memory (EPROM), Programmable Read-Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. Multimedia component 603 may include a screen and an audio component. The screen may be, for example, a touchscreen, and the audio component is used to output and / or input audio signals. For example, the audio component may include a microphone for receiving external audio signals. The received audio signals may be further stored in memory 602 or transmitted via communication component 605. The audio component also includes at least one speaker for outputting audio signals. I / O interface 604 provides an interface between processor 601 and other interface modules, such as a keyboard, mouse, buttons, etc. These buttons may be virtual or physical buttons. Communication component 605 is used for wired or wireless communication between the electronic device 600 and other devices. Wireless communication, such as Wi-Fi, Bluetooth, Near Field Communication (NFC), 2G, 3G, 4G, NB-IoT, eMTC, or other 5G technologies, or combinations thereof, is not limited here. Therefore, the corresponding communication component 605 may include: a Wi-Fi module, a Bluetooth module, an NFC module, etc.

[0111] In an exemplary embodiment, the electronic device 600 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the log processing method described above.

[0112] In another exemplary embodiment, a computer-readable storage medium including program instructions is also provided, which, when executed by a processor, implement the steps of the log processing method described above. For example, the computer-readable storage medium may be the memory 602 including the program instructions described above, which may be executed by the processor 601 of the electronic device 600 to complete the log processing method described above.

[0113] Figure 7 This is a block diagram illustrating another electronic device 700 according to an exemplary embodiment. For example, the electronic device 700 may be provided as a server. (Refer to...) Figure 7 The electronic device 700 includes a processor 722, which may be one or more, and a memory 732 for storing computer programs executable by the processor 722. The computer programs stored in the memory 732 may include one or more modules, each corresponding to a set of instructions. Furthermore, the processor 722 may be configured to execute the computer program to perform the log processing method described above.

[0114] Additionally, the electronic device 700 may also include a power supply component 726 and a communication component 750. The power supply component 726 can be configured to perform power management of the electronic device 700, and the communication component 750 can be configured to enable communication of the electronic device 700, such as wired or wireless communication. Furthermore, the electronic device 700 may also include an input / output (I / O) interface 758. The electronic device 700 can operate on an operating system stored in the memory 732.

[0115] In another exemplary embodiment, a computer-readable storage medium including program instructions is also provided, which, when executed by a processor, implement the steps of the log processing method described above. For example, the non-transitory computer-readable storage medium may be the memory 732 including the program instructions described above, which may be executed by the processor 722 of the electronic device 700 to complete the log processing method described above.

[0116] In another exemplary embodiment, a computer program product is also provided, the computer program product comprising a computer program executable by a programmable device, the computer program having a code portion for performing the above-described log processing method when executed by the programmable device.

[0117] The preferred embodiments of this disclosure have been described in detail above with reference to the accompanying drawings. However, this disclosure is not limited to the specific details of the above embodiments. Within the scope of the technical concept of this disclosure, various simple modifications can be made to the technical solutions of this disclosure, and these simple modifications all fall within the protection scope of this disclosure.

[0118] It should also be noted that the various specific technical features described in the above specific embodiments can be combined in any suitable manner without contradiction. In order to avoid unnecessary repetition, this disclosure will not describe the various possible combinations separately.

[0119] Furthermore, various different embodiments of this disclosure can be combined in any way, as long as they do not violate the spirit of this disclosure, they should also be regarded as the content disclosed in this disclosure.

Claims

1. A log processing method characterized by, The method applied to a terminal comprises: In response to a first service generation request log, response data of the request log is acquired; When the data volume of the response data is greater than a set threshold, a response ID mapped with the response data is generated, and a mapping relationship between the response data and the response ID is recorded in a mapping log; The response ID is written into a general log instead of the response data; The general log is written into a general log file, and the mapping log is written into a large log file; wherein data with a data volume greater than a data volume threshold is a large log, and data with a data volume less than the data volume threshold is a general log.

2. The method of claim 1, wherein, The method further comprises: When the response data is less than the set threshold, the response data is written into the general log, and then the general log is written into the general log file.

3. A log processing method characterized by, The method applied to a server comprises: A general log file and a large log file are acquired; When there is an update in the large log file, a mapping log message containing the updated mapping log in the large log file is sent to a first database, so as to store the updated mapping log in the first database; A general log message containing a general log in the general log file is sent to a second database, so as to store the general log in the second database; When the response data of the request log of the first service is less than a set threshold, the response data is written into the general log, and when the response data is greater than the set threshold, a response ID corresponding to the response data is written into the general log instead of the response data, and the mapping log is used to record the mapping relationship between the response data and the response ID of the request log of the first service.

4. The method of claim 3, wherein, The method further comprises: When there is no update in the large log file, a general log message is sent to a second database, so as to store the general log message in the second database.

5. The method of claim 4, wherein, The method further comprises: In response to an instruction for displaying a first log through a visualization platform, it is determined whether the first log contains response data, the first log being any general log in the second database; In the case that the first log contains a response ID, response data corresponding to the response ID in the first log is acquired from a mapping log in the first database; After the corresponding response data is substituted for the response ID in the first log, the first log is displayed.

6. The method of claim 5, wherein, In the case that the first log contains response data, the first log is displayed. The general log file and the large log file are acquired through a filebeat collector, the general log message and the mapping log message are kafka messages, the first database is an ES database, the second database is a MongoDB database, the first database and the second database listen to the general log message and the mapping log message through a logstash, and the visualization platform is kibana.

7. The method according to claim 5 or 6, characterized in that, The device applied to a terminal comprises:

8. A log processing apparatus characterized by comprising: ​ The data acquisition module is configured to acquire response data of the request log in response to the first service generation request log; The generation module is configured to generate a response ID mapped with the response data when a data volume of the response data is greater than a set threshold value, and a mapping relationship between the response data and the response ID is recorded in a mapping log; The first writing module is configured to write the response ID into a normal log instead of the response data; The second writing module is configured to write the normal log into a normal log file and write the mapping log into a large log file; wherein data with a data volume greater than a data volume threshold value is a large log, and data with a data volume less than the data volume threshold value is a normal log.

9. A log processing apparatus characterized by comprising: The application is applied to a server, and the device comprises: The file acquisition module is configured to acquire a normal log file and a large log file; The first sending module is configured to send a mapping log message to a first database when there is an update in the large log file, the mapping log message comprising an updated mapping log in the large log file, so as to store the updated mapping log in the first database; The second sending module is configured to send a normal log message to a second database, the normal log message comprising a normal log in the normal log file, so as to store the normal log in the second database; When the response data of the request log of the first service is less than a set threshold value, the response data is written into the normal log, and when the response data is greater than the set threshold value, a response ID corresponding to the response data is written into the normal log instead of the response data, and the mapping log is used to record a mapping relationship between the response data of the request log of the first service and the response ID.

10. An electronic device, comprising: comprise: a processor; a memory for storing processor-executable instructions; wherein the processor is configured to execute the executable instructions to implement the steps of the log processing method in any one of claims 1-2.

11. An electronic device, comprising: comprise: a processor; a memory for storing processor-executable instructions; wherein the processor is configured to execute the executable instructions to implement the steps of the log processing method in any one of claims 3-7.

12. A computer-readable storage medium having stored thereon computer program instructions, wherein, The program instructions are executed by the processor to implement the steps of the log processing method in any one of claims 1-2.

13. A computer-readable storage medium having stored thereon computer program instructions, wherein, The program instructions are executed by the processor to implement the steps of the log processing method in any one of claims 3-7. The program instructions are executed by the processor to implement the steps of the log processing method in any one of claims 1-2. The program instructions are executed by the processor to implement the steps of the log processing method in any one of claims 3-7.

Images